Stefano Aldegheri

Nicola Bombieri

Integrating Simulink, OpenVX, and ROS for Model-based Design of Embedded Vision Applications

OpenVX is increasingly gaining consensus as standard platform to develop portable, optimized and power-efficient embedded vision applications. Nevertheless, adopting OpenVX for rapid prototyping, early algorithm parametrization and validation of complex embedded applications is a very challenging task. This paper presents a comprehensive framework that integrates Simulink, OpenVX, and ROS for model-based design of embedded vision applications. The framework allows applying Matlab-Simulink for the model-based design, parametrization, and validation of computer vision applications. Then, it allows for the automatic synthesis of the application model into an OpenVX description for the hardware and constraints-aware application tuning. Finally, the methodology allows integrating the OpenVX application with Robot Operating System (ROS), which is the de-facto reference standard for developing robotic software applications. The OpenVX-ROS interface allows cosimulating and parametrizing the application by considering the actual robotic environment and the application reuse in any ROS-compliant system. Experimental results have been conducted with two real case studies: An application for digital image stabilization and the ORB descriptor for simultaneous localization and mapping (SLAM), which have been developed through Simulink and, then, automatically synthesized into OpenVX-VisionWorks code for an NVIDIA Jetson TX2 board. OpenCL CUDA C/C++ OpenVX API standard interface API implem. Engines/ frameworks HW vendors Runtime engines Application developers Low-power heterogeneous board Systemlevel opt. Automatic optimiz. and synthesis Graph-based modelling Embedded vision application SDK OpenCL CUDA C/C++ OpenVX API standard interface API implem. Engines/ frameworks HW vendors Runtime engines Low-power heterogeneous board Systemlevel opt. Automatic optimiz. and synthesis SDK Matlab/Simulink Model-based design Embedded vision application Graph synthesis Multi-level design and verification flow (a) (b) ROS I/O ROS I/O

Introduction

Computer vision has gained an increasing interest as an efficient way to automatically extract of meaning from images and video. It has been an active field of research for decades, but until recently has had few major commercial applications. However, with the advent of high-performance, low-cost, energy efficient processors, it has quickly become largely applied in a wide range of applications for embedded systems [START_REF]Applications for Embedded Vision[END_REF].

The term embedded vision refers to this new wave of widely deployed, practical computer vision applications properly optimized for a target embedded system by considering a set of design constraints. The target embedded systems usually consist of heterogeneous, multi-/many-core, low power embedded devices, while the design constraints, beside functional correctness, include performance, energy efficiency, dependability, real-time response, resiliency, fault tolerance, and certifiability.

Developing and optimizing a computer vision application for an embedded processor can be a non-trivial task. Considering an application as a set of communicating and interacting kernels, the effort for such application optimization goes over two dimensions: the single kernel-level optimization and the systemlevel optimization. Kernel-level optimizations have traditionally revolved around one-off or single function acceleration. This typically means that a developer rewrites a computer vision function (e.g., any filter, image arithmetic, geometric transform function) with a more efficient algorithm or offloads its execution to accelerators such as a GPU by using languages such as OpenCL or CUDA [START_REF] Pulli | Realtime computer vision with OpenCV[END_REF].

On the other hand, system-level optimizations pay close attention to the overall power consumption, memory bandwidth loading, low-latency functional computing, and Inter-Processor Communication overhead. These issues are typically addressed via frameworks [START_REF] Rainey | Addressing system-level optimization with OpenVX graphs[END_REF], as the parameters of interest cannot be tuned with compilers or operating systems.

In this context, OpenVX [START_REF] Group | OpenVX: Portable, Power-efficient Vision Processing[END_REF] has gained wide consensus in the embedded vision community and has become the de-facto reference standard and API library for system-level optimization. OpenVX is designed to maximize functional and performance portability across different hardware platforms, providing a computer vision framework that efficiently addresses current and future hardware architectures with minimal impact on software applications. Starting from a graph model of the embedded application, it allows for automatic system-level optimizations and synthesis on the HW board targeting performance and power consumption design constraints [START_REF] Tagliavini | Adrenaline: An openvx environment to optimize embedded vision applications on many-core accelerators[END_REF][START_REF] Yang | Analysis for supporting real-time computer vision workloads using openvx on multicore+gpu platforms[END_REF][START_REF] Dekkiche | Investigation and performance analysis of openvx optimizations on computer vision applications[END_REF].

Nevertheless, the definition of such a graph-based model, its parametrization and validation is time consuming and far from intuitive to programmers, especially for the development of medium-complex applications.

Embedded vision finds a large use in the context of Robotics, where cameras are mounted on robots and the results of the embedded vision applications are analysed for autonomous actions. Indeed, Computer vision allows robots to see what is around them and make decisions based on what they perceive. In this context, Robot Operating System (ROS) [START_REF]Robot Operating System[END_REF] has been proposed as a flexible framework for writing robot software. It is a collection of tools, libraries, and conventions that aim to simplify the task of creating complex and robust robot behaviour across a wide variety of robotic platforms. It is become a de-facto reference standard for developing robotic applications. It allows for application re-use and easy integration of software blocks in complex systems.

This paper presents a comprehensive framework that integrates Simulink, OpenVX, and ROS for the model-based design of embedded vision applications (see Figure 1). Differently from the standard approaches at the state of the art that require designers to manually model the algorithm through OpenVX code (see Figure 1(a)), the proposed approach allows for a rapid prototyping, algorithm validation and parametrization in a model-based design environment (i.e., Matlab/Simulink). The framework relies on a multi-level design and verification flow (see Figure 1(b)) by which the high-level model is then semi-automatically refined towards the final automatic synthesis into OpenVX code. The integration with ROS has two main goals: first, to allow co-simulating and parametrizing the application by considering the actual robotic environment, and then, to allow for application reuse in ROS-compliant systems.

The paper presents the results obtained by applying the proposed methodology for developing and tuning two real-case applications. The first is an algorithm for digital image stabilization for two different application contexts. The second is the application implementing the the oriented fast and rotated brief (ORB) descriptor for simultaneous localization and mapping (SLAM). The paper presents the Simulink toolbox developed to support the NVIDIA OpenVX-VisionWorks library, and how it has been used in the design flow to synthesize OpenVX code for an NVIDIA Jetson TX2 embedded system board.

The paper is organized as follows. Section 2 presents the background and the related work. Section 3 explains in details the model-based design methodology. Section 4 presents the experimental results, while Section 5 is devoted to the conclusions.

Background and Related work

OpenVX relies on a graph-based software architecture to enable efficient computation on heterogeneous computing platforms, including those with GPU accelerators. It provides a set of primitives (or kernels) that are commonly used in computer vision algorithms. It also provides a set of data objects like scalars, arrays, matrices and images, as well as high-level data objects like histograms, image pyramids, and look-up tables. It supports customized user-defined kernels for implementing customized application features.

The programmer defines a computer vision algorithm by instantiating kernels as nodes and data objects as parameters. Since each node may use the mix of the processing units in the heterogeneous platform, a single graph may be executed across CPUs, GPUs, DSPs, etc. Figure 2 and Listing 1.1 give an example of computer vision application and its OpenVX code, respectively. The programming flow starts by creating an OpenVX context to manage references to all used objects (line 1, Listing 1.1). Based on this context, the code builds the graph (line 2) and generates all required data objects (lines 4 to 11). Then, it instantiates the kernel as graph nodes and generates their connections (lines 15 to 18). The graph integrity and correctness is checked in line 20 (e.g., checking of data type coherence between nodes and absence of cycles). Finally, the graph is processed by the OpenVX framework (line 23). At the end of the code execution, all created data objects, the graph, and the the context are released.

The definition of algorithms through primitives has two benefits: First, it allows defining the application in an abstract way while preserving an efficient implementation. Then, it allows enabling system-level optimizations, like internode memory transfers, pipelining, concurrent and overlapped node execution. To utilize the different accelerators on the board, data transfer management needs to be addressed. Each operation requires time and power resources, and this has to be considered in the mapping process. Pipelining and tiling techniques can be efficiently utilized together to achieve better memory locality. This greatly reduces the data transfer overhead between global and scratchpad memory [START_REF] Popp | Automatic control flow generation for openvx graphs[END_REF].

Different works have been presented to analyse the use of OpenVX for embedded vision [START_REF] Tagliavini | Adrenaline: An openvx environment to optimize embedded vision applications on many-core accelerators[END_REF][START_REF] Yang | Analysis for supporting real-time computer vision workloads using openvx on multicore+gpu platforms[END_REF][START_REF] Dekkiche | Investigation and performance analysis of openvx optimizations on computer vision applications[END_REF][START_REF] Aldegheri | Extending openvx for model-based design of embedded vision applications[END_REF]. In [START_REF] Yang | Analysis for supporting real-time computer vision workloads using openvx on multicore+gpu platforms[END_REF], the authors present a new implementation of OpenVX targeting CPUs and GPU-based devices by leveraging different analytical optimzation techniques. In [START_REF] Dekkiche | Investigation and performance analysis of openvx optimizations on computer vision applications[END_REF], the authors examine how OpenVX responds to different data access patterns, by testing three different OpenVX optimizations: kernels merge, data tiling and parallelization via OpenMP. In [START_REF] Tagliavini | Adrenaline: An openvx environment to optimize embedded vision applications on many-core accelerators[END_REF], the authors introduce ADRENALINE, a novel framework for fast prototyping and optimization of OpenVX applications for heterogeneous SoCs with many-core accelerators. The authors in [START_REF] Syschikov | Openvx integration into the visual development environment[END_REF] implemented a graphic interface that allows computer vision developers to create visual algorithms in OpenVX. The framework then automatically generates the corresponding OpenVX code, with a translation back-end that creates all the glue code needed to correctly run the OpenVX environment. This work extends the preliminary implementation of the model-based design presented in [START_REF] Aldegheri | Extending openvx for model-based design of embedded vision applications[END_REF] by including the interface towards ROS. This allows co-simulating the OpenVX application with the external application environment (e.g., input streams, concurrent interactive systems, etc.) and, as a consequence, tuning more efficiently the SW parametrization. Results on a more advanced Robotic application (ORB descriptor) underlines that making any application ROS-compliant is strategic for IP-reuse. 6 vx_image gray = v x C r e a t e V i r t u a l I m a g e (g , 0 , 0 , type ) ; 7 vx_image grad_x = v x C r e a t e V i r t u a l I m a g e (g , 0 , 0 , type ) ; 8 vx_image grad_y = v x C r e a t e V i r t u a l I m a g e (g , 0 , 0 , type ) ; 9 vx_image grad = v x C r e a t e V i r t u a l I m a g e (g , 0 , 0 , type ) ; vx_image out = vxCreateImage (c , w , h , VX_DF _IMAGE_ U8 ) ; Differently from all the work of the literature, this paper presents an extension of the OpenvX environment to the model-based design paradigm. Such an extension aims at exploiting the model-based approach for the fast prototyping of any computer vision algorithm through a Matlab/Simulink model, its parametrization, validation, and automatic synthesis into an equivalent OpenVX code representation.

vx_threshold threshold = v x C

The model-based design approach

Figure 3 depicts the overview of the proposed design flow. The computer vision application is firstly developed in Matlab/Simulink, by exploiting a computer vision oriented toolbox of Simulink1 . Such a block library allows developers to define the application algorithms through Simulink blocks and to quickly simulate and validate the application at system level. The platform allows specific and embedded application primitives to be defined by the user if not included in the

Analysis of results

OpenVX application toolbox through the Simulink S-Function construct [START_REF]S-Functions[END_REF] (e.g., user-defined block UDB Block 4 in Figure 3). Streams of frames are given as input stimuli to the application model and the results (generally represented by frames or streams of frames) are evaluated by adopting any ad-hoc validation metrics from the computer vision literature (e.g., [START_REF] Smith | Light field video stabilization[END_REF]). Efficient test patterns are extrapolated, by using any technique of the literature, to assess the quality of the application results by considering the adopted validation metrics.

The high-level application model is then automatically synthesized for a low-level simulation and validation through Matlab/Simulink. Such a simulation aims at validating the computer vision application at system-level by using the OpenVX primitive implementations provided by the HW board vendor (e.g., NVIDIA VisionWorks) instead of Simulink blocks. The synthesis, which is performed through e Matlab routine, relies on two key components:

1. The OpenVX toolbox for Simulink. Starting from the library of OpenVX primitives (e.g., NVIDIA VisionWorks [START_REF] Nvidia Inc | VisionWorks[END_REF], INTEL OpenVX [START_REF]Intel Computer Vision SDK[END_REF], AMDOVX [START_REF] Amd | AMD OpenVX -AMDOVX[END_REF], Khronos OpenVX standard implementation [START_REF] Khronos | OpenVX lib[END_REF]), such a toolbox of blocks for Simulink is created by properly wrapping the primitives through Matlab S-F unction, as explained in Section 3.1.

The OpenVX primitives-Simulink blocks mapping table. It provides the map-

ping between Simulink blocks and the functionally equivalent OpenVX primitives, as explained in Section 3.2.

As explained in the experimental results, we created the OpenVX toolbox for Simulink of the NVIDIA VisionWorks library as well as the mapping table between VisionWorks primitives and Simulink CVT blocks. They are available for download from https://profs.sci.univr.it/bombieri/VW4Sim.

The low-level representation allows simulating and validating the model by reusing the test patterns and the validation metrics identified during the higher level (and faster) simulation.

The low-level Simulink model is synthesized, through a Matlab script, into an OpenVX model, which is executed and validated on the target embedded board. At this level, all the techniques of the literature for OpenVX systemlevel optimization can be applied. The synthesis is straightforward (and thus not addressed in this paper for the sake of space), as all the key information required to build a stand-alone OpenVX code is contained in the low-level Simulink model. Both the test patterns and the validation metrics can be re-used for the nodelevel and system-level optimization of the OpenVX application.

The proposed design flow also allows the embedded vision application to be refined by considering the external Robotics system, which is supposed to be implemented as ROS-compliant application. The OpenVX model is interfaced to ROS through a set of interface templates, which implement the OpenV-ROS communication based on message passing. A lightweight target I/O module is responsible to handle the information sent by the system (e.g., sensors, controllers, etc.) and to translate it into an OpenVX data structure. A similar I/O module implements the initiator interface, which allows sending information from OpenVX (generally the results of a computation) to the ROS system. By relying on the ROS commuincation protocol, the embedded vision application can easily interact with multiple external actors, allowing an easy integration and reuse into real Robotics systems.

OpenVX toolbox for Simulink

The generation of the OpenVX toolbox for Simulink relies on the S-function construct, which allows describing any Simulink block functionality through C/C++ code. The code is compiled as mex file by using the Matlab mex utility [START_REF]mex functions[END_REF]. As with other mex files, S-functions are dynamically linked subroutines that the Matlab execution engine can automatically load and execute. S-functions use a special calling syntax (i.e., S-function API ) that enables the interaction between the block and the Simulink engine. This interaction is very similar to the interaction that takes place between the engine and built-in Simulink blocks. We defined a S-function template to build OpenVX blocks for Simulink that, as for the construct specifications, consists of four main phases (see the example in Listing 1.2, which represents the Color Converter node of Figure 2 summarized in Figure 4. The context wrapper allows creating the OpenVX context (see line 1 of Listing 1.1), which is mandatory for any OpenVX primitive execution. It is run once for the whole system application. The data wrapper allows creating the OpenVX data structures for the primitive communication (see in, gray, grad x , grad y , grad, and out in the example of Figure 2 and lines 4-11 of Listing 1.1). It is run once for each application block. The primitive wrapper allows executing, in the Simulink context, each primitive functionality implemented in OpenVX. To speed up the simulation, the wrapped primitives work through references to data structures, which are passed as function parameters during the primitive invocations to the OpenVX context. To do that, the wrappers implement memory locking mechanisms (i.e., through the Matlab mem lock()/mem unlock() constructs) to prevent data objects to be released automatically by the Matlab engine between primitive invocations.

Mapping table between OpenVX primitives and Simulink blocks

To enable the application model synthesis from the high-level to the low-level representation, mapping information is required to put in correspondence the built-in Simulink blocks and the corresponding OpenVX primitives. In this work, we defined such a mapping table between the Simulink CVT Toolbox and the NVIDIA OpenVX-VisionWorks library. The table, which consists of 58 entries in the current release, includes primitives for image arithmetic, flow and depth, geometric transforms, filters, feature and analysis operations. We implemented three possible mapping strategies:

1. 1-to-1: the Simulink block is mapped to a single OpenVX primitive (e.g., color converter image arithmetic). 2. 1-to-n: the Simulink block functionality is implemented by a concatenation of multiple OpenVX primitives (e.g., the opening morphological operation). 3. n-to-1: a concatenation of multiple Simulink blocks are needed to implement a single OpenVX primitive (e.g., subtract + absolute blocks).

For some entry, the mapping also depends on the Simulink block setting. As an example, the OpenVX primitive for edge detection is selected depending on the setting of the corresponding CVT block. The setting includes the choice of the filter algorithm (i.e., Canny or Sobel) and the filter size.

The blocks listed in the left-most column of the table form the OpenVX toolbox for Simulink. Any Simulink model built from them can undergo the proposed automatic refinement flow. In addition, user-defined Simulink blocks implemented in C/C++ are supported and translated into OpenVX user kernels. They are eventually loaded and included in the OpenVX representation as graph nodes. To do that, we defined the wrapper represented in Listing 1.3, which follows the node implementation directives required by the standard OpenVX for importing user kernels similar to the invocation of any built-in OpenVX node (i.e., vxN ode()) in the OpenVX context through the previously presented context wrapper (see the righmost side of Figure 4).

Finally, some restrictions on the Simulink block interfaces are required to allow the Simulink/OpenVX communication as well as the model synthesis. The set of data types and data structures available for the high-level model is reduced to the subset supported by OpenVX, whereby each I/O port of the Simulink blocks consists of:

-Dimension d ∈ {1D, 2D, 3D, 3D + AlphaChannel}, e.g., greyscale, RGB or YUV, and alpha channel for transparency.

-Size s ∈ {N × M × 1, N × M × 3, N × M × 4}.
-Type t ∈ {uint8, f loat}, where uint8 is generally used for representing data (pixels, colours, etc.) while f loat is generally used for representing interpolation data.

ROS integration

The adoption of ROS provides different advantages. First, it allows the platform to model and simulate blocks running on different target devices. Then, it implements the inter-node communication in a modular way and by adopting a standard and widespread protocol, thus guaranteeing code portability. ROS implements the messages passing among nodes by providing a publishsubscribe communication model. Every message sent through ROS has a topic, which is a unique string known by all the communicating nodes. An initiator Figure 6(a) shows the skeleton implementation of the server interface. The process init function is responsible to perform the node initialization in the ROS framework. It adds the current process to the ROS node list in the master server (lines [START_REF] Khronos | OpenVX lib[END_REF][START_REF]mex functions[END_REF]. This node is sensitive to the topic specified in line 23. Line 24 specifies the function that will be called on the server invocation. Lines 1-13 provide the invocation of the OpenVX application. Two parameters are necessary to the function: the request, which contains the input data, and the response, which will be updated by the computing function. Conversion functions are defined to convert the data format between ROS and OpenVX. Finally, line 26 implements the busy waiting until the ROS framework shuts down all the nodes. After adding the process to the list of ROS nodes (lines 3-4), the system informs the ROS framework that the client requests need to be forwarded to the topic service listener (lines 7-10). The wrapper creates the object to write the results of the computation (lines [START_REF] Aldegheri | Extending openvx for model-based design of embedded vision applications[END_REF][START_REF]S-Functions[END_REF]. Parameters are filled in line 15, and the call to request data is performed in line 16. In case of positive message receiving, the OpenVX computation is called (lines 20-21), through ad-hoc functions to convert the data format between ROS and OpenVX. The system publishes the results back to the network (line 19). Figure 8 shows the temporal evolution of such a client model process.

Experimental Results

We applied the proposed model-based design flow for the development of two embedded software: the first one implements a digital image stabilization algorithm for camera streams, while the latter calculate the ORB descriptor.

Image stabilization

Figure 9 shows an overview of the algorithm, which is represented through a dependency graph. The input stream (i.e., sequence of frames) is taken from a high-definition camera, and each frame is converted to the grayscale format to improve the algorithm efficiency without compromising the quality of the result. A remapping operation is then applied to the resulting frames to remove fisheye distortions. A sparse optical flow is applied to the points detected in the previous frame by using a feature detector (e.g., Harris or Fast detector). The resulting points are then compared to the original point to find the homography matrix. The last N matrices are then combined by using a Gaussian filtering, where N is defined by the user (higher N means more smoothed trajectory a the cost of more latency). Finally, each frame is inversely warped to get the final result. Dashed lines in Figure 9 denote inter-frame dependencies, i.e., parts of the algorithm where a temporal window of several frames is used to calculate the camera translation. We firstly modelled the algorithm application in Simulink (CVT toolbox). The nodes Optical flow and Filtering have been inserted as user-defined blocks, since they implement customized functionality and are not present in the CVT toolbox. We conducted two different parametrizations of the algorithm, and in particular of the feature detection phase: For an indoor and for an outdoor application context. The first targets a system for indoor navigation of an Unmanned aerial vehicle (UAV), while the second targets a system for outdoor navigation of an Autonomous Surface Crafts (ASCs) [START_REF] Aldegheri | Fast and powerefficient embedded software implementation of digital image stabilization for lowcost autonomous boats[END_REF].

We validated the two algorithm configurations starting from input streams registered by different cameras at 60 FPS with 1280x720 (1080P) and 1920x1080 wide angle resolution, respectively. Table 2 rics for light field video stabilization [START_REF] Smith | Light field video stabilization[END_REF]. The table reports the characteristics of such selected test patterns (sequences of frames).

We then applied the Matlab synthesis script to translate the high-level model into the low-level model by using the OpenVX toolbox for Simulink generated from the NVIDIA VisionWorks v1.6 [START_REF] Nvidia Inc | VisionWorks[END_REF] and the corresponding Simulink CVT-NVIDIA OpenVX/VisionWorks mapping table, as described in Sections 3.1 and 3.2, respectively. In particular, the low level simulation in Simulink allowed us to validate the computer vision application implemented through the primitives provided by the HW board vendor (e.g., NVIDIA OpenVX-VisionWorks) instead of Simulink blocks.

Finally, we synthesized the low-level model into pure OpenVX code, by which we run the real time analysis and validation on the target embedded board (NVIDIA Jetson TX1). Table 3 reports a comparison among the different simulation time (real execution time for the OpenVX code) spent to validate the embedded software application at each level of the design flow. At each refinement step, we reused the selected test patterns to verify the code over the adopted validation metrics [START_REF] Smith | Light field video stabilization[END_REF] for both the contexts and by assuming a maximum deviation of 5%. The results underline that the higher level model simulation is faster as it mostly relies on built-in Simulink blocks. It is recommended for functional validation, algorithm parametrization, and test pattern selection. It provides all the benefits of the model-based design paradigm, while it cannot be used for accurate timing analysis, power, and energy measurements. The low level model simulation is much slower since it relies on actual primitive implementation and many wrapper invocations. However, it represents a fundamental step as it allows verifying the functional equivalence between the system-level model implemented through blocks and the system-level model implemented through primitives. Finally, the validation through execution on the target real device allows for accurate timing and power analysis, in which all the techniques at the state of the art for system-level optimization can be applied.

ORB descriptor

In computer vision, visual descriptors or image descriptors represent visual features of image or video contents captured by an input video sensor. One of the most adopted is ORB [START_REF] Rublee | Orb: An efficient alternative to sift or surf[END_REF], which is generally integrated in complex localization and mapping systems (i.e., ORB-SLAM [START_REF] Mur-Artal | Orb-slam: A versatile and accurate monocular slam system[END_REF]). The ORB-SLAM algorithm performs ORB computation at different levels, to detect both fine and coarse features of images. The inputs generally consists of a gray-scale image, the number of such levels, and the number of features to be analysed per level.

We applied the proposed model-based design flow to define the ORB algorithm, which is depicted in Figure 10. For the sake of space, the figure shows the implementation of a single level in the Simulink environment. The data-flow oriented algorithm consists of 5 main steps: The input image is resized according to the scale level with the nearest-neighbors interpolation. The interesting points are detected by using the FAST corner detection algorithm with a specified threshold. Then, the computed keypoints are divided into a regular grid where a pruning is applied to each cell by using an higher threshold. This step is applied only to the cells where the results are non-empty (i.e., there exist at least one keypoint for each cell). The algorithm organizes the keypoints in a quadtree data structure, which allows achieving a uniform sampling of the keypoints in the image, carrying out the final pruning, and obtaining final keypoints, where as a result, an angle is computed (i.e., since FAST detector is not oriented). The algorithm applies a gaussian blur operation to the resized image, in order to improve the descriptor quality and to avoid artifacts that can be introduced by the nearest neighbor interpolation. Finally, the ORB descriptor is computed for each keypoint. The final coordinates of the keypoints are rescaled to the corresponding location in the original image.

Along the design flow, we measured the execution time of the algorithm implementations at different refinement steps, by using the KITTI dataset [START_REF] Geiger | Vision meets robotics: The kitti dataset[END_REF], which is a standard set of benchmarks for SLAM and computer vision applications. We also adopted the ROS interfaces described in 3.3 to receive the video stream (i.e., based on the slave model) and to integrate an external agent that reads the ORB result (i.e., based on the server model). We applied the semi-automatic translation process from Simulink to the final implementation as explained in in section 4.1, targeting an NVIDIA Jetson TX2 embedded board. We observed a slightly reduced execution time for the Simulink low-level model execution with respect to the high-level model despite the wrapper usage. This is due the fact that the algorithm implementation in Simulink required specialized MATLAB code that was not available with Simulink CVT library as native blocks. As a consequence, we developed custom code in MATLAB to meet the requirements, and imported such a code as user-defined Simulink blocks using the level-2 S-functions. As for the model-based design flow, the main focus of the Simulink implementation was to target the functional verification of the embedded application, with little effort on performance optimizations. On the other hand, such user-defined blocks were available in the OpenVX-Vision Works library thorough GPU-accelerated primitives.

Conclusion

This paper presented a methodology to integrate model-based design to OpenVX. It showed how such a design flow allows for fast prototyping of any computer vision algorithm through a Matlab/Simulink model, its parametrization, validation, and automatic synthesis into an equivalent OpenVX code representation. The paper presented the experimental results obtained by applying the proposed methodology for the development of two embedded software. The first implements a digital image stabilization, while the second implements an ORB descriptor for SLAM applications. The applications have been modelled and parametrized through Simulink for different application contexts. In particular, the ORB application has been validated by considering an external typical and dynamic Robotics environment. This has been done through the OpenVX-ROS interface generated with the proposed design flow, which allows co-simulating the OpenVX application with the external application environment (e.g., input streams, concurrent interactive systems, etc.) and, as a consequence, tuning more efficiently the SW parametrization. Both the applications have been automatically synthesized into OpenVX-VisionWorks code for an NVIDIA Jetson TX2 board.

Fig. 1 .

 1 Fig. 1. The embedded vision application design flow: the standard (a), and the extended with the model-based design paradigm (b)

Fig. 2 .

 2 Fig. 2. OpenVX sample application (graph diagram)

Fig. 3 .

 3 Fig. 3. Methodology overview

-Fig. 4 .

 4 Fig. 4. Overview of the Simulink-OpenVX communication

Fig. 5 .

 5 Fig. 5. The OpenVX-ROS communication thorough the server and client models

Fig. 6 .Fig. 7 .

 67 Fig. 6. Skeleton implementation for the (a) server model and the (b) client model

Figure 7

 7 Figure 7 shows the temporal evolution of the OpenVX-ROS communication based on the server model of Figure 6(a).

Fig. 8 .

 8 Fig. 8. Client model time evolution

Fig. 10 .

 10 Fig. 10. ORB design in Simulink

Table 1 .

 1 Table 1 shows, as an example, a representative subset of the mapped entries. Representative subset of the mapping table between Simulink CVT and NVIDIA OpenVX-VisionWorks

	Simulink block		Visionworks primitive	Notes to the developer
	CVT/AnalysisAnd -Enhancement/	vxuCannyEdgeDetector	If Simulink EdgeDetection set
	EdgeDetection			as Canny
	CVT/AnalysisAnd -Enhancement/	vxuSobel3x3	If Simulink EdgeDetection set
	EdgeDetection			as Sobel
	CVT/AnalysisAnd -Enhancement/	vxuConvolve	If filter size different from 3x3
	EdgeDetection
	CVT/Morphological	vxuErode3x3 +
	operation/Opening		vxuDilate3x3
	CVT/Filtering/Median Filter	vxuMedianFilter3x3
	CVT/Filtering/Median Filter	vxuNonLinearFilter	If filter size different from 3x3
	Math Op./Subtract +	vxuAbsoluteDifference
	Math Op./Abs
	CVT/Conversion/Color space con-	vxuColorConvert
	version
	CVT/Statistics/2D Mean CVT/Statistics/2D StandardDev	vxuMeanStdDev	Only mean and standard deviation of
	Simulink/Math operations/Real/-ComplexTo -Imag		the entire image supported Gradient magnitude
	Simulink/Math	operations/Re-	vxuMagnitude	computed through
	al/Imag to Magnitude		complex numbers

  2 . The wrapper invocation (i.e., vx userN ode()) is Listing 1.3. Overview of wrapper for user-defined Simulink block implementations

	vx_userNode () {
	vx_status p r o c e s s i n g O p e n VX ( vx_node node , const vx_reference * parameters ,
	vx_uint32 num )
	{
	// convert data in internal represen tation
	S i m u l i n k B l o c k F u n c t i o n a l i t y () ; // C / C ++ code of the UDB functionality
	return VX_SUCCESS ;
	}
	vx_status v a l i d a t i o n O p e n VX ( vx_node node , const vx_reference parameters [] ,
	vx_uint32 num ,	v x_meta_ format metas []) )
	{
	// insert parameter validation
	return VX_SUCCESS ;
	}
	vx_status s i n g l e S h o t P r o c e s s i n g ( vx_context context , parameters )
	{
	// create graph and execute it
	}
	vx_status r e g i s t e r C u s t o m K e r n e l ( vx_context context )
	{
	vx_status = v xA d dU se rK e rn el ( context , ...) ;// register kernel in context
	return VX_SUCCESS ;
	}
	}

Table 2 .

 2 Experimental results: High-level simulation time in Simulink

	Input
	stream		Feature
			Detection
	Color Converter	Remapping	Compute Homography	Filtering	Warping
			Optical
	Forward edge	Flow			Output
	Delayed edge					stream
		Fig. 9. Digital image stabilization algorithm
	Context	Original input stream	Selected test patterns
		Video real	Model	Frames	Video real	Model	Frames
		time (min)	simulation	(#)	time (min)	simulation	(#)
			time (min)			time (min)
	Indoor	364	492	1.296.278	20.5	30.5	72.112
	Outdoor 192	263	648.644	11.0	13.0	36.935

Table 3 .

 3 reports the characteristics of the input streams (columns Video real time and #Frames) and the time spent for simulating the high-level model on such video streams in Simulink (Model simulation time). Starting from the original video streams, we extrapolated a subset of test patterns, which consist of the minimal selection of video streams necessary to validate the model correctness by adopting the Smith et al. validation met-Experimental results: Comparison of the simulation time spent to validate the software application at different levels of the design flow. The board level validation time refers to real execution time on the target board

	Validation level	Sim./Exec. time (min)
		Indoor	Outdoor
	Simulink High-Level model	30.5	13.0
	Simulink Low Level model	59.0	26.5
	Software application on target embedded system device 20.5	11.0

Table 4 .

 4 Experimental results: Comparison of the simulation time spent to validate the software application at different levels of the design flow. The board level validation time refers to real execution time on the target board including the ROS communication overheadTable 4 reports the execution time we obtained by running the application on the input sequence 11 of the KITTI dataset at different refinement levels.

	Validation level	Sim./Exec. time (seconds)
	Simulink High-Level model	804.0
	Simulink Low Level model	762.0
	Software application on target embed-	30.0
	ded system device (with accelerators)
	and ROS interface overhead

In this work, we selected the Simulink Computer Vision toolbox (CVT), as it represents the most widespread and used toolbox in the computer vision community. The methodology, however, is general and can be extended to other Simulink toolboxes.

www.khronos.org/registry/OpenVX/specs/1.0/html/da/d83/group__group_ _user__kernels.html


domain: spi phys

text:
Ludivine Emeric

Claire Deeb

Fabrice Pardo

Jean-Luc Pelouard
email: jean-luc.pelouard@c2n.upsaclay.fr

Critical coupling and extreme confinement in nanogap antennas

Nanogap antennas are compelling structures for squeezing light into ultrasmall volumes. However, when gaps are shrunk down to the nanometer scale, mode losses dramatically increase. In this Letter, we report conditions of critical coupling, between arrays of nanogap resonant metal-insulator-metal (MIM) antennas and free-space. Adapting the antennas density, critical coupling is achievable for any thickness of insulator, from 100 nm down to 0.1 nm. The fundamental optical mode can be described as continuous transitions through three types of modes: perfect MIM mode, coupling between MIM mode and surface plasmon polariton (SPP), and gap plasmon mode. We found that the space between adjacent antennas is an essential parameter to perform critical coupling for thinner gaps. These results pave the way towards understanding extreme confinement in nanogap antennas structures such as MIM or nanoparticle arrays.

Metallic nanoantennas have been widely used to confine electromagnetic field at the subwavelength scale [1][2][3][4]. Their ability to support strongly localized surface plasmons make them excellent systems to design perfect absorbers.

The emergence of reliable fabrication techniques to produce well-controlled gaps have prompted the development of narrow gap antennas for achieving even higher field confinement and enhancing non-linear effects such as optical rectification for photodetection [5], biosensing [6], nonlinear harmonic generation [7,8], surface enhanced spectroscopy [9], and enhanced spontaneous emission [10]. Tunneling junction-assisted-antennas were also shown to support light emission [11][12][13]. With this variety of application areas, it is necessary to have a strong knowledge of the optical properties of nanogap antennas, particularly when their size is scaled down to deep subwavelength dimensions.

Under critical coupling, the incident energy is totally absorbed within the antennas. Critical coupling is characterized by a perfect match between the non-radiative and radiative damping rates, according to the temporal coupled mode theory [14,15]. At a fixed wavelength and in order to maintain critical coupling, decreasing the insulator thickness implies decreasing the width of the antennas, thus changing both radiative and nonradiative damping rates. Under-coupling and over-coupling cause non-perfect absorption, due to two opposite behaviors (Fig. 1(a)). An under-coupled system is characterized by a narrower resonance than that at critical coupling: the system is restrained by the limited amount of incident energy that can enter the resonator. Whereas an over-coupled structure has a broader resonance: the system is saturated and cannot absorb all the incoming light. The optical properties of antennas are therefore highly dependent on their dimensions [15]. Shrinking the gap size results in a higher confinement, but also a large part of the field penetrates into the metal, leading to higher losses. As a result, balance between radiative and non-radiative processes has to be reconsidered for each gap thickness. Moreover, conditions for achieving critical coupling in sub-nanometer gap antennas remain unknown.

In this Letter, we propose to study critical coupling of the fundamental mode in MIM nanoantennas as a function of the gap size. The studied antennas consist of a semi-infinite gold film, a continuous SiO 2 layer of thickness h gap , and a periodic grating of gold stripes. The width of the stripes is w, the grating periodicity is p (Fig. 1(a)), and the thickness of the gold stripes is h Au . The cavity under one stripe constitutes a Fabry-Perot (FP) resonator, where two plasmon waves propagate: a plasmon wave at the interface film/gap and another plasmon wave at the interface gap/stripe. These plasmon waves are coupled together [16] and the resulting gap mode travels along the stripe width [2]. The gap mode can be excited in normal incidence by a TM-polarized plane wave (Fig. 1(a)). The fundamental MIM mode is characterized by confined electromagnetic fields within the gap, with an enhanced electric field at the cavity corners and a maximum of magnetic field at the center of the cavity. The reflectivity spectra were numerically computed using RMCA technique (Rigorous Maxwell formulation and polynomial Constitutive equations Approximation) [17,18]. This approach is defined by a decomposition of the field on a basis of vertically propagating modes (along z-axis) and relies on a scattering matrices algorithm. In this study, we fixed the resonance wavelength l at 3 µm and the thickness of the gold stripes at 50 nm. The refractive index of silica was equal 1.42 [19], and the gold permittivity was set On the right, the width of the remaining space between two neighboring antennas (p w in red).

to 416.60 + 47.51i, in accordance with recent studies [20,21]. Then, we determined the conditions leading to critical coupling, which corresponds to a total absorption, i.e. a reflectivity close to zero, through calculating the width of the stripes and the periodicity of the grating (w and p) for gap thicknesses ranging from 0.1 to 100 nm. We found that critical coupling was achievable for every thickness of gap. The critical coupling position was obtained for a normalized reflected intensity below 10 8 . The results of these calculations are shown in Fig. 1(b). Three domains are identified. Perfect MIM mode extends from 2 to 40 nm. For these thicknesses, the stripe width perfectly satisfies the phase-matching condition (PMC) [22] in a FP resonator:

w = l 2 n gap eff ⇥ 1 j p ⇤
, with n gap eff the effective index of the mode and j the reflection phase at a single corner of the cavity. The grating periodicity and the gap thickness follow a linear condition.

For gap thicker than 40 nm, the grating periodicity saturates before 3 µm. It results from the coupling between MIM mode and SPP mode propagating at the metal/SiO 2 interface. If p were greater than l, diffraction modes would appear with electromagnetic fields radiating along new directions.

Below 2 nm, the space between two adjacent stripes is in the same order of magnitude of the gap thickness, facilitating the coupling between neighboring antennas and modifying the properties of the MIM mode.

To investigate the origin of critical coupling in gap antennas with gaps ranging from 0.1 to 100 nm, we develop an analytical model that accounts for the space between two gold stripes in nanogap antennas. It has been previously demonstrated that one horizontal MIM resonator is equivalent to two vertical slits [23]: the slits can be considered separately and the field can be seen as being totally reflected at the middle of the cavity. However, for sub-nanometer gap antennas, the system can no longer be seen as two independent slits because the space between two stripes is reduced for maintaining critical coupling. This space is integrated into our model as a third slit of air, where the effective index becomes highly greater than 1.

In this analytical model, we determine the exchanges between the three slits (Fig. 2(a)). First, the incident magnetic field, set to unity, can either reflect onto the grating (r 11 ) or enter into the vertical slit (t 21 ). After propagation within the slit (P 2 ), the field can enter under one or the other half-stripe (both t 32 ), or it can get reflected on top of the silica layer (r 2 ). When the field is transmitted toward the silica layer, it propagates (P 3 ), reflects (r 3 ), and propagates again (P 3 , nodes B and C) to exit toward the other half-cavity (t 33 ), the vertical slit (t 23 ), or reflect again (r 33 ), and so on. When the field gets reflected on top of the SiO 2 layer, it propagates backwards until reaching the interface slit/free space (node A), where it can either exit toward free space (t 12 ) or gets reflected on the top of the slit (r 22 ).

This intertwined system can be decomposed into three independent configurations: the top of the vertical air slit (Fig. 2(b)), the interface air/silica with the silica layer at the bottom of the slit (Fig. 2(c)), and the two half-cavities (Fig. 2(d)). Periodic boundary condition is here equivalent to a total internal reflection at the end of a half-cavity, leading to r 3 = 1. The propagators inside the vertical slit and one half-stripe depend on the effective index of their respective medium: P 2 = exp ik 0 n air eff h Au and

P 3 = exp ⇣ ik 0 n gap eff w/2
⌘ , with k 0 = 2p/l. All quantities were deduced from RMCA modal method into these three configurations. r 11 , r 22 , t 12 , t 21 and n air eff were deduced from Fig. 2(b) configuration. r 2 and t 32 were deduced from Fig. 2(c). t 33 , r 33 , t 23 and n gap eff were deduced from Fig. 2(d). The effective indices are deduced from the fundamental eigenmode inside each slab. Reflectivities and transmissivities are the related Smatrices elements, except the two orthogonal transmissions t 23 and t 32 : independent slits were considered for determining the projection of the combination of all the vertical modes onto the horizontal propagating mode.

At steady-state, the total reflectivity can be written as:

r tot = r 11 + At 12 (1)
with A, the amplitude of the magnetic field at node A in Fig. 2, satisfying the following equations:

A = P 2 2 r 2 t
with B and C, the amplitude of the magnetic field at nodes B and C, respectively. The total response does not depend on the normalization of the eigenvectors in each single layer, however transmissivities t 12 and t 21 do. In order to get rid of this effect, we factorize the equations by their product t 12 ⇥ t 21 . For each gap thickness, a small deviation of t 23 ⇥ t 32 has been considered to remove any uncertainty due to the arbitrary origin taken for their phase and permitted to lead to a total reflectivity converging toward the value deduced above. We conducted the calculation until an absolute error of 10 10 was reached.

As we previously assumed, the studied system presents three domains as a function of the gap thickness, illustrated by the evolution of the Fresnel's coefficient and effective indices (Fig. 3). As the gap thickness decreases, the modulus of t 12 ⇥ t 21 decreases, indicating the necessity to reduce the incident field (i.e. the period) in order to maintain the critical coupling. 1 Furthermore, the real part of the effective index of the gap mode (<(n gap eff )) increases due to a stronger coupling between the two surface plasmons. The corresponding imaginary part increases as well, demonstrating a stronger absorption.

For gap thicknesses between 2 to 40 nm, we observe the typical behavior of a FP resonator illustrated by a high reflection on both sides of the cavity (r 33 is real negative and lower than -0.8) and a week coupling between adjacent antennas (|t 33 | < 0.3).

Gaps thicker than 40 nm still show a noticeable MIM mode as <(n gap eff ) in the SiO 2 gap remains above the bulk refractive index. The emergence of an SPP mode implies fewer exchange between the vertical slit and the gap layer, counterbalanced by a slight increase of |t 33 | and decrease of |r 33 |. In this domain, the vertical slit does not show any vertical resonance (<(n air eff ) = 1). For gaps thinner than 2 nm, a resonance appears in the vertical air slit, as confirmed by the rise of the effective index of 1 The turning points observed on r 33 and t 33 for h gap = 33 nm are due to a FP mode in the air slit between two adjacent antennas (p w = 1500 nm). This mode is not excited at normal incidence but it introduces a noteworthy high transmissivity between antennas at resonance. Confinement inside the total system benefits from a much stronger coupling between neighboring antennas, assisted by the dual role of the vertical slit: giving access to the half-cavities and storing a little energy by resonating. The lower exchange between free-space and the structure is compensated by a higher storage and stronger exchanges inside the system: the vertical slit gives more flow to both SiO 2 half-cavities, which also exchange with one another.

Re(n

Indeed, the maps of the electric field components (Fig. 4) for a gap of 1 nm, show a strong field inside the gap in agreement with MIM mode properties, but also exhibit a large field in the vertical space between gold stripes. It reveals the coupling between neighboring antennas, characterized by a nearly constant field inside the vertical slit, close to the value of the field within the SiO 2 gap.

Besides, below 0.5 nm the width is lower than twice the penetration depth: at this scale, evanescent modes from each side of a stripe couple together within the metal, leading to an attenuated confinement and a higher absorption inside one single nanoresonator [22]. Antennas have to be brought closer to counterbalance this effect. A side effect of this coupling is a lowering of the effective index inside the slit compared with independent slits (i.e. larger stripes), resulting in a saturation of the top reflectivity r 11 around 0.8.

It is worth noticing that for the thinnest gaps (i.e. near 0.1 nm), gap modes may no longer exist: the imaginary part of the index of the effective mode becomes close to its real part. Furthermore, the study performed in this paper is fully classical, and non-local and tunneling effects are not included [24,25].

Although quantum plasmonics effects have not been accounted for, this study shows interesting aspects for designing MIM nanostructures at critical coupling and understanding the role of each element in the global response. This model can be applied to higher wavelengths, thus extending its validity.

For small gap thicknesses (< 2 nm), the structured gold layer needs to be very dense to maintain critical coupling (p w < 20 nm and w < 190 nm, leading to a filling factor greater than 90%). This design is hard to fabricate and the optical response can be very sensitive to fluctuations in the antennas dimensions. This fabrication challenge can be overcome through diverse techniques, such as Atomic Layer Deposition [26,27], Focused Ion Beam milling [28], or assembly of cubic nanoparticles [29]. Two-dimensional array can also be considered, introducing modes along both x-and y-axes. Despite the more complex situation due to mode coupling, the changes in results should be minimal under polarized light.

Critical coupling can also be reached through changing other parameters, such as varying the thickness of the gold stripes or etching the insulator layer at the bottom of the vertical slit [30]. Furthermore, it can be convenient to introduce an adhesion layer between gold and silica to improve the adhesion of silica on gold.

To conclude, critically coupled MIM nanoresonators can be obtained for any thickness of insulating gap. Three ranges have been identified: the regular MIM mode, its coupling with SPP for thickness higher than 40 nm and gap plasmon mode for thickness below 2 nm. This relation between the width, the periodicity and the thickness of the gap gives simple rules to design a perfect absorber at a fixed wavelength. An analytical description of the system discussed the coupling between two adjacent half-MIM and revealed the central role of the vertical slit for thinner gaps. This system can be described by a unique system of equations, valid for any thickness from 0.1 to 100 nm.

Fig. 1 .

 1 Fig. 1. (a) 3D Schematic view of MIM antennas, consisting of a gold mirror, a SiO 2 layer, and a periodic grating of gold stripes. Typical reflectivity spectra for three cases: under-coupling, critical coupling, and over-coupling. (b) On the left, optimal periodicity p (blue solid line) and width w (blue dashed line) at critical coupling as a function of the gap thickness h gap , superimposed with a linear fit of p (p = 61.1 h gap + 91.7, black solid line) and the width satisfying PMC (black dashed line). On the right, the width of the remaining space between two neighboring antennas (p w in red).

Fig. 2 .

 2 Fig. 2. (a) Schematic cross section of two half-antennas. The parameters of the S-matrices are presented on the figure. The system is excited by a magnetic field amplitude set to unity and polarized along y. The system is decomposed into three configurations to extract the parameters of the S-matrices: (b) top of the vertical air slit, (c) bottom of the vertical air slit, and (d) the two half-cavities. The amplitude of the magnetic field is extracted at three nodes. A: top of the vertical air slit, B: border of one half-cavity, and C: border of the other half-cavity.

Fig. 3 .

 3 Fig. 3. Modulus (a) and argument (b) of Fresnel coefficients used in our model. All parameters are presented as a function of h gap in a logarithmic scale. The space p w between antennas is shown as an indication for h gap = 0.1, 1, 10 and 100 nm. Note that |r 11 | ' |r 22 |. Real (c) and imaginary (d) parts of the effective indices in the air slit and SiO 2 gap.

Fig. 4 .

 4 Fig. 4. Maps of the intensity of the electric field in logarithmic scale, normalized by the illuminating field E 0 for a gap of 1 nm: (a) z-component and (b) x-component. This structure is at critical coupling with p = 129 nm and w = 124.3 nm.

  21 + P 2 2 r 2 r 22 A + P 2 t 23 B + P 2 t 23 C B = P 2 3 r 3 (t 32 P 2 t 21 + t 32 P 2 r 22 A + r 33 B + t 33 C) C = P 2 3 r 3 (t 32 P 2 t 21 + t 32 P 2 r 22 A + t 33 B + r 33 C)

FUNDING INFORMATION

This work was partially supported by the Agence Nationale de la Recherche under the project HORUS ANR-16-CE39-0007-03.