Haptic Optical Tweezers with 3D High-Speed Tracking

Keywords: Asynchronous Event-Based Tracking, Haptic Feedback, Force Sensor, Optical Tweezers, Micromanipulation, Biological application Mapping Algorithm . . . . . . . . . .

Micromanipulation has a great potential to revolutionize the biological research and medical care. At small scales, microrobots can perform medical tasks with minimally invasive, and explore life at a fundamental level. Optical Tweezers are one of the most popular techniques for biological manipulation. The small-batch production which demands high flexibilities mainly relies on teleoperation process. However, the limited level of intuitiveness makes it more and more difficult to effectively conduct the manipulation and exploration tasks in the complex microworld. Under such circumstances, pioneer researchers have proposed to incorporate haptics into the control loop of OTs system, which aims to handle the micromanipulation tasks in a more flexible and effective way. However, the solution is not yet complete, and there are two main challenges to resolve in this thesis:

• 3D force detection, which should be accurate, fast, and robust in large enough working space;

• High-speed up to 1 kHz force feedback, which is indispensable to allow a faithful tactile sensation and to ensure system stability.

In optical tweezers micromanipulation, vision is a sound candidate for force estimation since the position-force model is well established. However, the 1 kHz tracking is beyond the speed of the conventional processing methods. The emerging discipline of biomorphic engineering aiming to integrate the behaviors of livings into large-scale computer hardware or software breaks the bottleneck. The Asynchronous Time-Based Image Sensor (ATIS) is the latest generation of neuromorphic silicon retina prototype which records only scene contrast changes in the form of a stream of events. This property excludes the redundant background and allows high-speed motion detection and processing. The event-based vision has thus been applied to address the requirement of 3D high-speed force feedback.

The result shows that the first 3D high-speed haptic optical tweezers for biological application have been achieved. The optical realization and event-based tracking algorithms for 3D high-speed force detection have been developed and validated. Reproducible exploration of the 3D biological surface has been demonstrated for the first time. As a powerful 3D high-speed force sensor, the developed optical tweezers system poses significant potential for various applications.

List of Symbols

C t : [X t , Y t , R t ] t ∈ R
The detected circle during particle detection at time t. X t , Y t are the x-, y-position of the circle center. R t is the radius of the circle.

All the parameters are in pixels in the camera coordinate.

D (X,Y ),(x,y) X, Y, x, y ∈ R Euclidean distance between the 2D coordinate (X, Y ) and (x, y).

E t : [x t , y t , t t ] t ∈ R
The event collected at the time t. x t , y t , t t are the x-, y-position in pixels and the time stamp of the event respectively.

F opt

The 3D optical force exerted on the trapped probe.

k B The Boltzmann constant.

K : [K x , K y , K z ]
The stiffness of the optical tweezers. K x , K y , K z are the optical stiffness in x-, y-, and z-direction respectively.

P laser

The center position of the optical trap in the world coordinate.

P probe

The center position of the trapped probe in the world coordinate.

ROI(C t )

The Region of the Interest of the detected circle during particle detection at time t.

ROI(R 1 t ), ROI(R 2 t )
The Region of the Interest of the detected ring during particle detection at time t. R 1 t , R 2 t represent the outer and inner circle of the ring respectively.

R t : [X t , Y t , R 1 t , R 2 t ] t ∈ R
The detected ring during particle detection at time t. X t , Y t are the x-, y-coordinate of the ring center respectively. R 1 t , R 2 t are the outer and inner radius of the ring respectively. All the parameters are in pixels in the camera coordinate.

T

The absolute temperature.

General Introduction

Micromanipulation gives access to unreachable small scales with robotic techniques.

It brings revolutionary potential to many fields such as biological research and health care [Nelson 2010]. As the dimension of robot decreases, vast amount of in vivo or in vitro applications become possible, ranging from minimally invasive intervention to exploration of fundamental biological phenomenon [Fazal 2011]. Among a variety of microrobotic techniques, Optical Tweezers (OTs) are considered to be one of the most suitable techniques for biological characterizations and manipulation tasks [Moffitt 2008]. OTs are achieved from the attractive force provided by tightly focused laser beam. By controlling the position and the number of laser beam, the trapped microobjects can be manipulated in a non-contact way and implemented into high accuracy tasks. According to the OTs force model [Ashkin 1987], the optical trap is approximated as a linear spring within certain scale. The force exerted on the optically trapped object is linearly proportional to its displacement from the equilibrium position. Therefore, in order to detect force using the probe, the major problem is to resolve its displacement using tracking methods. To date, several manipulation strategies are developed for OTs system. They are mainly divided into three categories: automatic [Sun 2013], teleoperated [Bowman 2011b], and haptic manipulation [START_REF] Pacoret | [END_REF]]. This thesis mainly focuses on the development of haptic OTs manipulation. Haptic techniques allow remote control of a micromanipulator in an intuitive manner: the operator's hand directs the microprobe motion with the haptic device, and simultaneously perceives the interaction force from the microenvironment produced by the device's motors. Compared to the automatic approach, it is more flexible since the human's intelligence is integrated into the control loop, leading to more dexterous motions and manipulations in small-batch biological tasks. Compared to the teleoperation, haptic manipulation is more intuitive since not only the visual information, but also the additional force feedback is provided to users to facilitate the manipulation. Therefore, the multi-channel (vision, tactile or kinesthesic) information enables users to better understand the micro-scale phenomenon and be more effective to achieve desired results during manipulation tasks. On the basis of these advantages, we believe that the haptic OTs system will greatly benefit biological research through on-line manipulation.

As haptic OTs have a great potential for increasing the operations success rates and decreasing the required time, it might become a standard tool for biological laboratories or state-the-art diagnosis. In addition, they can also serve pedagogical purposes to display the microworld. Pioneer achievements have been demonstrated by the Ph.D. thesis of Pacoret and Ni at the Institute of Intelligent Systems and Robotics (ISIR). They demonstrated the first application of touching the micro-surface with 2D haptic feedback. However, the previous works is inadequate for addressing complex manipulation, as presented in biological samples. Two crucial challenges remain on the path:

• 3D non-contact force sensing is still underdeveloped in the literature of microrobotics. To achieve the 3D force feedback in OTs, the 3D position tracking method is required. It should be accurate and robust in complex environment.

• 1 kHz low latency control rates are essential for obtaining faithful tactile sensation and stable haptic control loops [Verrillo 1963]. Despite this interest, the existing force detection methods hardly satisfy this required performance.

The core problem is a lack of the high-speed visual tracking methods.

These two challenges will be detailed in Chapter 1 as they define the requirements for the novel tracking approach. As highlighted by the previous study, the biomimetic event-based vision is a promising technique to address these challenges. Therefore, it will be further developed and be applied in this thesis for 3D highspeed haptic feedback rendering. The basic idea of this vision technique is to mimic the mechanism of retinas of animal/human eyes. Instead of providing full frames of images, this biomimetic sensor performs a basic level of processing to the captured images. It filters out the static background and provides only the dynamics in the form of "spikes". These spikes (coined as "events" throughout the thesis) indicate spatial and temporal light contrast changes. They are asynchronous and continuous, like the electrical potentials transmitted on nerves and between neurons in living beings. The latest generation of this event-based sensors will be integrated into our system. However, in order to achieve the 3D high-speed haptic OTs system, several aspects of work must be studied.

• In the domain of 3D force detection, both the optical realization and the tracking algorithm should be developed in parallel. The Chapter 2 and Chapter 3 will work on the hardware and algorithm level respectively in order to obtain the 3D fast and robust force measurement.

• Given novel asynchronous camera with better spatial and temporal resolution, the need arises for specially tailored tracking algorithm above 1 kHz processing rates. This problem will be tackled in Chapter 3.

• As a proof of concept, exploration and force interaction in a real biological environment will be demonstrated in Chapter 4.

This new approach to realize the OTs system with 3D high-speed force feedback will lead to a considerable amount of attractive applications in the future. For example, biological tasks such as unfolding an RNA hairpin kinesin described in [Moffitt 2008], or twisting DNA in [Fazal 2011], might be performed by the users in a more intuitive manner with the sensation of molecular force. 3D high-speed force measurement empowers the OTs as a versatile on-line force sensor. That makes it possible to characterize the biological objects at the first expedition, or even explore a labyrinth-like biological environment like cell actin fibers or blood vessels. Micromanipulation gives access to unreachable tiny scales with robotic techniques. For flexible and intuitive control, human users need both vision and haptic feedbacks. Although various microscopy techniques have been developed for providing vision information to users, few researchers have addressed the problem of haptic feedback during micromanipulation tasks. The aim of our work is thus to develop a manipulation system with 3D faithful haptic feedback. This chapter begins by examining the manipulation techniques. Then the existing methods for 3D faithful haptic feedback rendering will be investigated.

Haptic Optical Tweezers

The micromanipulation have a great potential to revolutionize the biological research and medical care [Nelson 2010]. At small scales, microrobots can perform tasks that cannot be handled otherwise, such as exploring life at a fundamental level ranging from cellular to the more subtle molecules [Fazal 2011] [Kulik 2015].

Among numerous of manipulation methods, Optical Tweezers (OTs) are viewed as the promising solutions due to their advantages over other techniques in biological applications, as will be explained in details in the next part.

Regarding small-batch biological tasks which demand high flexibility of the microrobots, existing automatic control solutions are still far from addressing all the needs in industrial and research level. In such case, including human's supervision and expertise in the control loop is indispensable. However, most of existing research works are focused on teleoperated OTs. That means during the tasks, only visual channel feedback is provided to users. In contrast to direct manual manipulation or touching where various channel information is available to help users understand the situation, teleoperation with only visual feedback is less intuitive, and less effective for achieving complex tasks.

Under such circumstances, several researchers started to incorporate haptics into the control loop of OTs system to enhance the intuitiveness of the micromanipulation. The research work [START_REF] Pacoret | [END_REF]] established the concept of haptic OTs, and [Ni 2013b] integrated dynamic vision sensor (DVS) to OTs system and achieved simple touch demonstration in 2D. These pioneer achievements displayed the potential of haptic OTs. However, as will be explained, they are still far from being applied in complex biological applications. The critical gap lies in the lack of both 3D and high-speed haptic feedback. Achieving any of them is already difficult, let alone the combination of both! In this chapter, such challenging problem will be solved step by step. It motivates my thesis to bridge the gap to achieve the first 3D high-speed haptic OTs system, which is dedicated to biological applications.

Furthermore, one significant potential that arises from achieving 3D high-speed force sensing is to exploit OTs as a powerful Force Sensor in microenvironment. Compared to conventional force sensing techniques in microworld, this sensing method has much higher resolution. Besides, it does not need complex design and construction of the microactuator [Lu 2006]. This means OTs will no longer be considered as merely a micromanipulation tool, but can be extended to plenty of applications thanks to its 3D high-speed force sensing capability, such as on-line characterization of microobject features, automated position control using force signals, direct force control, etc. In the last chapter of this thesis, a preliminary development of motion planning has been conducted which uses OTs as force sensor in a complex environment.

In this section, the OTs' principle and their advantages over other techniques will be firstly explained, followed by the operation strategies of OTs in biological applications. At last, the objectives of this thesis will be described.

Working Principle and Properties

The first successful demonstration of an OTs system was presented by Ashkin et al. [Ashkin 1987]. The principle of OTs is that: an optical trap is created by focusing a laser with a high numerical aperture (NA) microscope objective. The tightly focused laser beam provides force near the beam foci. The microobjects ranging from several F opt P laser -P probe Model F opt = K × ( P laser -P probe ) K (K x ,K y ,K z )

Figure 1.1: A microsphere is trapped to the center of the optical tweezers. The 3D optical force F opt exerted on the object depends linearly on its displacement from the trap center P laser -P probe as a spring model with stiffness K(K x , K y , K z ).

nanometers up to tens of micrometers can be trapped in stable configurations by this force. When the trapped objects are laterally or axially displaced from the equilibrium position, the direction or divergent angle of laser passing through it will be changed correspondingly. That means the momentum of the light is changed by the objects, and a reaction force is exerted on the objects correspondingly. Thus the force acts to pull it back towards the equilibrium point [Ashkin 1992].

During micromanipulation and quantitative force measurement, silicon or polystyrene microsphere are often used as a force probe to indirectly manipulate and sensing the force. That is because directly exposure cells to a laser beam during OT-based operation may inflict photo damage to the cells. What's more, determining the optical force applied on arbitrary objects with reasonable accuracy is a huge challenge. It is known that the optical stiffness depend on several factors e.g. properties of the material, the refractive index of both object and environment, the sizes and shapes of trapped objects [START_REF] Jun | Calibration of optical tweezers for in vivo force measurements: how do different approaches compare?[END_REF]] etc. Therefore, when trapping arbitrary biological objects, the not well known material properties, and the deformable shape and size pose great difficulties for calculating the optical stiffness, and thus the optical forces. Instead, by using artificial microbeads which are uniform in material and shape, the optical force applied on the microbead can be obtained easily. Normally, the optical force is described by a spring model with a trap stiffness K : [K x , K y , K z ]: [Ashkin 1987] F opt = K × (P laser -P probe )

Where P laser and P probe represent the laser and the probe position respectively. F opt is the 3D optical force. The stiffness of the trap K is a 3D vector. K x , K y , K z represent the stiffness in x-, y-, and z-direction respectively. Therefore, the optical force is obtained by measuring the particle's displacement from the equilibrium trap position as is illustrated in Fig. 1.1. Many studies show more profound mathematical and physical analysis on the principle of generating the optical trap by the focused laser [Bowman 2013]. Several methods are developed to measure the trap stiffness, and some of the most popular ones are the power spectrum method, equipartition drag force method, optical potential analysis etc. More details can be found in [Neuman 2004]. Normally, the stiffness varies at different situations depending on the optical gradient, laser power and so on, at the order of pN/nm for typical OTs.

Compared to other micromanipulation techniques presented in the past decades, for example, Atomic-force Microscopy (AFM) [Binnig 1986], Magnetic Microrobots, and Scanning Electron Microscopy [START_REF] Todokoro | Scanning electron microscope[END_REF]] etc., the OTs are the most suitable technique in our case for biological applications. For SEM, the specimen chamber requires high vacuum, thus the sample is normally required to be completely dry, which will damage the living cells and tissues [Williams 1996]. A SEM based manipulation systems is shown in Fig. 1.2 (a). For AFM, the main drawbacks stem from the large size and relatively high stiffness of the cantilevers. Thus the frequencies range of the detected force is reduced, particularly in aqueous conditions [Neuman 2008]. For magnetic microrobots, multiple target control is very difficult using the uniform magnetic field. Because the applied magnetic field affects the entire workspace, and makes it hard to address individual magnets. An examples of magnetic microrobots system is shown in Fig. 1.2 (b). By comparison, OTs have many advantages in biological applications as specified:

1. non-contact. Since no actuator is in contact with the target during manipulation, thus physical damage and contamination to biological objects are avoided. Combined with infrared optical trap, which is poorly absorbed by most living matter, OTs are particularly suitable for manipulating biological objects (e.g. cell transportation [START_REF] Wu | [END_REF]]) with less damages;

2. High accuracy. OT is able to exert forces of 0.1 -100 pN in 3D working space with measure displacements over 0.1 -10 5 nanometers. This is the ideal range for measuring the forces of biological and macromolecular responses [Grier 2003]. Particles size ranging from nanometers to microns can be manipulated. These prop- erties make them extremely well suited for the measurement of force and motion with exquisite precision. An example is displayed in Fig. 1.3 (a) , where the kinesin motor motion and RNA molecule unfolding process are measured by OTs [Fazal 2011].

3. Multi-trap. The development of the holographic OTs enables the manipulation of several (up to hundred) objects simultaneously with complete individual control over each of the trap in 3D. The multi-trap technique has largely increased the popularity of OTs. Many applications are realized benefit to this capability e.g. cellular assembly [Kirkham 2015] as shown in Fig.1.3 (b).

Based on these arguments, the OTs were chosen as our manipulation technique because it is one of the most practical ways to develop a versatile manipulation system for biological applications.

Manipulation Strategies

Nowadays, there are three manipulation strategies for OTs: automatic, teleoperated and haptic manipulation.

The automatic manipulation are developed rapidly in the past decades. e.g. Sun and Chen developed an automatic arraying method to place groups of cells into a predefined array [START_REF] Sun | Moving groups of microparticles into array with a robot-tweezers manipulation system[END_REF]]. A snapshot of the arraying experiment is shown in Fig. 1.4 (a). Chowdhury et al. introduced an automatic control and planning approach for indirectly manipulating cells with silica beads arranged gripper formations [Chowdhury 2014]. An picture is shown in Fig. 1.4 (b). However, the current development of automatic OTs is still limited to some specific tasks and conditions. For some tasks, such as those arise in small-batch productions, the complexity and flexibility can not yet be satisfied by the automatic strategy.

Under such situation, teleoperated manipulation is developed for conducting complex tasks by integrating the human's intelligence and expertise into the microrobot control loop. JPK instrument has commercialized the teleoperated OTs by using a joystick to control the optical traps. Universities of Glasgow and Bristol have developed a multi-touch-based application for manipulating OTs using commercial tablet [Bowman 2011b], as displayed in Fig. 1.5 (a). This technique provides a simpler way to control the position of optical traps. However, during the manipulation, the users don't have intuitive force perceptions. With only visual channel feedback, tasks are hardly performed effectively. For example, trapped objects are easy to be lost, damage may be caused to both tools and targets [Bolopion 2013].

In an attempt to enhance the intuitiveness and effectiveness of the manipulation, researchers proposed to provide haptic channel feedback to users. Incorporating haptics into OTs recreates the sense of touch to users when interacting with microtarget. With the help of haptic feedback, human's hands are extended to the inaccessible microscale environment to realize flexible manipulation tasks on-line. So that we could be able to implement tasks more effectively with improved successful rate and reduced force damage to biological samples. Haptic feedback is especially useful when the camera view of scene can't provide sufficient information(e.g. at low contrast or curved places). In [START_REF] Bukusoglu | [END_REF]], haptic device is coupled with OTs as a virtual guide to help users to assemble microspheres. This research proved that haptic feedback significantly improves the user performance by reducing the task completion time, the number of undesired collisions, and the positional errors during binding. Pacoret et al. reported a haptic OTs system which used the trapped probe as a force sensor and allowed the users to perceive the Brownian motion and the forces during manipulation [START_REF] Pacoret | [END_REF]]. Ni et al. proposed to integrate the dynamic vision sensor (DVS) to OTs system and achieved 2D highspeed force feedback [Ni 2013b]. They realized a task of touching the non-planar target surface using microprobe for the first time. A photo of a user controlling the optical trap through haptic device is shown in Fig. 1.5 (b). However, the previous developments have failed to provide 3D force feedback and their systems are not yet compatible with biological requirements. Furthermore, the realized force sensation was not faithful enough due to limitations of computational frequency in the non-real-time system and the low resolution of vision sensor. These drawbacks lead to the motivation of this thesis.

Objective

Based upon the previous developments, the main objective of this thesis is to develop the first versatile 3D real-time high-speed haptic OTs system for biological manipulation. Compared to the previous developments, the workspace must be significantly extended. Particularly in depth (z-axis), it must allow the complete range where OTs is able to traverse. Asynchronous camera DVS was used in the previous touch demonstration in 2D. It paves the way for high-speed application, but it suffers from weakness such as low resolution. As will be explained in this thesis, a novel generation of asynchronous camera, ATIS, with higher resolution and high-speed capability, is applied in our haptic OTs system. The entire solution will be deployed in a hard real-time framework, so as to guarantee 1 kHz control-loop rate in a deterministic manner. Finally, we aim to apply the haptic OTs in a biological application to evaluate the manipulation intuitiveness and effectiveness. The key contribution of this thesis is to establish biologically-compatible instruments and to develop methods to enable both 3D and high-speed haptic feedback in a 1 kHz real-time framework. Developing such a system is complex and involves numerous aspects of considerations. Among all those factors, the two key points that acquire extensive investigation are the 3D force sensing and high-speed haptic feedback.

The first key point is the 3D force detection. Many force detection are realized by installing the dedicated force sensors inside the microtools, or designing special microactuators [START_REF] Beyeler | [END_REF]]. These methods are adaptable to many environments. While they significantly increase the complexity and the cost for fabricating tools. Besides, most of them still lack force measurement capabilities in full 3D space [START_REF] Andersen | [END_REF]. Two examples of force sensing microactuators are shown in Fig. 1.6. An alternative solution to resolve such pitfall is to use vision to provide force information. Visual tracking techniques are widely used in OTs system as well as the other micromanipulation systems due to its convenience [Bowman 2013]. By measuring the displacement or deformation of microactuator, microforce can be obtained [START_REF] Chawda | [END_REF]]. However, it is well known that the microscopy only provides images of the cross-sectional planes with a small depth of field. How to detect the probe's position in 3D space during micromanipulation is still a challenging problem.

The second key point is the real-time high-speed haptic feedback. As discussed in [Verrillo 1963] and [Berthoz 2000], the human sense of touch is over a significant band of frequencies reaching one kHz. In order to provide faithful force to users, the force detection ought to be more than 1 kHz. Or else the information will be lost or tainted. Additionally, the haptic manipulation system is extremely sensitive to delays due to the huge scaling factors that convert pN optical force to Newton. Therefore, for achieving faithful haptic feedback, as well as maintaining the stable sensation, the control loop should be more than 1 kHz with low latency [Bolopion 2013].

In summary, this section outlines the current development of OTs based micro-manipulation and our objectives: the first 3D high-speed haptic OTs for complex biological tasks. To realize our objective, we are mainly facing two challenges: 3D force detection and high-speed (kHz) haptic feedback during manipulations. In the following content, the existing methods for solving these two problems will be investigated. Specifically, the 3D force detection methods in OTs system will be investigated in the next section firstly, followed by the high-speed haptic feedback solution.

Force Sensing through 3D Tracking

According to the OTs force model in equation ( 1), the optical trap can be well approximated as a linear spring within the certain scale. Therefore, the optical force exerted on the trapped object is linearly proportional to its displacement from the equilibrium position. Thus, to obtain the 3D force exerted on the probe, the core issue is to detect the 3D position of the probe. Then, the displacement can be converted easily to force using the force model. However, the 3D tracking methods for OTs or general optical microscopy are very different from that of the conventional macro-scale due to the image differentiation. For optical microscopy, the depth of field is usually very limited due to the high magnification objectives. Therefore, instead of the perspective image in the macro world, the objects under microscopy are easy to be blurred or defocus when they leave the focus plane. There is a considerable amount of literature on 3D tracking methods for optical microscopy. This section will investigate these methods in order to find out if they are suitable for realizing our goal of 3D force detection under the high-speed constraint. The 3D tracking methods in other kind of microscopies, such as in SEM are beyond the scope of this study [Piat 2016].

The 3D target tracking methods for optical microscopy are divided into three categories in this thesis according to the image acquisition mechanism. The first class is tracking based on the simplest single camera view, where only one image is needed to resolve the 3D position of target e.g. wide-field microscopy, digital holographic microscopy. The second class is to use the 3D image of working space to calculate the target position e.g. confocal laser scanning microscopy, light sheet fluorescence microscopy, and multifocal plane microscopy. Lastly, stereo-microscopy which works imitating the stereo vision techniques is also an alternative for 3D target tracking. The above three categories of 3D tracking methods will be explained in the following three subsections respectively. Specifically, in each subsection, the image acquisition techniques will be introduced firstly followed by the corresponding image processing methods. Their advantages and shortages for our usage will be demonstrated and compared in the end.

Realization Using Monocular View

This subsection will introduce two 3D target tracking techniques. Both of them resolve the target's 3D position from the single-camera view. These techniques thus are the most convenient with the simplest and fastest image acquisition mechanism.

Wide-field Microscopy

Wide-field microscopy is the standard and most conventional technique for imaging the microobject. Image of the sample is directly obtained from the microscope without any additional optical manipulation. The tracking scale is usually as large as the field of view of microscope in laterally, and varies from several to hundreds of micrometers axially. Besides, multiple targets in the same field of view can be measured simultaneously.

• Tracking Algorithm

For the standard wide-field microscopy, 3D particle positions are resolved directly from the single camera view. Generally speaking, for the x-y position (perpendicular to the optical axis of the objective lens), the basic particle tracking algorithms are used. For example, iterative 2D Gaussian fit which finds the position by iteratively determining the parameters of the Gaussian model to make it best fit to the target's features [Titus 2013]; template matching which obtains the coordinate by matching the image to the predefined template [Brunelli 2009] [Barnkob 2015]. However, for z (along the optical axis of the objective lens), positions are determined by comparing object's images with the calibrated images at different depths since the images varies with the axial coordinate of the object. For instance, the intensity difference [Zhang 2008], the size of the object [Gardini 2014],

Figure 1.7: The principle of the general defocus particle tracking: a target image is compared to a set of calibration images, and the similarity is determined by using the normalized cross correlation. The out-of-plane z-coordinate of the target particle is found where the correlation C m is the maximum. [Barnkob 2015].

the similarity to the calibration images [Barnkob 2015], the modulated point spread function [Shechtman 2015] etc. An example is shown in Fig. 1.7, where the target particle image is compared to a set of calibration images, and the z is determined as the position where two images have the maximum similarity.

After the targets are detected in each frame, particle linking algorithms are implemented to associate the detected particles and make connections [Chenouard 2014]. Nearest neighbor (NN) is the easiest particle linking method, which links every segmented object in the current frame to the nearest object in the next frame. Besides, more statistical data association techniques are presented which aim at tackling with more complex problems e.g. target overlap, appear/disappear or too close to each others. While they are usually more time consuming compare to the NN. For example, Kalman filter [Godinez 2011], Multiple Hypothesis Tracking (MHT) [Chenouard 2013] etc. Depth measurement of more than one hundred µm with nanometers accuracy in sub-second per particle is able to be achieved by these methods.

Digital Holographic Microscopy

Another monocular view microscopy is the Digital Holographic Microscopy (DHM). The holography was invented by Dennis Gabor who later received the 1971 Nobel Prize in Physics for this invention [START_REF] Gabor | [END_REF]], and it has been rapidly developed as one of the most competitive techniques for microobjects analysis recently [START_REF] Memmolo | [END_REF]]. The most distinguishable property of DHM is that it captures the light wavefront information as a hologram instead of the conventional image, and reconstruct the hologram to obtain the 3D information. More details can be found in [Sung 2009]. Especially, in-line holographic microscope (DIHM) is the simplest realization of DHM [Garcia-Sucerquia 2006]. It can be realized by the off-the-shelf components which makes it easy to customize for hardware and embedded software. Fig.1.8 shows an example of using DIHM for multi-targets 3D trajectories tracking. DHM is full-field and label-free. It achieves tracking arbitrary objects (algae, plankton, beads, etc.) in a large depth space with micrometer spatial resolution in sub-second.

• Tracking Algorithm

For DHM, the 3D positions are resolved from the reconstruction of digital holograms [Schnars 2002]. There are many algorithms for reconstructing hologram with nanometers resolution e.g. Lorenz-Mie fitting, back propagation, deconvolution, inverse problem approaches etc. More details are found in [START_REF] Dixon | [END_REF]], [Bourquard 2013], and [Cheong 2010]. However, the reconstruction are computational costly due to the process of retrieving the depth of the target and evaluation of the lateral position of the target through quantitative phase map [Akhter 2013]. For example, sub-second is needed to process a hologram of multi targets [Miccio 2014]. Improved reconstruction algorithm could reach 45 frames per second for each hologram [Girshovitz 2015]. Although methods are proposed to speed up this process e.g. parallel processing, graphics processing unit (GPU), reducing the size of the hologram etc., it is still difficult to process holograms on-line at speed of kHz level [START_REF] Seifi | [END_REF][START_REF] Dimiduk | [END_REF]].

Realization Using 3D Image

3D microscopy techniques provide the 3D images of samples by collecting a set of high-resolution images over a big depth of field. These techniques can also realize 3D target tracking by comparing a stack of images at different focus planes [START_REF] Ram | High accuracy 3D quantum dot tracking with multifocal plane microscopy for the study of fast intracellular dynamics in live cells[END_REF]].

In this section, the optical realization of three popular 3D microscopy techniques used for target tracking will be introduced firstly. Then their common tracking methods will be illustrated.

Confocal Laser Scanning Microscopy

The confocal imaging was first patented by Marvin [START_REF] Minsky | Memoir on inventing the confocal scanning microscope[END_REF]]. It aims at eliminating the out-of-focus light, thus to increase the resolution and contrast of the image in 3D space. The specific principle of Confocal Laser Scanning Microscopy (CLSM) can be found in [Webb 1996] and a simplified scheme is shown in Fig. 1.9. CLSM can be used for single particle tracking (SPT) through several µm depth with tens of nanometers and tens of ms as spatial and temporal resolution [START_REF] Liang | [END_REF]]. The downside of this technique is the image acquisition speed compared to the widefield microscopy due to the long exposures time and 3D scanning mirror mechanism.

Light Sheet Fluorescence Microscopy

The concept of Light Sheet Fluorescence Microscopy (LSFM) is firstly proposed in 1994 [START_REF] Ah Voie | Orthogonal-plane fluorescence optical sectioning: Three-dimensional imaging of macroscopic biological specimens[END_REF]]. This type of microscopy images the sample by projecting a sheet of illumination from a plane perpendicularly to the observation direction. The working scheme and 3D image are displayed in Fig. 1.10. This technique is used in It is hard to obtain a universal choice for different applications.

• Tracking Algorithm

The position of target in the above microscopies can be obtained by 3D model fitting in the 3D space [van den Broek 2013], or comparing images at different depths [Katayama 2009] etc. Then the particle linking algorithms will be used to associate target positions. The major drawbacks to adopting these methods are the limited tracking speed. That is because the 3D images are usually obtained point by point or plane by plane by using hardwares (e.g. dynamic mirror scanning, nanostage translating etc.). So that achieving the 3D image is slower than directly capture the single image by the camera. Besides, processing 3D images will be not be as fast as extracting the 3D position directly from 2D image.

Realization Using Stereo Vision

Stereo-microscopy is the stereo vision technique used in microscopy field. Two cameras with two objectives from different viewing angles are used to record the sample. More economic stereo-microscopy technique is developed using only one objective and camera with the structured light as illumination [Dam 2009]. e.g. Bowman et al. present a stereo-microscopy illuminated by two inclined optical fibers [Bowman 2010] as shown in Fig. 1.11. This system attains microsphere tracking accuracy of lower than 8 nm over ±10µm depth scale.

• Tracking Algorithm

Compared to the conventional stereo vision method, the tracking under stereomicroscopy is usually simplified. Firstly, the particles are detected from a couple of images. After that, the x-y coordinates can be obtained from the midpoint of the two images, while the z-position of the particle could be deduced if one knows the distance between two corresponding images on the camera and the angles of the two illuminations [Bowman 2010].

Summary

To summarize, in order to achieve the 3D force applied on the optically trapped probe, the key is to obtain its 3D position. Therefore, this section mainly investigates the 3D position tracking methods under microscopy. Most of these methods satisfy the 3D working scale requirement of more than several µm 3 , while few of them meet our high-speed constraint. They are compared and summarized in the in the Table 1.1. Concretely, these methods are introduced in three categories: 3D tracking using monocular image, 3D image, and stereo image. Among where, the methods based on 3D image have limited tracking speed due to the image acquisition mechanism. The stereo-microscopy has very limited popularities due to the complexity of manufacturing. The single camera based tracking methods shows the greatest potential for realizing 3D tracking at kHz. Since they acquire the image in the most direct way and are beneficial from the least number of images for processing. By combining the single camera based tracking methods with the high-speed image acquisition and processing method, 3D high-speed position tracking can be realized and thus obtaining the 3D high-speed haptic feedback.

However, the challenge has not been solved. As will be shown in the next section, although high-speed image capturing is easy for the modern vision sensor technologies, the kHz image processing are hardly realized by the up-to-date methods. Various solutions are carried out for speeding up these processes, and they will be investigated in the next section.

High-Speed Haptic Feedback

Similarly, for achieving the high-speed haptic feedbacks, the most crucial problem is to realize the high-speed probe position tracking according to the OTs force model. Our idea is to combine the single image based 3D tracking method as discussed in the last section to the high-speed image acquisition and processing techniques, thus the 3D high-speed position is resolved. Although the high-speed image acquisition is not a problem for the modern camera, the serious bottleneck lies in the real-time high-speed image processing. As explained in [Handa 2012], the digital real-time visual tracking algorithms rarely exceed 60 Hz. Therefore, in order to realize our objective, a solution for high-speed on-line tracking is required.

In this section, the necessity of why we need real-time high-speed vision will be described firstly. After that, the existing high-speed tracking solutions in the field of micromanipulation: both the instruments and the processing methods will be investigated and compared.

Necessity of High-Speed Haptic Feedback

The human sense of touch is exquisitely sensitive. The haptic sensation signals are resolved by human sensorimotor system at a significant high frequency range reaching one kHz for healthy adult individuals [Verrillo 1963] [Berthoz 2000]. Meanwhile, microworld is characterized by high dynamics and non-intuitive physical phenomena different from macroworld [START_REF] Chaillet | [END_REF]]. As a result, in order to recreate force information occurred in the microworld and thus provide faithful haptic feedback to human users, the force detection ought to be conducted at high frequency rate of more than 1 kHz. Otherwise the haptic information that takes place in the microworld will be significantly truncated or lost during detection. Therefore, in order to obtain faithful haptic feedback, high-speed processing rate must be achieved.

A more strict condition is that, the processing not only needs to be high-speed, but also be carried out in a deterministic manner. That means, in computation loop, the haptic processing must be completed within a bounded time in each sampling period. This strict condition leads to real-time implementation. There are mainly two reasons: firstly, the irregular signal processing rate will deform or taint original force signals in frequency domain. Secondly, since the scaling factors to convert pN optical force to Newton perceptible force is very large, the system is extremely sensitive to delays, because dephasing will give rise to instability for a large gain system [START_REF] Pacoret | [END_REF]] [Bolopion 2013]. Therefore, irregular sampling rate will cause unfaithful haptic signals as well as instability during manipulation.

To resume, in order to obtain faithful haptic feedback as well as stability, highspeed force detection must be realized in a real-time framework. In the following section, high-speed tracking method for haptic rendering will be investigated. The details for real-time implementation will be explained in the Chapter 4

High-Speed Force Detection

As discussed before, high-speed force detection is crucial for achieving faithful haptic feedback. It is converted to position tracking in OTs. While the kHz tracking techniques suitable for micromanipulation are seldom. This subsection will investigate these existing high-speed (kHz) tracking solutions. They are classified into three categories: digital camera based high-speed tracking, photodiode based highspeed tracking, and a novel event-based vision which is particularly suitable for real-time high-speed target tracking. Fig. 1.12 shows the photos of the digital camera, photodiode and event-based vision sensor respectively.

Digital Camera based Tracking

Tracking targets through digital video stream is the most widely used way. It is convenient since most microscopy systems has already been equipped with a camera to observe the sample. Plenty of tracking tasks are realized with the digital camera, e.g. high-speed two-fingered microhand system for cell assembly [Avci 2015]. The microfinger's position is processed at 60 Hz, which is a limitation for further closedloop control of high-frequency vibration. The CCD (charge-coupled devices) and CMOS (complementary metal-oxidesemiconductor) are by far the most widely used camera. Modern digital cameras provide a large number of pixels, and high spatial resolution (µm) with frame rates of tens of frames per second. High-speed camera could reach tens of kilo fps, which is sufficient for most of the microscopy applications [Otto 2011]. However, the realtime high-speed visual processing at kHz level is hardly realized due to the limitation of the processing speed [Handa 2012]. Special measures are developed in order to speed up the processing rate. To sum up, there are mainly three solutions.

Firstly, to decrease the field of view (FOV). For example, Bowman et al. presented a 3D particle tracking method for processing stereo-microscopy image under microscopy [Bowman 2011a]. They achieved 1 kHz close loop control for tracking the 3D position of microspheres in a small region of interest of 60 × 60 pixels. Naoko Ogawa et al. proposed a column parallel vision (CPV) servo system for continuously observing free-swimming cells with high magnification. 128 × 128 pixels gray-scale image is processed at 1 kHz frame rate, and used to control the moving stage to "Lock-on" tracking the swimming cells [Hasegawa 2008]. The disadvantage of using a small field of view is the highly constrained working space. During the tracking procedure, the target may easily go out of sight without the dedicating target following designs. Secondly, to use the simplest tracking algorithms. One of the most commonly used algorithms for high-speed microparticle position tracking is the centroid. This algorithm can be implemented in order of 10 -2 ms when tracking single fluorescence sphere particle in dark field microscopy. Parthasarathy introduced a non-iterative 2D particle tracking algorithm running at sub-millisecond with better accuracy robustness than the centroid method [Parthasarathy 2012]. As explained in the Fig. 1.14 this method uses the fact that the intensity gradient line of each pixel of the particle image would intersect theoretically the true center of it. Therefore, the desired single-particle localization was achieved by calculating the point which has the least total distance to all gradient lines. Shu-Lin Liu et al. present similar result in 3D particle tracking. The algorithm has the similar principle as the above one. They use this method to fit 3D image by sampling the point spread function (PSF), and achieve 3D localization with nanometer accuracy and sub-millisecond running time [START_REF] Liu | [END_REF]]. It can be clearly inferred that these methods are only suitable to simply situations, e.g. spheric target with no obstacles around, fluorescence image in dark field microscopy where omit image segmentation procedure. When contact or occlusion happen, which are very common during manipulation, the detection results will be seriously impacted.

Thirdly, accelerate the processing rate by hardware. Dedicated image processing hardware e.g. FPGA (Field Programmable Gate Array) and GPU (Graphic Processing Unit) parallel processing technique are used in some cases. For example, Huhle et al. demonstrated a camera-based 3D particle tracking method, which achieves kHz real-time data processing with GPU acceleration. They realize multi-particles tracking with sub-nanometers resolution [START_REF] Huhle | [END_REF]]. However, parallelism is not suitable to all vision algorithms since most of the algorithms are designed in a sequential processing way. Besides, additional money and time are needed to implement the hardware system, which is not efficient compared to a well designed software. Last but not least, the customized hardware system is not generic.

Photodiode based Tracking

The Photodiode is an optical position sensor, that can measure the position of the incident light. Usually, the incident light is either the spot beam coming from fluorescent micro/nanoparticles, or a laser beam passing through the object. Thus by detecting the position of spot beam, or the shift and intensity of the passed laser beam, the 3D position of the target can be obtained [Bowman 2013]. An example of obtaining the 3D position of the particle by detecting the reflected laser spot on the photodiode is illustrated in Fig. 1.15.

Here are many applications of photodiode based tracking in microscopy. Ruh et al. present a fast parallel 3D tracking system by combining the quadrant photodiode and acousto-optic deflector [START_REF] Ruh | Fast parallel interferometric 3D tracking of numerous optically trapped particles and their hydrodynamic interaction[END_REF]]. Their system is proved by tracking 9 spheres of different sizes around 1 µm with excellent tracking precision of 1-5 nanometers at 50 kHz. Welsher and Yang introduced a method to capture the cellular uptake process in real-time by using two single photon counting avalanche photodiodes. They applied this method to observe the landing of nanoparticles on the 3D cellular contour with a very precise spatial resolution about 10 nm in 3 axis and close-loop control signal at 100 kHz [Welsher 2014].

The most outstanding advantages of photodiode based tracking are the high precision (reach down to 0.1 Å ) and high bandwidth (up to 1 MHz) [Bowman 2013]. However, this method provides only the position without other information of the Lateral detection: the illuminated difference between the two halves of the photodiod is proportional to the particle's displacement. (b) Axial detection: when the trapped particle moving closer to the microscope objective, the light signal is decreased. On the contrary, when the particle moves away from the microscope objective, the light signal is increased [Bowman 2013].

target e.g. shape, texture etc. What's more, if there are other obstacles attached to the target, the tracking beam will be distorted, and the result will be influenced directly. That means this method is vulnerable to disturb and contact. Thus it is not suitable for micromanipulation tasks since during the manipulation, the probe will always contact with other objects in order to sense the forces and implement manipulation. Besides, this method needs more precise alignment than its counterpart, and the measurement range is limited by the size of the incident light, around 1 µm in each axis.

Dynamic Vision Sensor Based Tracking

The practically usable Dynamic Vision Sensor (DVS) was firstly developed and applied for target tracking by ETHZ, Switzerland. Although it is not widely used due to novelty, the distinctive capability of DVS for high-speed real-time target tracking has been proved by many researchers recently. More details about this vision technique will be presented in the next chapter. In this part, only the overview information will be introduced.

The development of DVS is inspired by the biological vision mechanism. Unlike the popular vision sensors which are driven by manually set global control signals, DVS is driven by "events" happened in the field of view like their biological models. It has 128 × 128 independently spiked pixels. The events generated by DVS pixels can be conveyed with a minimum temporal resolution of 1 µs. The DVS's pixel response frequency attains to 3 kHz under 1 klux scene illumination.

The processing method of events is different from the conventional color or gray images. They will be called "Event-based" method in the throughout thesis. Many event-based vision algorithms are proposed e.g. event-based target tracking, noise filter, stereo matching, correspondence determining, optical flow and so on. More algorithms and applications will be explained specifically in the next chapter.

The first application of DVS in the microscale was presented by the group of Regnier. Benefiting from the high-speed acquisition and processing rate of eventbased vision, the OTs system with 1 kHz haptic feedback is achieved by ni20132d. The efficiency of the system is demonstrated by performing the task of exploring a spherical surface. [START_REF] Ni | Asynchronous event-based high speed vision for microparticle tracking[END_REF]] presented a vision-based microrobotic system which assisted users to manipulate the 50µm microsphere with microgripper in a pickand-place task.

Others

Except for the techniques explained above, some other high-speed tracking methods are presented in some researchers. For example, Ogawa et al. proposed a column parallel vision (CPV) technique . CPV is a vision sensor equipped with 128 × 128 photodetectors with an all parallel processor array. 1 kHz closed-loop on-line vision servo has been achieved [Hasegawa 2008]. Another example is the line scan cameras, which record only one line of pixels at one time, and achieve a line scan rate of 10 -90 kHz. Two-dimensional image can be obtained by moving the observed scene in the direction perpendicularly to the camera scan line, or using galvanometer scanning motors. It is used to image endothelial cells in [Ang 2016

Summary

As explained, force detection in OTs is acquired from the position tracking. Thus in order to carry out the solution for high-speed haptic feedback, the key is to find out the high-speed target tracking method. This section investigated the high-speed tracking techniques. Based on the high-speed digital camera, several solutions are developed for obtaining kHz image processing rate. However, these methods have to compromise their performances with processing speed e.g. small working space, limited to simple single particle tracking tasks etc. The photodiode based tracking is not suitable to the micromanipulation tasks since it is vulnerable to disturbs traced back to its working principle. DVS is a promising technique, while it suffers from low resolution and still limited to 2D force detection under microscopy. These techniques are summarized and compared in Table 1.2.

Challenges

The aim of this thesis is to develop a biological-safety OTs system with 3D highspeed (kHz) haptic feedback. This system will be used for the biological application and achieves faithful 3D haptic perception. To realize this, the most two crucial problems are the 3D force detection and high-speed haptic feedback. As explained before, in OTs system, the force detection is translated to the position tracking according to the optical force model. Therefore, the two core problems are 3D high-speed position tracking in OTs system. This chapter has surveyed the existing 3D and high-speed tracking methods. However, as discussed, the existing techniques hardly meet our requirements. Concretely, the main problems are:

• As presented, many attempts has been made in order to realize the 3D particle tracking under microscopy. Although these techniques fulfill the requirement of 3D working space and resolution, only few of them have the potential to meet the kHz tracking speed constraint. Some of them are limited by the 3D image acquisition speed, while most of them are limited by the 3D image processing rate.

• Many solutions are developed to meet the requirement of high-speed on-line tracking. Unfortunately, they are either not suitable for micromanipulation tasks, or limited by some constraints such as small working space and nonefficient. The event-based tracking is a promising technique for overcoming these limitations, while the solution for 3D high-speed tracking is still underdeveloped.

As a consequence, none of the current techniques address our need of 3D highspeed (kHz) haptic feedback. We must propose a novel solution. To resume, we must:

• design and construct a biological-safety OTs, which must incorporate the latest event-based camera in a designed light path so as to provide suitable single-camera image for subsequent 3D high-speed force rendering;

• realize faithful haptic rendering by developing 3D high-speed tracking algorithms, which must be robust during complex micromanipulation tasks in large workspace;

• apply the 3D high-speed haptic OTs system to biological exploration tasks, and evaluate the effectiveness of this system.

These work will be realized in the following three chapters respectively. Specifically, this thesis is organized as follows:

• The Chapter 2 will develop the 3D high-speed OTs from the hardware level.

Specifically, a novel event-based sensor will be introduced and calibrated, which paves the way for realizing high-speed probe tracking. Then, the eventbased sensor will be integrated to the designed biological-safety OTs. Besides, the suitable monocular image for realizing 3D high-speed tracking is developed.

• The Chapter 3 will focus on realizing haptic rendering by developing 3D highspeed particle tracking algorithms based on the system developed by Chapter 2. The algorithms' performances in the experiment condition will be analyzed in detail. As a result, we achieve tracking algorithm with less than 1 ms running time and robust in the complex environment with large 3D working space of 6 × 7 × 5µm 3 . Besides, novel algorithms for future multi-particle tracking are also developed.

• In the Chapter 4, the haptic feedback in real-time framework will be established into the OTs system, and the system will be applied to biological applications. User experiments will be conducted to evaluate system's intuitiveness and effectiveness. Convincing results show that with the assistance of the 3D high-speed haptic feedback, the users' performance during complex tasks are greatly improved. At last, a preliminary work of using optical trapped probe as a 3D high-speed force sensor to implement motion planning in 3D complex environment will be presented. As presented in the previous chapter, the main issue to achieve 3D high-speed haptic feedback in OTs system is the lack of 3D fast tracking method. This chapter will partially solve this problem at the hardware level. Concretely, the novel eventbased sensor ATIS will be introduced into the OTs system to capture the high-speed dynamics. Besides, the optimal monocular image for conducting 3D probe tracking will be studied. In the end, the biological safety OTs equipped with the suitable image for the subsequent 3D high-speed force rendering will be established.

Event-based Vision

As discussed in the last chapter, although the commercial digital cameras have exciting performance in high-speed image capturing, they are not able to achieve high-speed on-line haptic feedback due to the speed limitation of image processing. The challenge of on-line kHz image processing is seemingly unsolvable until the revolutionary bio-inspired solution is found. In this section, the concept of this bio-inspired vision technique will be explained firstly. Then, more details about its particular working principle and advantages for high-speed tracking will be highlighted. At last, the state-of-the-art of this vision technique in both macro and micro scale will be investigated.

Silicon Retina

Silicon retina which comes from biomorphic engineering is our solution for overcoming the image processing speed constraints. It is a typical device of biomorphic engineering which aims at executing the computation by imitating the biological behaviors. Silicon retina works mimicking the biological retina -automatically filter the visual data collected by the eyes and leave only the essential information for the brain to compute [Posch 2011b]. The pioneer prototype of biomorphic silicon retina was built by Mahowald [Mahowald 1992].

The most distinctive difference between the conventional CCD/CMOS camera and the silicon retina is the image capture mechanism. For the digital camera, image sensor turns light into the discrete signals. The brighter image at a given pixel produces a larger intensity value at that pixel. The image sequence is collected frame by frame with a fixed sampling interval. While for silicon retina, for each pixel, as long as the outside stimuli or the input light intensity surpasses a threshold, an "event" is triggered independently and asynchronously without any global clock. The temporal resolution of the triggered events is up to several µs.

Principle and Properties

Asynchronous Time-based Image Sensor(ATIS) is the latest generation of eventdriven time-encoding silicon retina [Posch 2011b]. This sensor is able to deliver fast image sequence and has better resolution compared to the old DVS. It will be used as our vision sensor in this thesis. Fig. 2.1 shows the circuit layout and the principle of ATIS. It contains fully autonomous 304 × 240 pixels. Each pixel combines an illuminance change detector circuit and a conditional exposure measurement block. The working principle of ATIS is shown in Fig. 2.1 (c), for each pixel, when the brightness change of incident light reach certain magnitude, positive or negative events are generated depending on if the brightness is increased or decreased. Each pixel arbitrates the output channel asynchronously when it has relative information, without waiting for the whole frame. Compare to the DVS, ATIS is equipped with an additional exposure measurement circuit which provides more information about the brightness. Although this function is not used in this thesis, it will be explored in the future perhaps for removing the noise, target tracking, correspondence determining etc.

The event-based imaging scheme is particularly suitable for on-line high-speed object tracking.

1. High-speed acquisition. The events are updated very fast (µs time interval) as long as they are spiked without waiting for the arrival of the entire frame. Hence system responsiveness to the outside stimuli is significantly increased. The consequent acquisition speed is thus higher than the conventional frames of image.

2. High-speed processing. Events are generated corresponding to light contrast. Thus in a scene with the stable light environment, only the dynamic information which is stimulated by moving object is recorded. The static background are ignored automatically and the targets' contour are encoded by the event flow. Therefore, the image segmentation is omitted naturally and the target can be directly tracked. An example of event-based image is shown in the Fig. 2.2 (b) and (c), a human face is encoded by a set of events and displayed with certain accumulation time interval.

3. Economical. Last but not least, the silicon retina is economical in both energy and price compare to the other high-speed imaging solutions. It has zero consumption to static scenes which is important in consumer electronics. It is light weighted and can be easily carried by the mobile robot for on-line high-speed object tracking [Yuan 2016]. Also, this sensor achieves large dynamic range (DR) of 143 dB which allows robust operations under the scene with an extreme contrast intensity without encountering over-or under-exposure problems [Posch 2011b].

Therefore, this technique provides huge potential for realizing high-speed object tracking under microscopy. However, as it can be seen, the processing method of asynchronous events is basically different from that of the conventional images. Dedicated event-based vision algorithms are required. In the next part, state of the art of the event-based vision's applications and corresponding algorithms will be investigated.

State of the Art: Event based Target Tracking

The silicon retina is extremely adaptable to high-speed object tracking due to its efficient encoding of movements. However, due to its novelty, few applications and event-based algorithms are demonstrated so far. In the following content, superior applications in both macro and micro scales will be displayed.

Applications in Macro-scale

The earliest application of silicon retina is the vehicle detection in traffic monitoring [Litzenberger 2006]. A cluster detection algorithm is proposed for real-time fast vehicle position tracking. This invention patent has then been commercialized as part of the Intelligent Transportation System (ITS) to count the number of vehicles and estimate their speeds. This algorithm is also used in a sensory-motor reactive controller, which use visual servoing to track the position of the ball and keep the goal as shown in [Delbruck 2007]. In a pencil balancing robot, fast visual feedback is provided by using Hough line transform and stereo vision with a pair of DVS sensors viewing from different angles [START_REF] Conradt | [END_REF]]. Delbruck introduced several eventbased vision methods on background noise filter, orientation feature extraction, and cluster tracking [Delbruck 2008].

Many event-based processing methods are studied by the group of Benosman. The event-based optical flow is presented by Benosman, which benefit to significantly reduce running time compare to the conventional method [Benosman 2012]. Another matching method is present in [Schraml 2015], which based on the distribution of events rather than pixel-wise correspondence, and this algorithm is able to handle the absence of information with improved depth reconstruction accuracy.

Applications in Micro-scale

Although many applications of event-based vision are presented in macro scale, its usage in micro scale is an almost unexploited area. As introduced in the previous chapter, our group did the pioneer work. In the previous work of [Ni 2013b] and [START_REF] Ni | Haptic feedback teleoperation of optical tweezers[END_REF]], 2D high-speed haptic feedback has been achieved. This was the first time that human touch the curved surface using a probe with high-speed haptic feedback as shown in Fig. 2.4.

To sum up, the silicon retina is basically different from the digital CCD/CMOS camera on both image capturing mechanism and the processing method. It is particularly suitable for high-speed target tracking due to its advantages of highspeed motion acquisition and processing etc. Thus, it shows great potential for fulfilling the objective of 3D high-speed haptic feedback in OTs. In the following Figure 2.4: (a) Photo of the probe "coming into contact" with the target object. Inset(i) and inset (ii) show the moment before and during touch, respectively. Below is a side view schematic representation of the "come-into-contact" procedure. The probe locations (i) and (ii) correspond to the inset images (i) and (ii). (b) The positions and forces when the probe "walks around" the surface of the big sphere. The probe trajectory is depicted in black dots, the laser position in green line and forces in blue arrows. For clarity, the force arrows are subsampled to one thirty [Ni 2013b].

part, ATIS will be applied into OTs system for the first time. However, before using it, its high-speed motion acquisition capability in the microscopy condition needs to be verified. Thus in the next section, several experiments will be conducted to test its performance.

ATIS Calibration

The goal of this section is to prove the ATIS's high-speed acquisition capability under microscopy condition by quantitatively testing the parameter of:

• pixel's response frequency under low light condition.

Where the pixel's Response Frequency indicates the ATIS's capability of catching high frequency changes. Thus high pixel's response frequency is essential for capturing dynamics. As previously explained, the events are generated due to the relative light changes. However, it is known that the biological image under microscopy usually suffer from low contrast, thus the pixel's response frequency under this poor light condition needed to be tested. The procedure of determining ATIS's response frequency will be called "ATIS calibration" in this thesis. In the following part, the calibration method will be explained firstly. Then, the specific experiments will be conducted to quantitatively analyze the response frequencies of ATIS under different low light contrasts.

Experimental Method and Design

In order to obtain the pixel's response frequency, the idea is to stimulate the ATIS with a flashlight signal over a band of frequencies. When over a certain frequency, the output of ATIS can't well respond to the input signals, this frequency is defined as the pixels' maximum response frequency. This test will be conducted at different light contrasts by adjusting the intensity of the stimulus light to obtain the pixel's response frequency in different light conditions. The experimental design is shown in Fig. 2.5. A high bandwidth laser diode (Gated Cameo 650 nm 1 mW) is modulated by a function generator (TTI 0-3 MHz) to provide ON and OFF square wave flashing light. The rising time of the laser diode is less than 5 µs, which is negligible compared to the response time of the ATIS pixels [Posch 2011a]. Thus it is an ideal stimuli. The light firstly goes through two optical densities (OD3 and OD4) to decrease the power. Then 3 lens (f=-25, f=75, f=100) are used to focus the light well on the ATIS as well as obtain the adjustable spot area. This spot is composed of the pixels stimulated by the diode. Therefore, they are our research object and will be studied in the following subsection. The flashing frequency is changed by adjusting the signal generator, and the illuminance intensity is adjusted by the driven current. As a result, pixel response under different frequencies and contrasts will be obtained by analyzing the output of these stimulated pixels. This configuration is designed for reproducing the microscopy conditions which usually have low light contrast. Bias parameters, which are used for tuning the parameters of the ATIS circuit, are set in order to obtain high-speed and high-contrast sensitivity as shown in the table of Fig. 2.5.

Notice that too fast or sensitive bias setting may cause too much background noise and signals. Compromise is made experimentally to achieve the optimal settings: high-speed performance as well as the proper amount of noise.

Response Frequency under Low Light Condition

As explained in the last subsection, the stimulated pixels are our study object for the pixel's response frequency testing. The assembly of these stimulated pixels will be called "active area" in the following content. However, as will be shown, the active area changes with the stimulative frequencies and intensities. Thus the method of determining the size and position of this active area need to be proposed before testing the response frequency. Therefore, in this subsection, the method of determining the active area will be explained firstly. Then the response frequency under 1 lux will be studied with a proposed experiment. At last, the same experiment will be conducted at different light conditions from 0.5 to 10 lux to provide the response frequencies under different contrasts.

Active Area

The method of determining the active area is described in the first experiment. Take the light intensity of 1 lux for example, as shown in the Fig. 2.6 (a), laser diode generates an event-based light spot with accumulation time of 33 ms. As the stimulation frequency changes, the stimulated pixels produce the different number of events correspondingly. The (b) and (c) illustrate average of event numbers of each row and column under different frequencies (from 10 to 1300 Hz), with collecting time of 1 s. From the graphics, average event numbers in both x and y coordinates show Gaussian distributions due to the characterization of the Gaussian beam of the incident light. From the edged to the center of the Gaussian curve, the higher light intensity leads to higher contrast, thus generates more events.

The active area is defined as the smallest square which contains the stimulated pixels, where the stimulated pixels are the ones that generate more events than the background pixels. Fig.2.6 (d) shows an example of how to determine the length and width of the active area at 1 kHz. As calculated, the non-stimulated pixels generate less than 10 events per second. Thus a threshold of 10 is set to the curve in x and y and obtain the length and width of the active area respectively. The same threshold is set for all the curves in (b) and (c) thus obtain the active areas under different frequencies.

As shown in Fig. 2.7 (a), the active areas decrease with the increasing stimulative frequency. The inset figures show two examples of the active area at 10 Hz and 1 kHz, the number of spiked events decreases from red to blue in the color bar. The sharply decrease at low frequency (under 50 Hz) is caused by the artifact from the light source. As the frequency increases, only the pixels in the center of the Gaussian beam are able to be stimulated due to the larger light contrast in that region. Fig.2.7 (b) shows the number of events collected from the whole sensor and the active area respectively. Specifically, the collected maximum number of events reaches 1.9 Mevents/s at 100 Hz. The number increases first from 10 to 100 Hz, and then fall down with the frequencies. At higher frequencies, the number of events from the whole sensor converged to the noise level. While the events collected from the active area tend to zero. That means ATIS pixels do not respond to the signal of ultra high frequency under this light condition.

In conclusion, this experiment proposed the method for determining the active area which will be further used in the following experiments. At the same time, the qualitative analyze of how the event number varies with stimulative frequencies under 1 lux light is displayed. The overall trends are: events number increase with light contrast, while decrease with stimulative frequency. In order to quantify the pixel's response frequency, the following experiments is conducted.

Response Frequency

Pixel's response frequency is defined as the highest stimulative frequency that pixels respond to. The goal of this experiment is to quantify the pixels' response frequency. Specifically, the events flow in time domain will be shown firstly to provide an intuitive impression of how event number changes with the stimulative frequency.

Then the specific response frequency under 1 lux light contrast will be calibrated with a proposed method. Then, the same method will used to calibrate the response frequency of ATIS under other light contrasts.

The pixels which generate the maximum number of events are used to study the events series in time domain (almost the same pixel due to the best contrast). Theoretically, at least 2 events should be obtained during each modulation, that means one event at each light on and off. As can be seen in the Fig. 2.8, at low frequency, more than two events are generated at each modulation. This number decrease gradually from 10 Hz -1.3 Hz. Below 100 Hz, redundant events are detected in spite of no light contrasts, which leads to inaccurate detection of change. At 1.3 kHz, each modulation corresponds to almost two events, and seldom redundant events are detected otherwise. The even higher frequencies lead to incorrect event series: not enough events are collected.

To study the response frequency of ATIS pixels, the following methods is used: the average numbers of events from the active area and the optimum area (the square in the center of the light spot where get the maximum power of light, 5 The events flow from one pixel which generates the maximum number of events during 1 s with different stimulative frequencies. pixels ×5 pixels) were calculated in 1 s. Then the numbers are divided by the stimulative frequency, and the corresponding area respectively to obtain the average event number in each modulation cycle.

The average event number of the active area with the error bar, the optimum area, and the maximum event number under different frequencies are shown in Fig. 2.9. As can be seen, the average number of events in the optimum area and the maximum numbers of events by one pixel reduce sharply first and slowly after, and reach a threshold of 2 at around 1 kHz and 1.3 kHz respectively. The reason is that many redundant events are detected at low frequencies as mentioned in Fig. 2.9. Meanwhile, the average events numbers of the active area increase before 100 Hz since the areas below 100 Hz are much bigger due to the artifact. After that, the numbers decrease gradually and reach 2 at about 800 Hz with a standard deviation (STD) of 1 event. The result of the optimum area is better than the active area since it benefits from a higher light contrast.

The response frequencies of ATIS under other light contrasts are studied with the same method. The result is shown in Fig. 2.10 (a). Both the average and maximum response frequencies increase sharply with the illumination from 1 to 2 lux and slowly from 2 to 10 lux. It is noticed that light contrast less than 1 lux weaken the response speed of ATIS seriously. Fig. 2.10 (b) shows the events number stimulated by the increasing and decreasing brightness. PE stands for the positive event and NE stands for negative events. The events number of different polarities per second is almost the same with frequencies. That means ATIS pixels response similarly to increasing and decreasing light changes. Specifically, the average number of events in the active area of positive events is slightly higher than the negative ones, while the maximum number of negative events appears to be higher than the positives ones in high frequency (500-1600 Hz). The red line is the threshold of 2 events: one event is generated at the brightness rising edge, the other is generated at the brightness falling edge.

Summary

To summarize, this section studied the ATIS's response frequencies under low light contrasts. It confirms ATIS' high-frequency dynamics tracking capability in microscopy. Specifically, under the same stimulative frequency, the higher light contrast leads to more events generated, and higher pixel response frequency. With the same light contrast, the increasing frequency leads to decreasing number of events at every modulation. With 1 lux Gaussian beam stimulation, the average response frequency of pixels is about 800 Hz and the pixels in the center of Gaussian beam which get the best contrast have the response frequency of 1.3 kHz. As the light contrast increases to several lux, higher pixel response frequency is obtained. The ATIS generates almost the same number of events of different polarities at different conditions i.e. frequencies, contrast. For accurate dynamic detection, ATIS is suggested to be used at more than 1 lux light contrast.

The following section will integrated the ATIS to the OTs system. As explained in the Chapter 1, the previous system suffers from some of limitations e.g. low- resolution of the DVS, the 2D force detection, and damage to biological samples.

The following sections will solve these problems at the hardware level.

OTs for 3D High-Speed Force Sensing

As proved in the last section, ATIS is able to capture high-speed motions at kHz under low contrast condition. Therefore, in this section, it will be built into the light path and pave the way for achieving the high-speed haptic feedback. The main objective of this section is to carry out the solution to the 3D force detection from optical realization. Specifically, three OTs setups are developed at the hardware level in the following three subsections respectively for obtaining the optimal monocular ATIS image suitable to 3D force detection. Their performances will be analyzed and compared.

The First Setup

The first setup is built based on the previous 2D high-speed OTs setup. While several improvements are made: 1. ATIS is integrated into the system. 2. The light path is optimized for providing more light to ATIS. Thus achieves better pixel's response frequency and less background noise as discussed in the last section.

3. A 3D nanostage is applied instead of the previous 1 nano-stage in z direction with 2 microstages in x and y directions respectively. Therefore, more precise position control is achieved. 4. The old laser is changed to a new power adjustable green laser. The optical path is shown in Fig. 2.11 (a). A 532nm laser source (working power output: 300 mW, maximum 1.5 mW) is used to generate the optical trap. This laser beam is directed into an oil immersion objective (Olympus UPlanFLN 40x, NA 1.3). A fixed optical trap is formed on the sample plane. Besides, the original two microstages are used for x-y plane coarse positioning (PI, M126.CG) and for x-y-z fine positioning, a three-dimensional motorized nanostage (PI, P562.3CD) is adopted. Thus the operating precision is improved from µm to nm. The illumination (LED, 3W) used to image passing through the objective and is divided by a polarizing beam splitter cube to two cameras: a conventional CMOS camera (Basler 659×494 pixels) and the asynchronous event-based image sensor (ATIS). The CMOS shows the overview environment thus provides the visual perception to operators, while the ATIS is used to track the position of the probe in order to achieve the force applied on it. A photo of the whole platform is shown in Fig. 2.11 (b).

The 3D frame of the OTs working space is shown in the Fig. 2.12. To a probe trapped by the OTs, the 3D coordinate is defined: the x-y plane is perpendicular to the optical axis of the objective lens, z-direction is along the optical axis of the objective lens. The throughout thesis will use this definition of x, y, and z.

As discussed in the Chapter 1, to realize the 3D high-speed force detection, our idea is to develop the monocular view based 3D high-speed tracking methods. The axial position of particles in this method is obtained by comparing the defocus image under different depth. Therefore, for 3D position detection, there are mainly three steps: firstly figuring out the image feature which varies with its axial position. Then using some methods to parameterize these features, and calibrate these

X Y Z Figure 2
.12: The 3D coordinate of the force measurement working space. The z-axis is defined to be the axial direction along the optical beam, while the x and y are in the lateral plane perpendicular to the optical beam. parameters at the different axial position. Thus, during the experiment, the axial position of particle is extracted by comparing its parameters with the calibrated ones.

A series of probe images under CMOS camera and ATIS are shown in the Fig. 2.13. As can be seen from the figure, the events are displayed as 2D points. The accumulation time is 33 ms, and the positive and negative events are displayed in the color of green and purple respectively. When the bead moves, events are mostly spiked by the beads' edge which shows the different brightness compare to the background. Therefore, the collected set of events implicates bead shape, names a circle in these images. By observing, it is found that the diameter of the circle varies slightly with the z-position from z=-1 to z=1. Thus by detecting the radius of the circle, we could have a hint of the axial position of this probe. The specific tracking algorithm and the performance will be presented in the next chapter.

It is noticed that around the focus plane (z=0), the number of events is the maximum and these events compose a clear circle edge. While as the bead moving away from the focus plane along the z-direction, the events become less and the circle edge become ambiguous. At ± 2 µm from the focus plane, seldom events are collected, which poses great difficulties in position detection. The limitation mainly comes from the defocus image of the standard wide field microscopy. Although the defocus image can be used in digital image processing and provide 3D position over large axial scale [Barnkob 2015], it can not be used by the ATIS. The low contrast of the edge in the defocus image hardly generate useful events for tracking as explained in the last section. Thus information is insufficient for the further calculation. Specific parameters will be shown in the next chapter to prove this argument. What's more, the change of radius is too small around the focus plane, thus the noise has a strong impact on the detection, especially in complex environments. The research of Ni et al. has also explained this limitation during 3D position detection [Ni 2013a].

Z=0 Z=1 Z=2 Z=-1 Z=-2
To sum up, this setup integrated ATIS into microscopy for the first time. The light path is optimized and more light are obtained on ATIS. The change of better laser emitter and nanostage provides more powerful trapping capability and better operating accuracy of nanometers in three axes. However, the working scale in the z-direction is not satisfied, which should be at least several µm for our probe of 3µ diameter. Besides, the detection is vulnerable to noise which represents a serious drawback for 3D manipulation. A tracking setup with bigger axial working scale is required, and it will be explored in the following part.

Inline Holographic Setup

The DHM (Digital Holographic Microscopy) is recalled due to its large axial working space. Besides, it is the other monocular tracking technique except for the wide field microscopy as discussed in the Chapter 1. Among the DHM, the in-line microscope Normally, each single hologram collected by camera contains all the 3D information of the target. By reconstructing the hologram at different z-coordinate, 3D information of specimen is able to be obtained [Verrier 2015]. However, the widely used reconstruction methods are not suitable in our case since:

1. The reconstruction process is computational costly as discussed in the Chapter 1. It is impossible to obtain the 3D position at the kHz refreshing rate. 2. The complete hologram information can not be captured by ATIS. Due to the weak light contrast of part of the hologram, only the first several orders of the diffraction rings generate events for tracking.

Inspired by [Gardini 2014], where microbead's z-position is obtained the by detecting the radius of its outermost ring of diffraction pattern, we find some interesting clues for z-position estimation. A set of holograms of a single microbead at the different axial position under CMOS camera and ATIS are shown in Fig. 2.15.

From the figure, it can be seen that the hologram of a single bead is a set of concentric circles. The first several orders of diffraction rings that own the biggest contrast generate several event-based ring on ATIS. The radii of these rings vary with probe's axial position. Therefore, by detecting the radius of the first several rings, the z position is able to be calculated. By estimation, the working scale is enlarged to about 6 to 7 µ m as observed visually.

This method is suitable for high-speed on-line 3D single particle tracking (SPT). While when it is applied to haptic micromanipulation, the drawback is obvious:

1. Too much noise when touching or approaching other objects. Since the diffraction rings of different objects will intersect with each other as shown in Fig. 2.16, thus the image of the probe is interrupted and deformed.

2. Bad visual experience for users. Not only the dark and complex background, but also the diffraction rings of probe make users feel confused and difficult to find the best axial plane to trap the probe. Besides, it is hard to figure out which ring indicates the edge of the probe from the visual information. Force feedback combined with a better vision experience will be more helpful for users to conduct the manipulation.

In summary, the in-line holography method provides bigger working scale than the last setup. By tracking the radius of the first several orders of the diffraction ring, the 3D high-speed SPT can be realized. However, this method is not suitable for 3D haptic micromanipulation tasks due to the interruption of other diffraction patterns, and the poor visual experience. Therefore, better tracking configuration with better user experience and enough 3D working scales are still required. In the next part, the feasible solution will be presented.

Optimized Setup for Biological Application

In the above subsections, two optical strategies are proposed. Both of them have advantages and disadvantages: the first tracking system is benefit to superior visual experience on both ATIS and CMOS camera. While it suffers from small detection scale in the z-direction and the detection is vulnerable to disturbs. The second proposal shows enough 3D working space, while didn't provide a clear view of the scene, especially in the complex working environment. In this subsection, the optimal solution which combines the advantages of both the above systems will be developed. Besides, IR laser is built into the system for achieving the biological safety optical trap.

The scheme of the optimized optical paths is shown in Fig. 2.17(a). A picture of the whole platform is shown in Fig. 2.17(b). A 1070 nm power adjustable laser source (maximum output: 10 Watts) is used to generate the OTs. This laser beam is enlarged firstly and directed into the same oil immersion objective to make a fixed optical trap. The same microstages and nanostage are used for x-y plane coarse positioning and x-y-z fine positioning. The illumination(LED, 3W) pass through a LED lens to create the collimated beam. After passing through the sample, all the outcome visible light is reflected by a long pass dichroic mirror to cameras. Compared to the last two setups which only reflect part of the visible light to cameras, this light path is more light efficient. Then the beam is divided by an unpolarized beam splitter cube (9:1) into the CMOS camera and the ATIS, where 90% of the light is led to the ATIS for higher light intensity. Compared to the previous design which split the light 50:50 to ATIS and CMOS, this light path provide even more light for ATIS. The optimized light intensity will not only decrease the background noise on ATIS, but also increase the pixel's response frequency as discussed before.

By experimentally adjusting the illuminator light path, the following images of the probe is obtained in Fig. 2 In summary, this optimized setup is suitable for 3D manipulation in biological use since it possesses the advantages of the last two systems. Firstly, the better visual experience is provided. Secondly, large 3D force detection scale is achieved. Thirdly, the distinctive parameter changes at different z-position promise the antidisturbance ability of the detection. Combined with the cell safety optical trap, the 3D high-speed OTs setup for biological application is developed from the hardware level.

Conclusion

This section revealed the solutions to the challenges of high-speed tracking and single-camera based 3D tracking solution in OTs from the hardware level.

Specifically, for high-speed tracking, the latest biomorphic silicon retina, ATIS is introduced into the OTs system. Its special working principle was presented, and the application of event-based target tracking are surveyed. Its high-speed tracking performance: pixels' response frequency under low light contrast of microscopy is studied. The test result shows that the ATIS pixels' response frequency increase with the light contrast. Under the low illuminance of 1 lux Gaussian beam, the average response frequency of pixels are around 800 Hz. Some pixels at the center of Gaussian beam could reach 1.3 kHz. As the light contrast increase to more than 1 lux, kHz response frequency will be achieved easily.

The 3D tracking is solved experimentally by investigating the suitable monocular ATIS image for the subsequence 3D high-speed probe tracking. Three imaging configurations are explored, and the optimal scheme is determined due to its advantages of large enough force detection scale in 3D, improved robustness, favorable visual experience, and simplicity of realization. The quantitative parameters will be shown in the next chapter. The biological safety OTs is developed with IR laser trap. The light path is optimized to provide more light to the ATIS.

Until now, the optical construction of 3D high-speed OTs have been realized. However, to achieve 3D high-speed haptic feedback, there is still a need for the corresponding event-based 3D high-speed tracking algorithm. The algorithms should be able to provide the 3D position of probe based on the asynchronous event within 1 ms running time. Besides, the tracking precision and robustness in the complex environment are required for manipulation. The next chapter will seek to develop the algorithms meeting the above requirements. The tracking system consists of the hardware developed above and the proposed algorithms will be evaluated. The previous chapter has only focused on the optical construction of 3D highspeed OTs. In an attempt to achieve the force, the 3D high-speed probe tracking method is required. Therefore, the aim of this chapter is to develop the event-based probe tracking algorithms. The essential requirement for the algorithms will be described firstly. Then in the next two sections, two probe tracking algorithms will be proposed corresponding to the first and lastly optimized system respectively. The performance of these two systems will be quantitatively evaluated to prove that the optimized system is largely improved compared to the first one. In the last section, a novel event-based multi-probe tracking algorithm will be exploited. It will be used in the future for more complex manipulation tasks with multiple probes.

Problem Description

As discussed in the previous chapter, the 3D high-speed haptic feedback necessitate the high-speed tracking of the position and size of the probe. However, as explained in Chapter 1, the lack of high-speed tracking algorithms poses a serious bottleneck. Moreover, to track the probe for micromanipulation tasks is even more difficult compared to the SPT problems. The reasons are:

1. Micromanipulation tasks further demand robust tracking. The robustness is essential for achieving reliable measurement in the complex environment such as in contact with obstacles, occlusion, impurities, shadows etc., which are commonly encountered during these tasks.

2. The precision and 3D working space should be ensured. Normally, the working scale of the optical trap is around one microbead radius displacement from the equilibrium position [Pralle 1999]. Thus, the measurements scale in our case should achieve more than 3 µm in 3 axes with convincing precision and robustness.

3. As discussed in the Chapter 1, high-speed processing rate of less than 1 ms is required for realizing faithful and stable haptic feedback.

To resume, the requirements for the proposed algorithms are: more than kHz processing rate, high precision, enough 3D working space and robust in the complex environment. In the next two sections, tracking algorithms meeting the above demands will be presented. Specifically, tracking algorithms corresponding to the first OTs system will be presented firstly, followed by the tracking algorithm corresponding to the final optimized system.

Circle Tracking for the First System

As illustrated in the previous chapter, the event-based image on ATIS in the first system indicate the edge of probe. Their 2D coordinates are displayed in Fig. 3.1. Therefore, the 3D probe tracking problem is formulated as: determining the center position and radius of the event-based circle. This section will introduce the specific algorithm for solving this problem followed by the tracking performance evaluation.

Typically, the process of determining a model to produce a compact representations of a group of data is called "fitting" [Forsyth 2002]. Generally, fitting involves three general problems: finding out the possible structures, the parameters of the structures, and number of the structures. However in our case, the type and number of models are already known in advance. Thus the problem is simplified to: clustering events together and fitting a single circle to them. In the following subsection, the single circle fitting algorithm will be explained.

High-Speed Robust Circle Fitting Algorithm

In this part, a robust circle fitting algorithm will be demonstrated in detail. Besides, to verify whether this tracking method meets our crucial requirements as discussed in the last section, the essential parameters will be evaluated with proposed methods.

Math

The principle of this algorithm is to use the least square fitting method to minimize the distance of the events' coordinates and the circle model [START_REF] Gander | [END_REF]]. A standard approach to fit a circle to data is geometric fit, also called orthogonal distance regression (ODR). In this approach, the practical minimization problem is nonlinear and has to be solved iteratively using e.g. general Gauss-Newton, Levenberg Marquardt scheme [START_REF] Boggs | [END_REF]] etc. However, a major concern of this method is that it often takes dozens or hundreds of iterations to converge, which is computationally intensive. An alternative to the complicated geometric fit is the fast non-iterative procedures names algebraic fits. Here we use one of the most popular algebraic circle fits: Kåsa fit which is introduced in the 1970s by Delogne and Kasa [START_REF] Kåsa | A circle fitting procedure and its error analysis[END_REF]]. One can find the circle C t : [X t , Y t , R t ] by minimizing the function:

f (X t , Y t , R t ) = m k=1 d k 2 (1)
Where

d k = (x k -X t ) 2 + (y k -Y t ) 2 -R 2
t is the so called the "algebraic distance" from the event U k : [x k , y k , t k ] ∈ U 1,m to the circle. U 1,m is the container of m inlier events of the circle model. More details about the U 1,m will be discussed later. Introducing parameters A = -2X t , B = -2X t , and

C = X t 2 + Y t 2 -R t 2 .
the equation ( 1) is transformed to a linear least squares problem by minimizing:

f (X t , Y t , R t ) = m k=1 (Ax k + By k + C + x k 2 + y k 2 ) 2 (2)
Now the problem is reduced to solve a system of linear equations with respect to A, B, and C. The extreme value is evaluated by making the partial derivatives to A, B, and C equal to zero. The three equations are represented in a matrix:

   x k y k N x k 2 x k y k x k x k y k y k 2 y k       A B C    =    (x 2 k + y 2 k ) (x k 3 + x k y 2 k ) (y k 3 + y k x 2 k )    (3)
Let D denotes the left matrix, u denotes the unknown vector, and b denotes the right matrix. Then the normal equations can be written as:

Du = b (4)
Where u can be solved easily by matrix inversion, and the parameters of circle [X t , Y t , R t ] are recovered from A, B, and C.

Robustness Solution

Fitting methods are vulnerable to outliers [Huber 2011]. Even a single inappropriate data point can affect result that is dominated by inliers, and lead to very poor fits in practice. This situation is particularly commonly encountered during micromanipulation with ATIS sensor, e.g. fix pattern noise from ATIS pixels, noise due to light changing in the environment, obstacles around the tracking target etc. Therefore, a high-speed robust solution is required. Various methods are proposed to limit the influence of the outliers. One of the most commonly used methods is M-estimator [START_REF] Huber | [END_REF]]. Briefly, the principle of M-estimator is to choose the associated weights of each data so that the outliers will have smaller dominant weights to the result, yet inliers have bigger weights. Many functions can be used as the weight function e.g. Huber, Bisquare etc. [Zhang 1997]. However, the iteratively reweighted least-squares computation will consume much more time than the basic fitting. In order to speed up the processing rate, two measures are taken.

Firstly, the weight function is simplified. If the residual is within a predetermined range, the weight function value is 1. Otherwise, the value is equal to 0. Specifically, a region of interest (ROI) is set to the circle, ROI(C t ). For every incoming event E t , the Euclidean distance D (Xt,Yt), (xt,yt) between this point and the circle center is calculated. If the distance is within a predefined range, namely ROI, then the E t is considered as an inlier and is pushed into the buffer U 1,m for the next calculation. In the other case, it will be discarded. In this way, the events around the circle are used while the others are abandoned. Thus the influence of outliers that are far away from the circle is significantly reduced. The range of ROI(C t ) is predetermined experimentally to achieve speed as well as robustness.

Secondly, the iteratively reweighted calculation is directly performed during tracking. For instance, as mentioned in the above content, the circle fitting needs the initial guess the position and radius, which further determine the ROI. Thus the ROI(C t ) determined by the current circle's parameter [X t , Y t , R t ] is applied to calculate the parameters of the next circle C t+1 : [X t+1 , Y t+1 , R t+1 ], and so forth. With this, on-line tracking is achieved. The FIFO event buffer. When the newest inlier event is pushed in, the oldest event in the buffer is popped off.

Attention should be paid in the situation when the target suddenly stops, moves faster, or other obstacles appear around the target and so on. They will cause a sudden disappearance or burst of events which may induce peak noise and instability to the system. In order to obtain more stable haptic feedback, two measures are taken.

Firstly, a smoothing solution is implemented by using the "update-from-lastposition" mechanism. Specifically, the buffer U 1,m that contains m inliers is defined as continuous FIFO (first in first out) data flow as demonstrated in Fig. 3.2. When a new inlier comes, the oldest event in the buffer is popped out.

Secondly, buffer with predefined size (with fixed event number) is used instead of temporary size (with fixed timestamp). Since if the buffer is of certain time window, the fitting result is significantly affected by the events' generation rate. Thus the concerned situation is not improved.

As a result, with the size-fixed FIFO buffer, fast moving objects benefit from a higher update rate than slower objects. The suddenly stopped objects will remain at the same position, since the new calculation will use similar events as the previous computation. Besides, the influence of suddenly appeared noise is impaired. In our experiment, we choose 200 events as the buffer size. This number varies in different cases depending on the target size, moving speed etc. The oversized buffer will filter out the fast dynamics. While undersized buffer has few effect to the peak noises. Compromise should be made. Recently, the parameters are chosen manually. In the future, machine learning method e.g. regression may be applied to calculate the model and predict the optimal parameters. In summary, this part develops a 3D probe tracking algorithm. The pseudo-code of the robust circle fitting algorithm is written in Algorithm 1. The 3D position Algorithm 1 Robust Circle Fitting 1: for every 1 ms do 2: If E t is in the predefined ROI(C t ) 3:

Update the U 1,m . 4: Compute the least square solution for: X t+1 , Y t+1 , R t+1 . 5: Threshold validation 6: Update the circle to C t+1 : (X t+1 , Y t+1 , R t+1 ) and the new ROI(C t+1 ) 7: end for of probe is represented by the detected circle center and radius respectively. The high-speed robustness solution is carried out for obtaining stable force detections. Combined with the hardware of the first OTs system developed in the last section, the first OTs with 3D high-speed force detection capability is achieved. In the next subsection, the specific performances of this system in terms of: running time, 3D working space and precision, and robustness will be evaluated.

Performance Analysis

This subsection will test the essential performances of the first OTs system's 3D high-speed tracking ability. The specific relationship between the vision result (in pixels) and real position (in µm) will be determined. The working space of this system will be quantitatively analyzed. Besides, the high-speed processing rate and robustness of tracking method will be proved.

The following experiments are made by tracking a 3 µm silica microsphere in the OTs system. The target is firmly attached to the bottom of a petri dish. A 3-axis nanostage (PI, P562.3CD) is used to control the position of the petri dish associated with the target. All experiments are implemented under the real experimental conditions, i.e. in water under room temperature. Three experiments are performed in the following three subsections to test the first system's 3D working scale for force detection, robustness, and running time respectively.

Working Scale

• Experiment Description

The x-y working scale is tested by fixing the z-position of the target while moving in x-and y-directions. The target will move across the view of ATIS from one side to the other with the step size of 500 nm.

Similarly, the z working scale is tested by fixing the target position at x=120 pixels, y=150 pixels on ATIS, and moving it along the z-direction starting from 2µm lower than focus plane. Since ATIS does not catch static information, sinusoid movement with a period time of 10 s and amplitude of 2 µm is applied by the nanostage in the perpendicular direction of the tested direction, i.e. move in x when testing y-scale and in y when testing x-and z-scale. The tracking result within a entire sinusoidal periodic is recorded and shown in Fig. 3.3.

• Result Analysis

ATIS has 240 × 304 pixels, thus in order to reach those limits in x-y, the nanostage travels from 0 to about 6µm for x-axis and 7µm for y-axis at the focus plane. As expected, the behavior of the detection is linear as shown in Fig. 3.3. The linear regression coefficients show the conversion between pixel in ATIS coordinate and micrometer in the world coordinate on each axis (42.2, 42.4 pixel/µm in x and y). It is important to note that the errors are low on the border of the sensor which means that the system is robust with truncated view of the microbead. More quantitative results will be studied in the next subsection.

On z-axis, the radius of the bead at the best focus plane (z=0) is 69.5 pixels for this configuration. The variation of this radius is plotted on Fig. 3.3 (c). It is observed that the radius has a linear relation with the axial displacement of the probe within -0.7 µm to 0.7 µm around focus position (z=0). The conversion parameter is then (2 pixel/µm). The STD is big compared to the detection scale, that is up to 70% on average. This result proves the argument in Section 2.3.1: the working scale is insufficient and the radius detection is vulnerable to noise. The non-linear edge happens when the target moves more than 0.7 µm away from the focus plane. The reason is that when the image is blurred, few events are generated due to low light contrast, and insufficient information leads to incorrect tracking result.

Robustness

During micromanipulation, the probe is often required to contact various kinds of objects with different shapes, colors and so on. In order to obtain reliable haptic feedback in these complex environments, the tracking method must be robust. Robustness in the following content is defined as the ability to extract the 3D position of the target from noisy data, or even when this information is carried only by a small subset of data. The robustness of this system will be tested under two most commonly encountered situations during micromanipulation: obstacles disturbances i.e. the target image is connected with other obstacles, and partial occlusion i.e. only part of the target image is captured by ATIS. 1. Obstacles disturbance is the most frequently happened case when the probe touch other objects to detect the react force and implement manipulation. In this experiment, the tracking performance under this situation will be tested at different focus planes. The target bead as shown in the center of Fig. 3.4 (a) is surrounded by two beads. They are fixed on petri-dish and make together the same sinusoid movement as in the previous experiment, along x-axis. The tracking error is defined as the difference between the detection and the ground truth divided by the working scale. The ground truth radii at different depths are determined the same as in the previous experiment. The ground truth x-y position is the detected position at focus plane. The mean and standard deviation of the tracking error are calculated during the sinusoidal periodic at different depths and displayed in Fig. 3.4 (c).

• Experiment Description

2. Partial occlusion usually happens when the probe touches other objects. The 3D structures of the objects may occlude part of the image of the probe. It also happens when free objects cross over the probe at different focus planes. This experiment tests how this tracking method will behave in these cases in a repeatable way. In the experiment, the nanostage moves along x-axis to make the bead image partial out of the view of ATIS. The tracking results under 30% and 60% occlusion ratio during a sinusoidal movement along the y-axis are shown in Fig. 3.5.

• Result Analysis

As shown in the Fig. 3.4 (c), the robustness for x-and y-axis within ±1 µm is satisfied with small errors and STD. While the radius detection hardly meets the demands. Specifically, lateral errors are less than 5% around focus plane with less than 5% STD compared to the working scale. As the z-position moves away from the focus plan, the tracking errors increase gradually. When the bead becomes very defocus, at z=-1, the tracking errors increase largely to less than 10 %. For the radius, the STD is always more than 200% compared to the tracking scale. The errors are acceptable within 0.5 µm around focus plane with less than 100% error. However as the image becomes defocus, the errors increase dramatically to more than 500 % when z=-1. Notice that the radii errors always have a positive value which means that the detected radii are bigger than the ground truth. That is because the events from the surrounding obstacles are counted for calculation, which is inevitable in this case.

The result of occlusion tests are shown in Fig. 3.5 (c). To sum up, The tracking algorithm is robust within some occlusion limitation i.e. 30% occlusion give satisfactory performance. However, some prevention must be taken to avoid major occlusion like 60%. Besides, the performance of x-y detection is better than that of radius detection. Specifically, under 30% occlusion, the errors remain about 5% with 5% for x-y. For radius, the errors are less than and ± 100 % with about 200% STD. Under 60% occlusion, the performance for x-y and radius becomes worse. The x-y errors increase largely to about 30 % when the image becomes defocus. And the errors of radius increase to about 200% with hugely increased STD.

Running Time Complexity

As explained, kHz processing rate is critical for obtaining faithful and stable haptic feedback. Thus the running time of tracking algorithm should be less than 1 ms. This part evaluates the running time and complexity of this algorithm.

• Experiment Description

Run-time analysis of an algorithm is to estimate the increasing running time with its input problem size. This parameter defines the processing speed of the algorithm. The steps of 1,2,3,5,6 in Algorithm 1 mainly did the judgment and assignment. Indeed, they request negligible processing time compared to the least square computation in step 4. Thus the problem size of this algorithm is the size of buffer U 1,m which determines how many coordinates are used for least square calculation. The goal is to find the relationship of the algorithm running time with the U 1,m buffer size. The program implemented in C++ under Linux is running in hard real-time. The testing relies on a 2.9 GHz Dual core desktop PC, with a total CPU load about 50 % of its power and a memory consumption of about 4 MB. The average running time under different buffer sizes from 62 to 4000 events are tested.

• Result Analysis

As shown in Fig. 3.6 , the algorithm has a linear characteristic, with the main errors of repeatability (STD) mainly come from the processing the steps of 1,2,3,5,6 in Algorithm 1. According to Fig. 3.6, with buffer size of 200 events, it needs less than 30 µs to run. Therefore, more than 30 kHz real-time sampling rate could be achieved with this algorithm.

Today, the data transmission of the ATIS camera is the limiting factor. Indeed, during the transmitting, the events have to be accumulated until 256, and then all of them will be sent when the user asks for data. The delay depending on the time required for accumulating 256 events, that means more dynamic events in the view cause no or less transmission delay, and vice versa. Progresses are made by the manufacturer in this direction.

Summary

In conclusion, a high-speed robust circle tracking algorithm is presented in this section. The tracking capability of the first OTs system which combines the tracking algorithm and optical hardware is evaluated. Experimentally, this system achieves till 33 kHz processing rate for refreshing the trapped probe position. Its calculating speed outperformed the conventional computer vision techniques at hundreds of orders. The tracking method handles well defocus from 0.5 to 1 µm, but lack robustness for more serious defocus. For an optical trapped 3 µm probe, at least a defocus range of ± 1.5 µm should be allowed. The optimized system in the next subsection will improve this performance. The measuring scales are 6 µm, 7 µm in x-and y-direction respectively before reaching the sensor edges, and 1 µm in Table 3.1: Probe tracking performance of the first system.

total for z-axis by considering the linear domain (± 0.5 µm). However, the STD is as high as 70% for those measurements, so this measurement of the z position have very limited performance which will be also improved in the following method.

Although this system can track 3D circles effectively close to the trap center, the limitation that comes from the defocusing phenomenon of the microscopy has not been addressed. Table 3.1 summarizes the performance of this tracking system. In the next section, a tracking system with bigger axial working scale and resolution will be presented.

Ring Tracking for the Optimized System

This section will prove that the final optimized system introduced in the previous chapter has a better performance than the first one. A modified probe tracking algorithm corresponding to the optimized image will be explained firstly. Then the same performances will be evaluated for this system using the same method as introduced in the Section 3.2.2.

Camera image of the micro-probe ATIS image of the micro-probe

The image from the first system The image from the optimized system Figure 3.7: The CMOS and ATIS image of probe under the first system compared to the image under the optimized system. The event accumulate time is 33 ms.

High Speed Robust Ring fitting Algorithm

As explained in the last chapter, the probe's event-based image in the optimized system is a ring instead of a single circle as illustrated in Fig. 3.1. Thus the 3D tracking problem turns out to determine the center position and radii of the eventbased ring. Therefore, ring fitting algorithm is required in this situation.

For obtaining the parameters of the ring, robust ring tracking algorithm is proposed here by modifying the Algorithm 1. Since the ring can be considered as two concentric circles, the main principle is also the circle fitting. However, special constraints are added for tracking the ring:

1. Assign to closest: an incoming event is assigned to the closer circle model. This condition makes the two circles' position update independently.

2. Reject all: throws away all the events in the ambiguous region. This condition makes sure that the inner and outer circle will not become into one.

3. Make the two circles concentric: the circle center position is determined by the inner circle since this circle is less disturbed by the surrounding obstacles during manipulation tasks.

The pseudo-code of the robust ring tracking algorithm is written in Algorithm 2.

Performance Analysis

In this subsection, the same experiments as in the previous section are performed to evaluate the working scale, robustness and running time of this optimized OTs system. The detailed experiment descriptions can be seen in section 3.2.2. This subsection will directly analysis the results in the following three parts.

• Working Scale

The working scale result is shown in Fig. 3.8. As expected, the x-y detection is almost the same as the last tracking system. Furthermore, great improvements are achieved for the radius detection. The working scale is extended from ±1 to ±2.5 µm, which is enough for out application. Besides, the STD compared to the working scale is much smaller than the previous system, decrease from 70 % to Algorithm 2 Robust Ring Fitting 1: for every 1 ms do 2: about 5 %, which provides more reliable detection. Specifically, x-, y-detection is linear with regression coefficients between pixel and micrometer are 42.3 and 42.4 pixel/µm respectively. The outer and inner radius of the bead at the focus plan (z=0) are 79 and 64 pixels respectively. It is noticed that each of the two radii have a quadratic relationship with the axial displacement within -2.5 µm to 2.5 µm. The coefficients of determination of the fit are 0.9919 and 0.9821 for outer and inner radius detection respectively. That means the quadratic models match well the variables. From 1.5 to 2.5 µm, the outer radius changes slowly while the inner radius changes quickly. That is mainly due to the appearance of image: from the diffraction image, the bead center becomes very bright at about 2µm, which leads to a big change for the inner radius, while the outer radius is not influenced too much.

If E t is in the predefined ROI(R 1 t ) or ROI(R 2 t ) 3: Update the U 1 1,m

• Robustness

The robustness is tested with the same experiment as illustrated in section 3.2.2, and the results are shown in the Fig. 3.9 and 3.10 respectively.

1. In the obstacles disturbance situation, the x-y detections are as robust as the first system. However, the radius detection is much more robust than the previous system with largely decreased STD. Concretely, x-y errors are less than ± 5 % with less than 2 % STD in 5 µ m. The radii errors are less than ± 20 % around the focus plane with less than ± 10 % STD. As the image plane move far away from focus plane, the tracking error increases up to about 30 % with less than 10% STD. The detection of outer radius shows the similar robustness than that of the inner radius.

2. In the partial occlusion experiment, the robustness of both x-y and radius detection is largely increased. Notice that occlusion ratio takes the outer circle as the reference, that means 30% and 60% of the outer circle is partial occluded during the experiment. In detail, the detection is more robust under 30% occlusion than under 60%. Under 30% occlusion, the tracking error and the STD are less than 5% for both laterally and axially detection. For the 60% occlusion, the x-y detection is still robust, and the outer radius most often has less then 20 % error with less than 20 % STD in the most time. On the other hand, the inner radius error shows huge fluctuations. This is because under 60% occlusion of outer radius means more occlusion for the inner radius, even the major part of the inner circle is occluded. Thus incorrect tracking is caused by the over occlusion.

• Running Time Complexity Similar to Algorithm 1, the least square computation in Algorithm 2 is still the most computational costly part. Thus the problem size of this testing is the size of the event buffer. The program is implemented in the same condition as the Algorithm 1, that is in C++ under Linux running in hard real-time, with nearly the same CPU load and memory consumption. Fig. 3.11 shows the average running time under different problem sizes. The running time of this algorithm is almost twice than the previous one. That is due to the least square computation is made twice to track the ring, which is composed of two circles. Thus this algorithm needs about 60 µs to run, twice than the former. It achieves more than 10 kHz real-time sampling rate of each probe position measurement. As expected, the algorithm also has a linear characteristic, and the main errors come from processing the other steps of Algorithm 2.

Summary

To summarize, much improved 3D tracking results are obtained for the optimized system. The axial working scale is improved to about 5 µm, which is triple times than the previous system with the same lateral working space. The detection shows hugely improved STD of 5% compare to 70% before. The 3D detection space is sufficient for our OTs application, since during the micromanipulation experiment, the displacement of the trapped target is limited by the optical trap and too much displacement will cause the loss of trapped probe and the nonlinearity of optical force model. Experimentally, this algorithm achieves more than 10 kHz sampling rate, which satisfies the requirement of kHz processing rate. This method shows largely improved robustness compared to the first system in both obstacle disturbance and partial occlusion situations. Particularly, the radius error in obstacle disturbance situation decreases from ± 100% to ± 5% during small occlusion i.e. 30%, and from nonrobust to less than 20% in large occlusion i.e. 60%. Therefore, this optimized OTs system with the corresponding algorithm is much more suitable for implementing on-line fast and accurate micromanipulation tasks. It is the first time to our knowledge that allow this performance. This system will be used for further application in the next chapter. Table 3.2 summarizes the performance of the optimized tracking system. The parameters of the first system are displayed again in order to compare.

Event-Based Multi-Circle Tracking

The current system is equipped with only one optical trap, while our future goal is to develop a more flexible and dextrous manipulation system with multiple trapped probes. 3D multi-probe tracking algorithm will be required then. This section presents a preliminary development of the 3D multi-circle tracking algorithm for future use. Instead of extending the above algorithm to multi-probe tracking, this section will propose a novel alternative event-based 3D circle tracking algorithm. Different from the Algorithm 1 and the Algorithm 2, this algorithm takes full advantage of the event-based acquisition paradigm and processes each event as the basic processing unit instead of a whole buffer of events. Therefore, it has the potential of even smaller running time, and more memory-saving. Ni et al. has presented a 2D multi-circle tracking algorithm using the single event processing mechanism, and achieved high 2D position updating rate of 30 kHz per event [Ni 2013b]. In this section, a even faster and more light weighted 3D circles tracking algorithm will be developed. This algorithm will be applied to a multi-circle tracking task, and its performances will be verified.

Problems Description

When it comes to the multi-target detecting, more problems need to be considered compared to the previous single target tracking: 1. Parameter estimation: which means the positions and radii of each single circle must be estimated. The novel event-based algorithm for parameter detection will be explained in detail in the following subsection. Figure 3.12: An example of the counting problem: these points can be considered as seven small circles as well as be grouped together to one big circle.

2. Counting: given a set of points, it is necessary to determine how many circles are presented. This is a substantially difficult problem. The answer depends on the type of model as well as its parameters constraints. An example is shown in Fig. 3.12, one could pass 7 small circles through every group of data, or directly fit a big circle to all the data.

3. Grouping : given the number of circles, determine which event comes from which circle. Generally, this problem is not independent from problem 1 and 2, since a usual method of judging whether some points belong together to one circle is to check if they can be well fitted to this circle. Specific judging details will be explained in the following subsection.

Algorithm

The proposed event-based circle tracking algorithm is inspired by the means-shift tracking algorithm, which iteratively locates the target position maximizes a probability density function (PDF) in the feature space [Comaniciu 2000]. In this algorithm, each E t is first assigned to the corresponding circle based on a distance criterion (as ROI method discussed in 3.2.1.2). Then the target circle's parameters will be updated. Every E t is be processed independently without any buffer, thus there is no memory consumption. This algorithm is briefly described as follows:

1. Assign event to the corresponding circle. The same ROI method as discussed in section 3.2 is used for judging if this event belongs to a certain circle. The ROI of each circle is predetermined, and the parameter is the same as in the Algorithm 1. For each incoming event E t , if it lies in the ROI of any of the circles in the circle list, assign it to this circle.

2. Update parameters. When a new E t is assigned to the nth circle in the circle list as sketched in Fig. 3.13, update the parameters of this circle C n t . The new center-coordinate C n t+1 : [X t+1 , Y t+1 , R t+1 ] is calculated as:

X t+1 = X t + α 1 • (x t -X t )(D (Xt,Yt),(xt,yt) -R t ) (5) Y t+1 = Y t + α 1 • (y t -Y t )(D (Xt,Yt),(xt,yt) -R t ) (6)
Furthermore the circle size is updated as:

R t+1 = R t + α 2 • (D (Xt,Yt),(xt,yt) -R t ) (7) R t E t (X t ,Y t ) (X t+1 ,Y t+1 ) R t+1 ROI Figure 3
.13: Working principle of the event-based circle tracking algorithm. If the incoming event E t lies in the ROI(C t ), then update the parameters from (X t , Y t , R t ) to (X t+1 , Y t+1 , R t+1 ) according to the criteria described in equation ( 5), ( 6), and ( 7).

where 0 <α 1 , α 2 <1 is a step size for updating. Too big α will cause overshoot and big noise in result. Experimentally, it is chosen to be less than 0.1. Unlike the common mean shift tracking, which maximizes the PDF of a solid region, this algorithm maximizes the density of position in a ring region as defined by the ROI. The D (Xt,Yt),(xt,yt) -R t acts as a weight function. This function determines the weight of nearby events for re-estimation of the circle parameters. Specifically, when D (Xt,Yt),(xt,yt) -R t = 0, that means E t located exactly on the C n t . According to the equations ( 5), ( 6), ( 7), X t+1 = X t , Y t+1 = Y t , and R t+1 = R t , which means the parameters of circle will not change. On contrary, as D (Xt,Yt),(xt,yt) -R t increases, that means E t is farther away from the C n t . Thus leads to a increasing update speed of [X t+1 , Y t+1 , R t+1 ] to promise a faster following. In the future, more kernels (e.g. Gaussian kernel) will be explored and tested. The step size of center position and radius: α 1 and α 2 can be set differently for a practical application.

3. Delete disappeared circles. The list of existing circles is scanned periodically every 10 ms. For the inactive circles, if the event number in their corresponding ROI is less than a threshold, then they will be deleted from the list. 4. Add new circles. The events not belonging to any of the circles are collected and counted independently according to its location. When the event number located in any background region R k exceeds a threshold in a period of time (10 ms in the experiment), then a new circle is created in that region, with initialized center position and radius. A new label will be assigned to this circle.

Compared to Algorithm 1, this algorithm consumes much smaller amount of memory since each event is processed directly without the need of buffering, Algorithm 3 Event-based 3D multi circle tracking 1: for every incoming E t do 2: if E t belongs to the ROI(C n t ) in the circle list 3: Compute the distance from the event to the center D (Xt,Yt), (xt,yt) . 4:

Estimate the C n t+1 : [X t+1 , Y t+1 , R t+1 ] according the equation ( 5), ( 6), ( 7). 5:

Update the current position to C n t+1 . 6: else 7:

put E t in corresponding background region R k . 8: end for and only the circle list has to be kept in memory. Besides, the processing time is much smaller since for each incoming event, only three equations are calculated. Furthermore, the running time increases very little with the increasing number of targets. Since this method will be used for probe tracking, probes overlapping will happen during manipulation. Thus this situation is not discussed in this algorithm. However, if this function is required in the other applications, the proper radius limitation and the maximum change threshold will help to prevent two circles from merging into one. Since this section is mainly focused on the algorithm's performance during 3D multi-circle tracking, a experiment in macro-scale will be conducted instead of under the microscopy. In the future, before using it to manipulation tasks in OTs system, the same evaluation experiments as illustrated in Section 3.2.2. will be implemented for quantitative parameters.

To resume, the complete event-based 3D multi-circle algorithm is summarized in Algorithm 3.

Performance Analysis

In this section, the event-based multi-circle tracking algorithm is used to track multiple moving circles with different diameters. Its precision and running time will be evaluated during this tasks in the following part.

Experiment Description

In order to prove the multi-circle tracking capability of Algorithm 3, a macroscale experiment has been implemented. This allows much more control of the experimental conditions than under the microscopy e.g. probes with different sizes, number of probes etc. A paper drew four circles of different sizes is used as the model. It is pasted on a motor and rotated with known speed, as shown in Fig. 3.14 (b). During the experiment, the camera faces the picture, thus the geometric relation between the model and the acquired information is an isometry. On ATIS, the moving trajectory for each circle is an ellipse considering the view angle. For the experiment, the motor is rotating at a speed of 50 rpm (revolutions per minute). The image is not put in the center of ATIS, thus due to the rotation, the circles will sometimes rotate out of scene. Thus to test the algorithm's ability of adding and deleting circles. Fig. 3.14 (a) gives a example of tracking process.

The step size parameters α 1 and α 2 for position and size translation respectively are experimentally set to be α 1 = 0.05 and α 2 = 0.001. They do not need to be changed according to the speed of the object.

Result Analysis

• Accuracy

The accuracy of this algorithm is defined as the ratio of the tracking error to target's radius. The result of one circle is shown and analyzed in the following part. In Fig. 3.15, the ratio of center position error to the radius of the circle is displayed during one motor rotation period. The mean is 3.3 %, with a STD of ± 3 %. It is observed that when the circle is fully inside the image (inset picture (b)), the errors are smaller. While at the moment of the target circle come in and go out from the scene, the tracking errors are bigger (inset picture (a) and (c)). This bigger error is caused by the partial occlusion of the sensor edge. The biggest error happens when the circle appears (inset picture (a)). That is because at that moment, a new circle need to be added, and there is a relatively big error between the initial position of the new circle and the real position of the target. It is noticed that when the target moves out, the tracking error does not increase too much, even in the situation of 90% occlusion. That gives a qualitative proof that this method is robust in x-y detection during partial occlusion disturbance. More quantitative tests will be conducted in the future. Fig. 3.16 shows the accuracy of radius detection during one motor rotation. Similarly, most often, tracking errors are less than 5 %, with STD 5.4 %. The biggest errors also occur at the beginning and the end of this set of data (inset picture (a) and (c)). The main reason is similar as in the above demonstration for the position tracking. It is noticed that when the target circle appears, the radius changes fast time (s) since the new added circle take very little time to fit the target. On the other hand, when the target circle disappears, the tracking errors increase gradually. We have the logical explanation: when the circle is partially occluded, the algorithm fits the events left on the image to a smaller circle to get a better approximation. This phenomenon could also be observed in the inset picture (c). To increase the robustness of the radius detection during partial occlusion, the maximum change limitation needs to be set. In Fig. 3.17, the tracking result is shown in spatio-temporal space. The blue line is the tracking result, and the red line is the ground truth. We can see that the detected center position fits well the ground truth and moves along the ellipse. They match together until before the end, where the tracking data (blue) stops since the target moved out of the view. When the circle appears, a big error on both position and radius are clearly seen from t=0s.

• Running Time

The experimental program is implemented in C++ under Windows on a 3.4 GHz i7 core desktop PC, running separately off-line. Since the event-based calculation is performed on each incoming event directly, the problem size of this running time test should be the number of processed events. The results in Fig. 3.18 shows the linear property of Algorithm 3. The running time increases with the number of events being processed. Processing each event is instantaneous: less than 0.05 µs.

This running time is much faster than that of Algorithm 1 (33 µs) and the 2D event-based circle tracking (32.5 µs) in [Ni 2013b]. The whole program is running with a CPU load less than 15% of its power. Fig. 3.19 displays the event numbers during experiment. A periodic change of event number can be observed, and the period matches exactly the rotation period: as the circles whirl into the view, an increasing number of events is produced with the increasing number of circles, as shown in the inset(c), and vice versa (inset(b)). When all circles come to the image, the number of events becomes about 300 /ms at first (e.g.inset (d)). While as the movement continues, the number shows big fluctuations from 300 to more than 800 events/ms. This is caused by the nonuniform motion of the spine motor, which leads to accelerations and deceleration of circles. When the circles move faster, more events are generated and on the contrary, slow circles produce fewer events. In total, the event number during 1 ms is from 200 to 1000 (rarely). According to the Fig. 3.18, less than 0.03 ms is needed to process these data in this situation. If these events come from 4 circles, 7.5 µs on average is needed for processing each circle in 1 ms. That is much smaller than any of existing circle tracking algorithms.

Summary

In summary, this section exploits the potential of the asynchronous event-based tracking by developing an event-based multi-circle tracking algorithm. It aims at tracking the arbitrary number of circles with arbitrary positions and radii. The event-based tracking method takes full advantage of the ATIS's asynchronous paradigm and implements the asynchronous and time-continuous computation presented in biological neural systems. The computation is data driven, thus provides a natural robustness to static occlusions. It allows 3D circle tracking at a minimum running time of 0.03 µs per event with the light weighted computational load and memory cost. The tracking precision has been tested on an indoor spinning model, where multiple circles with different radii appear and leave the view frequently. The mean error is most often less than 3 % and 5 % for center position and radius respectively. A bigger error of up to 20 % occurs when the target suffers from serious partial occlusion. More quantitative tests will be conducted in the future before using this algorithm for the micromanipulation tasks.

Method

Running time (µs)

Accuracy and STD (%) Comments

Event-based circle tracking algorithm 0.05/event (circle) 5 and ±5.4

Tiny memory cost and running time, position robust to occlusion

Table 3.3: Comparison of fast vision solutions under optical microscopy.

Both the extended Algorithm 1 (or Algorithm 2)and Algorithm 3 can be used for the 3D multi-circle tracking when there is only several targets. However, the latter is more suitable for tracking a large number of beads and provide high-speed real-time information with much lower time and memory cost. Other performances in terms of the robustness will be compared in the future. Table 3.3 summarizes the performance of Algorithm3.

Conclusion

In this chapter, three 3D high-speed probe tracking algorithms have been developed. The first two algorithms aim to track the probe in the first and the final OTs systems, as described in the previous chapter. The last algorithm is developed for the future multi-probe tracking tasks. In conclusion, 1. the developed event-based tracking methods have the ability to outperform the conventional tracking techniques in terms of computational cost (from tens of ms to less than 1 ms), memory consumption, and hardware price. By comparing the performances of the first and the optimized OTs systems, the optimized system is much more improved than the first one at the aspect of axial working space (enlarged from 1.4 to 5 µm), precision (the STD decreases from 70% to 5%), and robustness (the axial error decreases from 100% to 20% and 5% for obstacle disturbance and partial occlusion respectively). Significantly, this development provides a solid foundation for achieving 3D faithful haptic feedback during micromanipulation, which initiates a wide range of applications;

2. the novel asynchronous event-based circle tracking method is especially suitable to track probes at an extremely high refreshing rate of hundreds of kHz with tiny memory cost. The preliminary tests proved its precision and robustness. More tests will be made in the future before implementing it into manipulation tasks with multiple probes.

Besides, in order to track not only the circular objects, but also arbitrary shapes for future application (e.g. microswimmer, cells, biological organizations etc.), highspeed irregular shape tracking algorithm is studied. The result of tracking the non-linear transformation of an irregular shape object is displayed in Appendix A.

To sum up, we have found a cutting-edge solution for obtaining 3D robust highspeed force during the micromanipulation tasks. This development paves the way for achieving faithful and stable 3D manipulation with haptic feedback. It also bring the great potential to various applications as a powerful 3D high-speed force sensor. The final chapter will demonstrate two applications of this system. The primary application, haptic OTs will be developed by coupling the haptic device into the system in a real-time framework. The intuitiveness and effectiveness of this system will be verified by several biological experiments. Besides, some preliminary work will be carried out on exploring the complex biological structure, where the OTs will be used as a powerful 3D high-speed force sensor. In this chapter, the first 3D high-speed haptic OTs will be established. Several biological experiments are implemented to verify this system's effectiveness and 3D force sensing capabilities. Besides, 3D high-speed OTs will perform as a force sensor in another application: exploring the 3D biological site automatically. Motion planning inside unknown 3D vessels using the force signals will be demonstrated in simulation.

Real-time Haptic Optical Tweezers

As explained in the Chapter 1, the real-time high-speed haptic feedback enhance largely the intuitiveness and effectiveness of the micromanipulation process. Thus in this section, haptics will be coupled into the previously developed system in the real-time framework. Therefore, faithful microforce will be perceived by users, and the position of microtargets will be controlled intuitively.

System Description

The overall 3D real-time high-speed haptic OTs system mainly consist of four parts as displayed in Fig. 4.1:

• The optical microscope for realizing OTs and the optimized ATIS image.

• 3D high-speed target tracking algorithms.

• Haptic coupling for microforce perception and position control.

• Real-time computational system.

The first two parts have been developed in Chapter 2 and Chapter 3 respectively. Therefore the real-time system and haptic coupling will be developed in the next two subsections.

Hard Real-Time System Realization

The requirement of the real-time high-speed loop has been illustrated in the Chapter 1. Briefly, the low detection speed and irregular signal processing rate will deform or taint original force signals in frequency domain, even cause instability to the system. This subsection will introduce the realization method of this real-time system.

In broad terms, a real-time system is defined as a computer system that must satisfy bounded response time constraints, or else, the system will risk serious consequences e.g. weapon delivery failure [Phillip A. Laplante 2016]. There are mainly three kinds of real-time system: 1. hard real-time systems where tasks are critically The kernel approach to realize hard real-time in Linux operating system. required to be completed on time. 2. Soft real-time systems, which is less restrictive on punctuality. 3. Firm real-time system which is the system between soft and hard real-time system. Here we choose the hard real-time system so that the OTs can be used to detect the real dynamics in the microworld with more deterministic sampling rate.

Normally, all general-purpose operating systems are nonreal-time, e.g. Windows, Mac OS and standard Linux. Many schemes are available for realizing hard real-time in Linux operating system, e.g. RTLinux, RTAI, Xenomai etc. Here we choose Xenomai to realize the hard real-time processing environment since it has better punctuality performance than the other methods [Brown 2010]; allowing long-term maintenance; and its core is usable for building any kind of real-time interfaces.

The basic idea for making the standard Linux to hard real-time is to run a highpriority real-time kernel between the hardware and the standard Linux as shown in Fig. 4.2. When there are real-time tasks (high priority), the normal Linux processes (low priority) are suspended, and the real-time tasks are executed by this real-time kernel. The scheduler of the real-time kernel treats the standard Linux process as low priority, that means normal Linux tasks will run when they are given chances to run. Since the hard real-time kernel has a higher priority, the standard Linux tasks are possible to be preempted at any time by a real-time task.

Experimentally, the punctuality of Xenomai hard real-time system is much better than the nonreal-time system. During the experiment of [Blaess 2015], the system time interval of 1 ms of both nonreal-time and Xenomai systems are read during a period of time and counted as shown in Fig. 4.3. It is observed that for nonreal-time system, many time intervals are far away from 1 ms, ranging a big scale from 0 to 2 ms. The situation is even worse with disturbance. While for Xenomai, almost all the time intervals are distributed around 1 ms equally. It is further proved in [Blaess 2015] that even with a lot of disturbs, this system still maintains the similar punctual performance. The deterministic processing rate promise the detection of dynamic force in a bound sampling period.

The design and implementation of the practical real-time system requires numerous attentions including: dealing with the issues of concurrency and synchro-nization; distributing computing systems and determining each task deadline; specifying system fault tolerance etc. All these works need to be carefully designed and analyzed. Most of these structures are built in collaboration with a postdoc. In summary, The operating system is developed on a single PC (Intel Xeon core, 2.93 GHz), running a hard real-time co-kernel Linux and RTOS APIs Xenomai. 4 threads are constructed in the processing corresponding to the haptic, the nanostage, the ATIS sensor, and the vision algorithm respectively. During the experiments, they all run at the sampling rate of 1 kHz.

Haptic Coupling

This section will show how to couple the haptic interface into the micromanipulation system. Thus to control the movement of the microobject by haptic device, and obtain the haptic feedback during the micromanipulation process simultaneously.

The haptic interface is integrated with the 3D nanostage (PI, P562.3CD) and the force detection by the technique called "bilateral coupling" [Yokokohji 1994]. As illustrated in the Fig. 4.4, the "bilateral coupling" for the micro and macro world means that:

1. Position coupling streamline: The users give the 3D position translation orders, P hand , by moving the haptic interface. Then, these translations are scaled by G pos = 2 × 10 -4 and sent to the nanostage. The nanostage along with the samples fixed on it move according to the command, while the OTs stays at the same position. Thus the optically trapped probe is capable of the full space exploration, that is in a range of 200 × 200 × 200µm 3 thanks to the nanostage workspace.

2. Force coupling streamline: When the probe contacts with obstacles, the position of the trapped probe will be changed relative to the by OTs due to the counter force. The displacement of probe P laser -P sphere is detected by ATIS and the tracking algorithms. As explained in the first chapter, the OTs can be approximated as a linear spring within the certain scale, and there is a linear relationship between the optical force and probe displacement. Thus the contact force F opt is calculated by multiplying the detected displacement from the tracking result and the linear coefficient, which is the stiffness of OTs K. The next section will explain the procedure of calibrating K. Then, optical forces are scaled by G f orce = 1 × 10 12 to convert the optical force in pN to a perceptible force in N and transmitted to the users through the haptic device.

Additionally, during the haptic control, users can choose two operating modes between position or speed mode. Specifically, in the position mode, the position of user's hand through haptic interface and position of nanostage have a linear relationship. When the interface stop moving, the nanostage will stop as well. On the other hand, in the speed mode, the position of the haptic device command the velocity of nanostage. That means when the position of the user's hand is not at the origin, the speed of nanostage is non zero correspondingly. So that the nanostage will continuously move toward the direction of the displacement until the user's hand comes back to the origin. As a result, a large moving scale equals to the whole space of the nanostage can be obtained by the speed mode control. The speed mode is used for approaching any target in the workspace of nanostage. When the probe is close to a target, the position mode will be used for conducting manipulation tasks.

Omega 7 (Force Dimension) is used as our haptic interface. It has small inertia mechanical structure and the high degree of freedom (DOF): 6 DOF for translation and rotation along x-y-z directions and 1 DOF for active grasping. It will be particularly suitable to complex tasks with multi-finger in the future. In our application, only the 3D translation of the haptic handle along x-y-z is used to control the 3D position of the probe. Omega fed back the force obtained from the OTs according to the optical trap stiffness. We will explain the calibration of this parameter in the following subsection.

Calibration

The stiffness of OTs K is required for calculating the optical force from the probe's displacement according to the OTs force model [Ashkin 1987]. The process of determining the stiffness of OTs is called "OTs calibration" in this thesis. For an optically trapped microsphere, a simple spring model is applied to describe the optical force exerted on the target:

F opt = K × (P laser -P probe ) (8)
Practically, to obtain the laser position, the positions of trapped probe before touching anything are recorded for a period of time and the the average position is considered as the position of OTs, which is around (120,150) in the ATIS coordinate. To determine the stiffness, we use the equipartition theorem [Neuman 2004]. For an object in a harmonic potential:

1 2 k B T = 1 2 K d 2 (9)
Where k B is Boltzmann's constant, T is absolute temperature, and d 2 is the displacement variance of the particle from its trapped equilibrium position. Thus, by measuring the brownain motion variance of the trapped bead, the stiffness K can be obtained. Experimentally, with the 30 mW laser, the OTs stiffness in x-axis, y-axis and z-axis are K x = 12.3pN/µm, K y = 12.6pN/µm s and K z = 1.5pN/µm respectively under room temperature of 25.5 • C. The axial stiffness is less than the lateral stiffness which is normal according to [Rohrbach 2005]. This indicates that during manipulation, the loss of the optical trap is more likely caused by the reacting force in the axial direction, since the trap is week in this direction. The same laser power will be used for the following experiments, under the same condition, i.e. temperature, liquid and so on. In the future, stiffness adjustments may be required depending on the specific applications, and then the calibration process will be re-conducted for obtaining the OTs' stiffness.

Until now, the OTs system with faithful 3D high-speed haptic feedback is established. The system's parameter is summarized in the Table .4.1. In the following section, this system will be verified by several biological experiments.

Haptic loop

Trap stiffness (pN/µm)

Haptic Feedback in A Biological Application

In this section, the effectiveness of the 3D haptic OTs system in biological applications will be shown. Convincing results will prove that the 3D high-speed haptic feedback provide the 3D microforce in complex conditions to users effectively. Specifically, several biological experiments aim at touching the contour of cells will be demonstrated. Firstly, the method of preparing the biological sample will be introduced in the following subsection. Then, two experiments will be implemented to quantitatively show the system's effectiveness.

Sample Preparation

The red blood cells (RBCs) are chosen as our experiment object due to its simplicity of acquisition, and the interesting dumbbell-shaped profile. However, great difficulties are encountered during the preparation of the sample. These problems and solutions will be detailed in the following content.

Difficulties

For preparing the proper sample for experiments, several problems are encountered as follows:

1. Fix cell on the bottom. In order to discover the shape of the target cell, its position should be fixed on the bottom. This is generally not a problem for adherent cells which grow on cover slip. However for suspension cells, i.e. red blood cell, great difficulties are encountered.

2. Keep the 3D shape for a period of time. Hemoglobin leakage is commonly encountered during hours of experiment. Since the red blood cells have no nuclear, when the leakage happed, the 3D structures of RBC are ruined and they become flat as shown in Fig. 4.5. A lot of methods have been tried to fix the cell on the petri dish and also to keep the original shape. For example, biological glue is used for sticking suspension RBC, while it causes all the cells stick to the bottom immediately in a flat shape instead of the dumbbell-shaped profile. Centrifuge is also used for swing the cell on bottom, however, hemoglobin leakage often happens after 20 min experiment which is too short for the whole experiment. Finally, a proper solution has been found, and it will be detailed in the next part.

Solution

The solution is developed for sticky the RBCs on the petri dish and keeping their 3D shape for a couple of weeks. This method will be used for all the following experimental sample preparation.

• Material 1. Alcian Blue Solution, 1% Aqueous; 2. Distilled water; 3. Phosphate Buffered Saline (PBS); 4. 4% Formaldehyde Solution in PBS; 5. Petri dish.

• Procedure 1. Take a petri dish for the experiment. Add Alcian Blue solution to the chamber, and incubate for 20 minutes at room temperature. Ensure that the Alcian Blue solution is distributed over the entire surface area of the well.

2. After the incubation, remove Alcian Blue from the petri dish, and rinse thoroughly with distilled water 4-5 times. Then wash with PBS twice.

3. Red blood cells of a mouse are taken and put into a container. 4. Wash cells with PBS. Centrifuge at 400 r/min for 5 minutes after washing. Remove PBS supernatant by aspiration. 5. Gently add 4% formaldehyde in phosphate-buffered saline (PBS) twice the volume of blood cells into the container. Incubate for 15 minutes to fix the shape of cells.

6. Remove formaldehyde from the top of the liquid.

7. Count the cells and adjust to 1 x 10 6 cells/mL in a PBS. Pipette 1 mL of the cell suspension into the Alcian Blue pre-treated chamber.

8. Put the petri dish with liquid in the clean bench which is sterile and ventilate. Then leave the liquid to evaporate. 9. After the liquid evaporated, we obtained the sample with fixed 3D RBC for the subsequent experiments. This sample can be used for several weeks.

Experiment 1: Touching RBC with 3D Haptic Feedback

In the first experiment, we explore the RBC in different ways. To begin with, we will squeeze the cell from the top, and then, we will follow their shape from the sides. The system design should prove a effective 3D force sensation to users in different situations.

Z-axis Haptic Feedback

The first experiment will prove the possibility of axial touch of the RBC with the haptic feedback. Before the experiment, the probes (3µm polystyrene) are incubated in PBS and Ethylenediaminetetraacetic acid (EDTA) liquid to prevent the surface sticking. The protocol of the experiment is that: the RBC sample controlled by the user via the haptic interface, first moves below the optically trapped probe without touching. Then the sample gradually moves upwards along the z-axis until the probe touch the cell and users feel an obvious counterforce in z-axis from the cell. Then separate the cell and probe by moving the nanostage.

The experimental results are shown in Fig. 4.6. The nanostage moves 2 µm during this process, and an obvious axial force of more than 1 N is sensed by the users. The corresponding optical force changes from 0 to about 1.7 pN with small fluctuations caused by the Brownian motion and very low level of noise. In detail, the touching process is divided into three sections: before touch (I), start to contact (II) and deep contact (III).

The probe starts at the top of the cell (green point) as displayed in inset picture (a) I. The corresponding photo is shown in figure (b) I. The initial force in z-direction is around 0 as seen in the figure(c) region I. Small fluctuation is caused by the 3D Brownian motion of probe and the movement of the user's hand. The z-position of nanostage is shown in (c) with the orange line.

Then the probes move gradually to the cell by increasing the position of the nanostage, until first touch the top of the RBC. The touch point is shown in process II in (a), (b), (c), where the nanostage increased about 0.5 µm. At the contact point, a suddenly increased counterforce in the z-axis of about 0.3 pN is detected. And the Brownian motion becomes smaller compared to the process I.

Then the probe is pushed deeper into the cell thus obtain a bigger indention force until 1.6 pN in the z-direction, as shown in III. At this time, the users felt about 1.6 N force in the z-axis.

After that, the probe is retrieved and the counterforce decreases until back to 0 which means the probe doesn't contact with the cell anymore.

During the whole process, the position of nanostage totally corresponds to the force change: the higher position of nanostage, the deeper probe goes into the RBC, meaning bigger optical forces applied on the probe. The cell indention is made twice during this experiment. Similar results are obtained during the two indention process which also proves the repeatability of the axial force detection. More convincing results will be shown in Experiment 2.

Notice that this experiment aims at proving the effectiveness of axial haptic feedback during biological applications. In addition, the stiffness of the RBCs can be roughly obtained from the result. By using the Hertz model , which treat the cell as a homogenous smooth semisphere, the elastic modulus of the RBC is calculated as 33.7 Pa [Guz 2014]. This result is similar to [Planus 2002], where measured the elastic modulus of adherent living cells by OTs in axial direction. However, this result is two orders smaller than some other RBC elastic modulus measurements implemented by AFM [START_REF] Cross | [END_REF]]. The difference is caused by the different indention depth of the OTs and AFM [START_REF] Tomaiuolo | Biomechanical properties of red blood cells in health and disease towards microfluidics[END_REF]]. In the future, for more accurate biological stiffness detection, the OTs should be calibrated above the cell before testing [Nawaz 2012], because the biological objects will affect the OTs' stiffness thus influence the detection result.

3D Haptic Feedback

This experiment is dedicated to touch the 3D contour of cells and explore their shapes. The difficulty of these tasks comes from the uncertainty contour of the biological objects. Since the visual information may be blurry or lost at some parts, the haptic feedback will help users to maintain the contact and decrease the possibility of losing trap. In the first section, we explore one isolated RBC and in the second section we explore 2 RBCs attached together by following along a ∞ shaped path around them.

The 3D path of probe and 3D contact forces during surface exploration are shown in 4.7 (a), 4.8 (a) respectively. The photos during these procedure are displayed in 4.7 (b), 4.8 (b), and the 3D force curves are shown in 4.7(c), 4.8 (c). From the figure, it is observed that the 3D contact forces are maintained at the order of pico-Newton. Besides, the 3D force direction indicates a counterforce from the cell's surface. Particularly, for the two-cell exploration, when the probe pass through the connected part of two cells, a burst of axial force of 1 pN is detected as shown in 4.8 (a) and (c) pointed by the red arrow. This can be explained logically: since this connection part is higher in z-direction than the regular path of the probe. Thus when the probe crossing over this part, a force in z-axis is detected to show that there are obstacles below the probe.

In conclusion, the above experiments demonstrate the effectiveness of this 3D faithful manipulation system. The 3D obvious contact force is successfully perceived by operators. Under the help of haptic feedback, some incomplete vision information is compensated, and users are assisted to achieve a surface exploration with consistent force. It is the first time that the complex 3D biological surface is touched by human with a faithful haptic sensation. In the next subsection, more convincing user experiments will be conducted.

Experiment 2: Proof of Concept

To further prove the repeatability and effectiveness of the 3D haptic feedback during biological manipulations, a users study has been implemented in this subsection. The task of touching the shape of one RBC is made by 6 volunteers. Both the experimental data and the questionnaire about user experience will be analyzed to provide more convincing results.

Description

6 participants were gathered from the Institut Systemes Intelligents et de Robotique (ISIR). Each of them completed a task of exploring the shape of the same RBC in three different conditions: with only vision feedback, both vision and haptic feedback, and only haptic feedback respectively. Before the formal experiment, the participants were trained to use this system e.g. how to trap a probe, how to use the haptic device to control the position, how the haptic feedback works etc. About 10 minutes is required until they feel confident about the touching tasks. Before the experiment, participants will be told to ensure the contact between the probe and cell during the whole touching process. Task under each condition was conducted twice.

A pre-experiment questionnaire is required to be finished before the experiment, which is mainly about the basic information of participants. Besides, a post-experiment questionnaire will be required after the whole experiment, which focused on the user experience. The specific contents of questionnaires are attached in Appendix 2. In the next part, the results will be analyzed.

Results and Discussion

As investigate by the pre-experiment questionnaire, there are 6 experimental participants: 2 females and 4 males. The age ranges from 24 to 31 with an average age

Microscopy

Micromanipulation Haptics 2.7 3 3.2 Table 4.2: The average score of users' expertise in the field of microscopy, micromanipulation, and haptics. The scores are given from 1 (very bad) to 5 (very good).

of 26.5. They were all right-handed, and have the engineering background about robotics. The participants' expertise in the field of microscopy, micromanipulation, and haptic interface are collected from the pre-experiment questionnaire. To sum up, expertise in this three field was between average expertise (score=2.5) and good expertise (score=4). Specifically, the average scores in these three fields are shown in the Table . 4.2.

Under three experimental conditions, the participants performed differently during the exploration process. As shown in Fig. 4.9, the probe trajectory during three users' experiments are displayed in (a), (b) and (c). The green, blue and yellow lines corresponding to the touching process guided by only haptic, only vision and both haptic and vision respectively. As observed, by using only the vision, the probe is not always in contact with the cell contour compared to the experiment with both vision and force. The reason is that by only observing the microscopy image without force information, the shadows and invisible part of microobject may give wrong illusions to users. In (a), the user lost the trap at indicated by the red cross due to big contact force which is probably caused by some blurry parts of the cells that hardly noticed by the inexperienced users. By only using the haptic without visual information, more efforts are required to finish the tasks. Two successful intermittent force which leads to the smaller force with bigger irregularity. With both vision and haptic feedback, the maximum and average force is in the medium.

That is due to in this procedure, users kept contact with the target, and the contact force is maintained within a proper scale. The haptic feedback largely decreases the rate of losing trap as shown in the (d) to 14 % compare to the others (30% and 45% for only vision and only haptic respectively).

The results regarding the user experience collected from the post-experiment questionnaire are presented in Fig 4 .11. In conclusion, the averaging score of the system's intuitiveness, usefulness and user's sensation are 4, 4.7 and 3.8 respectively, which is between good and excellent level. Besides, most of the users found that our system is helpful for conducting micromanipulation tasks and exploring microworld.

Since the experimental subjects are of certain age and education, the results of this study cannot be extended to the general population in a statistically significant way. However, our results verified the usefulness of haptic feedback to a certain group of peoples in academic and engineering fields. In the future, more statistical experiments and analysis will be conducted to more people in the field of microscope, biological research, and microrobotics etc.

Summary

To summarize, this section verified the intuitiveness and effectiveness of the 3D haptic OTs system in the biological tasks through two experiments.

In experiment 1, the RBCs are touched by the operator via the haptic interface, and the 3D obvious contact forces are well perceived. It is the first time that users explore the biological non-planar surface through 3D faithful haptic feedback.

In experiment 2, the task of experiment 1 is reconducted by several users for obtaining more repetitive results. It is proved that with the assistance of our system, even non-expert in micromanipulation field could finish complex biological tasks within short training time. Furthermore, with haptic feedback, users have better performances in the micromanipulation tasks: the number of undesired collisions is reduced, the counter force is kept within reasonable scale with small force irregularity, and probe escaping rate is largely decreased.

Motion Planning Using OTs Force Sensing

The previous contents have verified the effectiveness of 3D high-speed haptic feedback in biological tasks. The established system demonstrates a promising potential for future small-batch manipulation. However, the haptic manipulation is not always efficient in some situations. For example, considering tasks in complex environment e.g. searching or delivering objects in an unknown labyrinth-like 3D biological structure, the user-guided exploration procedure would be too laborious and time consuming, or more likely, exceeds totally the capabilities of the human. Therefore, automated OTs capable of exploring in 3D complex environments is required, where one of the fundamental problem is the 3D fast motion planning.

The up-to-date automated OTs motion planning researches use the vision to provide the environment information. However, they are mostly restricted to 2D simplified conditions. There are two main reasons:

Firstly, difficult to obtain the 3D complex structure image under microscopy. As discussed in the Chapter 1, due to the shallow depth of field of microscopy, the out of focus and complicated 3D constructions are very difficult to determine;

The second difficulty lies in the low light contrast of the biological objects. It poses huge challenges for the optical camera to capture the full 3D structures. Although 3D microscopies are proposed to achieve the 3D image of microobjects, their popularity in OTs system is very limited due to the complexity of realization.

Under such circumstances, we propose to use the probe as a 3D high-speed force sensor to detect the 3D environment information and realize the motion planning.

To understand this, one can picture a scenario as a robot searches a target in a dark labyrinth with only force sensor. It touches a wall for updating the map without any risk of harming its safety. This method overcomes the defocus and low contrast limitations encountered by the visual sensor. Additionally, from the 3D high-speed force signal, the actual biological structures as well as their mechanical characteristics e.g. stiffness, viscoelasticity can be exploited. Thus by combining the conventional vision information with the force information obtained from our method, more comprehensive understanding of the biological site will be obtained.

As the preliminary study, this section aims at developing a simulation of exploring the 3D biological vessel inner structure with a probe as force sensor. Specifically, this section is organized as follows: the problem formulation will be descried firstl followed by the state-of-the-art of motion planning algorithms in OTs. Then the method of 3D motion planning using force signals will be described. Finally, the proposed method will be verified in a simulated CAD biological vessel model. Notice that this work is in collaboration with a post-doc, who was more in charge of the algorithm development. For more details about the algorithms, please refer to the appendix C.

Problem Formulation

We aim to develop an efficient solution for 3D motion planning with only force signals in OTs system. That means, vision is not used for 3D structure identification, and non a priori information about the environment is available before planning. We rely on force sensing during the search to determine an efficient path between the initial and the goal configuration without penetrating into obstacles.

Concretely, two problems need to be solved: 1. Searching. Giving the initial and the goal configurations in an unknown environment, determine an efficient path between these two states.

2. Mapping the 3D environment. When multiple initial and goal configurations are required in the same environment, the most efficient way of planning is to con-struct the map first. So that the subsequent planning process in such precomputed map will cost much less time.

In the next section, the algorithms for addressing these problems will be investigated.

State of the Art

Originally referred to as Piano Mover's Problem, motion planning can be considered as a kind of search in robots' multi-dimensional configuration space which contains implicitly represented obstacles. Researchers developed many efficient solutions to respond to single-(one initial and goal configuration) or multiple-query search (multiple initial and goal configurations). In the past decade, significant efforts are devoted to applying motion planning technique to achieve automated OTs. The following content summarized some of the most popular planning algorithms.

Wu et al. applied one of the most popular heuristic searching algorithm

A* [Hart 1968] to generate collision-free path for transporting living cells using OTs [START_REF] Wu | [END_REF]]. Tao Ju et al. applied Rapidly-Exploring Random Tree (RRT) algorithm [LaValle 1998] to generate planning roadmap for cell transportation [Ju 2011a][Ju 2011b]. The research work [START_REF] Banerjee | [END_REF]] developed a planning approach for uncertainty position detection by applying a partially observable Markov decision processes (MDP) algorithm on multiple trapped silica beads simultaneously. The research work [Chowdhury 2013] applied a dynamic planning algorithm: D* Lite [START_REF] Koenig | [END_REF]] to transport yeast cells in a microfluidic chamber.

In our situation, non a priori information is available beforehand as input to the planner. As search being processed, the topological graph of environment has to be constructed incrementally. In such situation, using A* algorithm is significant inefficiency since as long as the discrepancy are detected between the constructed topological graph and the actual environment, replanning is required. Similarly, Rapidly-Exploring Dense Tree (random or deterministic RDT) in mainly applied in static environments. This algorithm's advantage lies in the nonholonomic motion planning , which is not required in our case for the holonomic probe since all the constraints that it is subjected to are integrable into positional constraints. The uncertainty of probe position is not our current concern. Since the difference between the actual probe position and the detected position is greatly reduced by our 3D high-speed robust tracking method. Finally, since it is straightforward to consider the incrementally-revealed environment as dynamically emerged obstacles, the dynamic heuristic planning algorithm D* Lite is suitable for our scenario. This algorithm is an optimal path finding planner, which is far more efficient than the A* in dynamic environments. Therefore, in our study, we will mainly use the D* Lite combined with a proposed strategy to resolve the force based motion planning problem. The specific methods will be introduced in the following section.

Planning Method with Force Signals

As explained, we rely on the OTs' force sensing ability as collision detector to construct the topological graph. For protecting the probe and biological obstacles during the frequent contact, as well as improving the quality of force signal, we introduce the retreat-if-contact strategy. The principle of this strategy is similar to the tapping mode in AFM [START_REF] Hansma | [END_REF]]: when the probe encounters an obstacle during the search, instead of stay contact, it will first come back to its last noncollision configuration, then continue searching. This strategy prevents the tip from sticking to the surface, and avoid deforming the structure which will produce distorted force signals. Here, we tolerate slight contact with obstacles and accept a weaker notion of obstacle avoidance. The configuration space of probe and the geometry representation of the biological structures are specifically studied in C.2 and C.1.

Search Method

For generating an efficient path between the initial and goal configuration in an unknown environment, we developed a method by combining a modified D* Lite algorithm and retreat-if-contact strategy. More details about the algorithm is shown in appendix C.3.1, and the following content mainly explain the working principle.

Given a search query, the method must first generate a path using D* lite in the configuration space, assuming all unknown map-cells are obstacle-free. In each sampling period, the probe's next configuration is loaded from the planner, and then the probe conducts the corresponding motion to arrive at this configuration. If the probe encounters a collision, the retreat-if-contact strategy is applied, which means the probe comes back to its last non-collision configuration, and updates the partially-constructed topological graph. The efficient replanning is then carried out by D* lite planner. Such process will be repeated until the goal configuration is reached or no possible solution is reported.

Mapping Method

As explained in the 4.3.2, a preconstructed map of the environment can improve the efficiency of multiple-query motion planning.

We apply greedy algorithms [START_REF] Cormen | [END_REF]] as mapping method based on the following consideration: for a complex structure like biological tissue, the mapping could take a significant amount of time. In the middle of such procedure, interruption by any exceptional events, emergency stop or malfunctions could occur. For example, the probe may escape optical trap sometimes when mapping the environment using force. Therefore, this requires the mapping algorithm to be robust, which means first, the planner must be able to take into account the partiallyconstructed map, instead of restart mapping procedure. Secondly, it must not rely on the previously stopped configuration. Specifically, for instance, when the trap is lost during mapping, a new probe needs to be trapped. For a robust mapping algorithm, the new probe does not need to come back to the original place where the trap is lost, or mapping from zero. It should be able to continue the mapping tasks from any place.

The essence of the greedy algorithms is to recursively move the probe from current configuration to lowest-cost neighbor configuration that is not observed yet. The motion will still use the modified D* Lite and retreat-if-contact strategy to deal with the probably dynamics in livings, until the overall environment is mapped. Such method can be easily extended to distributed search by multi-traps [Koenig 2001], i.e., cooperative mapping using multiple probes. For more detail about the algorithm, please refer to the appendix C.3.2.

Physics Consideration

The current method is limited to the simulation on the algorithmic level. However, when applying them in real biological implementations, practical physics must be taken into consideration. For example,

• To make probe detectable by ATIS for force sensing, the biological structures must be tested by a minimum level of transparency or conducting the fluorescence treatment to the probe.

• The stiffness of biological structures must be higher than that of the OTs. So that during the navigation tasks with force sensing, the probe will not deform biological structure.

• Force threshold must be defined to distinguish the contact with biological structures or other dynamic motions (thermal agitation, Brownian motion, drag force, fluid force etc.).

• Motion velocity must be regulated at a level to reduce the risk of lost-trap during too fast motion or in retreat-if-contact situation.

• Probe must be chemically processed to prevent being sticky to biological tissue in the case of contact.

• The fluid inside the biological structure should not flow fast. Or else the fluid force pose great difficulties to maintain a stable trap and cause noise to the force signals.

All above problems needed to be studied quantitatively before real implementation. The physical environment must be carefully determined. These will involve a set of consideration and measurements.

Simulation and Result

This subsection will verify the searching and mapping method in a simulation. The test environment is the virtual biological vessel network as will be described in the next subsection 4.3.4.1. For searching, we are particularly focused on the time Figure 4.12: CAD model of a biological vessel. spent on planning/replanning, for each sample period as well as for the overall search process. For mapping, the whole 3D inner structure of the modeled vessel will be demonstrated.

Biological Vessel CAD Model

The role of this CAD model is to simulate the 3D structure of a vessel system. It emphasizes structural complexity but does not simulate physical phenomena from the microworld. As shown in the Fig. 4.12, a 3D biological vessel environment that contains multiple branches with various dimensions is simulated with the physical engine in Blender 2.77. This hollow structure simulates capillary network or any other biological vessel structures with complex branches and narrow passages. The overall structure can be enveloped by a rectangular box, which is scaled to a workspace in biological site with dimension 253 × 195 × 57 µm 3 . The maximum inner diameter, located at the root, is within 25 µm. To limit accessibility, all vessel walls are configured as obstacles that prevent the probe from penetration. When the probe encounters such obstacles, its motion immediately stops and the physics engine of Blender generates a collision signal and send it to the planner.

Result

The 3D motion planning simulation is tested on a laptop workstation equipped with an Xeon E3-1505M v5 processor (2.80 -3.70 GHz), RAM of 16 GB, as well as a graphic card Nvidia Quadro M1000M. In the program, the elapsed time spent on algorithm is recorded by using high resolution clock provided by the chrono library in the C++11 standard. The program runs in soft real-time environment.

Single Search Query

Various search queries have been tested, and results show that the motion planner is competent to resolve the problems within a reasonable amount of time. Here, in The overall simulation takes 5 minutes and 56 seconds to finish, while the cumulative time spent on planning is only about 14 seconds. The most amount of time is mainly spent on communication between blender simulation and planner. The time spent on planning at each sampling is shown in Fig. 4.15. The first planning takes 530 ms to finish, this process is equivalent to the efficient heuristic A* algorithm It must be understood that time spent on planning varies greatly with the algorithm input (partial-constructed graph, query pair, etc.). So it is completely normal that Fig. 4.15 depicts varying time cost at different sampling. One other important remark is that the planning algorithm and probe motion are not concurrent i.e., planning algorithm won't be executed until the position of the probe is in a steady state, and similarly, the motion of probe won't start until the decision is made by planning algorithm.

Mapping

With the mapping method described in the section 4. has fewer branches because many narrow capillaries are not accessible due to the geometric constraint of the probe.

Summary

In this section, we developed the 3D motion planning methods for automated OTs exploration. Instead of using vision for sensing the environment, we propose to apply the 3D high-speed force sensor into planning algorithm to overcome situations that cannot be handled by vision, and obtain more mechanical information of the biological structure. This section is mainly focused on the algorithm development rather than physical realization. As the preliminary implementation, the planner interacts with a blender simulation environment. Results demonstrate that our method can efficiently resolve the search query, though difficult conditions are imposed (no a priori information, large workspace, local minimum, etc.). In order to enable multiple search query efficiently, we construct the map of the environment by applying the greedy algorithm which is robust and efficient. The 3D map of the inner structure of vessel is obtained. In the future, we would like to implement this method to the real system and to handle autonomous tasks in complex 3D microenvironments.

Conclusion

The main work of this chapter can be divided into three parts: first, the haptic feedback loop is established in the hard real-time framework on basis of the previous achievements. As a result, the OTs with 3D high-speed faithful haptic sensation for biological micromanipulation is developed; second, several biological experiments are conducted to verify the effectiveness and intuitiveness of this system. The 3D interaction forces are percept by users through the haptic device. Complex tasks of exploring the 3D and nonplanar RBC's surface is demonstrated for the first time. In addition, convincing results from several participants shows that the haptic feedback assistants the nonprofessional users to fulfill the tasks in micro scale less laboriously, more efficient and more repetitively. Compared to the teleoperation without haptic feedback, the successful rate using our system is increased by reducing the possibility of losing trap, maintaining the contact force in certain scale, and decreasing the undesired collision etc. last, a preliminary work of using OTs as a force sensor to automated explore the 3D complex environment is demonstrated. This technique is competent to the situations where the visual sensor cannot handle. This method is proved in a simulation by exploring the inner structure of 3D vessels. The constructed map of the 3D biological environment can be used for not only automatic exploration, but also for the 3D visual augmented haptic manipulation. In the future, it will be used in the real 3D biological structure exploration e.g. cytoskeleton inside a cell, neuron network etc.

Conclusions and Future Works

To implement dexterous manipulations and explore complex biological environment, flexible and effective micromanipulators are required. However, the up-to-date automatic or teleoperated manipulation systems are not well suited to high flexible tasks or non-intuitive enough. To bridge this critical gap, in this thesis, the first 3D high-speed haptic OTs system for biological manipulation has been developed. This system provides not only visual information, but also faithful 3D haptic feedback to users during manipulation tasks. In this work, we improve the 3D force detection workspace (from 6 × 7 × 1.4µm 3 to 6 × 7 × 5µm 3 ), and robustness (the error decreases from 100% to 20% and 5% in the situation of disturbance and occlusion respectively). The processing time for force rendering is less than 1ms. The haptic loop remain at 1 kHz at hard real-time frame work, thus result in a faithful and stable the force sensation. Consequently, unprecedented complex biological tasks have been successfully performed. Convincing results from several experiments show that users fulfill the micromanipulation tasks more repetitively and effectively under the help of our development. In particular, subject performances have been improved in the aspect of: maintaining a contact force between the probe and object within a defined scale, reducing the trap loss, and increasing the number of successful trials. During the haptic manipulation, no special expertise or considerable training time is demanded. Therefore, this approach is both time and cost efficient to on-line biological research and manipulation.

Finally, a number of potential shortfalls need to be considered. First, the axial exploring space is limited. This constraints could account for the working distance of the current objective (0.2 mm). Considering the future in vivo applications where the OTs need to penetrate into thick biological tissues, this axial space may need to be enlarged. Possible solution may be to change the objective with bigger working distance, or to use laser actuators to move the OTs in axial direction. Second, the downside factor regarding our experiment is the lack of experimental data and statistic analysis. More experiments will be conducted in the future for more statistically significant results. Third, it can not be ruled out that the slow movement of nanostage might cause delay or even instability in the control loop. Therefore, in the current system, users should keep their hand moving smoothly without sudden acceleration. Future working into solving this problems by using a faster actuator to control the position of OTs is already underway. Lastly, ATIS is still used in low contrast condition. In order to improve this situation, phasecontrast microscopy may be considered for generating image with higher contrasts. Now that 3D faithful haptic OTs with single probe has been achieved, the technique should be transfered to multiple trapping, because that will make a vast number of applications possible. However, many new challenges will appear such as the realization of 3D multi-trap. Various approach has been proposed to realize multi-trap in the OTs system. The galvanometers are a feasible solution combined with time-sharing technique. The galvanometer scanning mirror capable of reflecting the laser to different directions at high-speed, thus generating multiple OTs. One important advantage of this technique is that the galvanometers are reflexive and reversible optical components. With this, the laser path and light path can be kept on the same path. So that different generated multi-traps can always be maintained inside the ATIS field of view. However, these switching mirrors are limited to 2D space. For 3D motion, additional lens moving along axial direction or focus tunable lens for axial steering may be required [Tanaka 2013]. As an alternative, Spatial Light Modulator (SLM) is a promising way to create multiple optical traps [Grier 2003]. SLM modulates a single laser up to hundreds of independent optical traps in 3D space. The state-of-the-art fast hologram generation during phase modulation can be realized up to hundreds of Hz [START_REF] Gibson | [END_REF]]. While the main concern of using this method lies in how to determine the moving laser center, which is indispensable for resolving the force feedback. Besides, tracking multiple probes simultaneously will result in lower force sensing resolution.

Multiple 3D high-speed haptic probe will facilitate many exciting applications. For example, the ultra-flexible "microhand", which consist of multiple simultaneously controlled probes act as fingers of a human hand. It can then be used to hold and manipulate complex biomedical samples [Chowdhury 2014]. The previous developed multi-probe tracking algorithm can be used here for rendering the force exerted on each probe. In addition to that, multi-trap also promotes the manipulation with "microtool end-effector". In this scenario, the optical traps are not meant to manipulate the singel object independently, but are used to hold a mechanical microtool with more complex structures. The microtools are the real end-effectors but they usually do not include force sensors [START_REF] Phillips | [END_REF]]. Parallelized 3D haptic OTs are easily adaptable to hold end-effector holder and act as powerful force sensor. The forces exerted on on each trapped areas will be distributed on the complex structure. Different forms of end-effectors can be attached to different configurations of traps, and implement 6D interaction motion. This will surely open a brand new way for efficient force sensing without integrating complex and expensive force sensors into end-effectors.

The second perspective to this work is automated exploration of 3D complex structure with force signal. The motion planning method and the simulation of exploring inside a 3D vessel structure have already been shown in this thesis. In the future, this need to be implemented in the actual OTs system to explored real 3D biological structures (e.g. cytoskeleton inside a cell, neuron network molecular structures etc.) for map construction, drug delivery, 3D augmented vision during teleoperation and so on [Nelson 2010]. The high-speed robust haptic signal will provide not only the geometrical structure, but also the mechanical information at each detection such as stiffness and adhesion due to its high-frequency sampling rate. The scientist might benefit to understand not only the 3D geometry structure, but also structural mechanical property variations while exploring 3D unknown or invisible samples. Therefore, a better understanding of the fundamental phenomenon will be achieved and might lead towards pharmaceutical solutions to treat diseases.

The third future perspective may be the "position control" through 3D highspeed object tracking. For automatic close-loop control in OTs, as well as the other microrobotic system, the highly dynamic forces from the microenvironment often need to be compensated (e.g. high-frequency vibration [Avci 2015]). With the developed high-speed probe and shape tracking algorithm, many applications are available such as automatic position and orientation control of objects ( e.g. cells, microactuators, biological tissues etc.), automatic tune of the stiffness of the trap to match the desired properties according to the tasks, compensate the Brownian motion and high frequency vibration during automatic control and haptic manipulation.

Besides, more improvements need to be done in the field of the haptic interface for manipulation. To render high dynamics that occur in the microworld to users, the ideal bandwidth of the haptic interface is required to be at least 1 kHz. The current employed device Omega.7 has limited bandwidth because of its limited rigidity and non-negligible apparent inertia. This situation can be significantly improved by applying the method described in the recent work [Tianming Lu 2016], where the key idea is to add a high frequency vibration transducer upon a conventional haptic interface to construct a dual-stage architecture. By using signal crossover for haptic channel combination, this method achieves high bandwidth with uniform magnitude response without sacrificing workspace or degree of freedom. Based on this idea, a vibration transducer can be embedded inside the handle of the Omega.7. With proper development on dynamic modeling, identification and compensation, a dual-stage high bandwidth haptic interface can be achieved.

A last, OTs with intuitive haptic interactions for biological use are commercially available. However, more effort should be made to simplify the experimental procedure for users. For example, the function of automatically trapping a probe, displaying the 3D position of the probe and sample in a more intuitive way, preventing the contamination inside the chamber during long time experiment, maintaining the suitable environment for biological livings and so on. Additionally, more biological applications should be investigated. By collaborating with other laboratory or industrial partners, advantages of the intuitive force feedback on biological, biomedical and nanotechnological applications should be continuously studied and demonstrated to the public in order to popularize the technology. In general situation, the event number in no more than 200, that means this method will cost 0.07 ms in the maximum situation, and achieves 14 kHz. The whole program is running with a CPU load less than 15% of its power. c-space, i.e., about 2.8 million configurations, as well as the upper bound of the number of vertices in the topological graph. In our study, considering the spherical form of the probe, the c-space can be computed by growing the obstacle mapcells (vessel wall) by the radius of the probe, and can be expressed in R 3 , i.e., the Cartesian coordinates. Before integration into planning algorithm, metric space must be defined, which requires distance function between two arbitrary configurations. With the c-space defined above, it is straight-forward to adopt L ∞ metric or L 2 metric (Euclidean distance). For each configuration, its neighboring configurations must be defined to prepare for edge construction in the topological graph. Here, the 3-neighborhood [LaValle 2006] is applied for most direct traversal. An analogy can be drawn that all 26 blocks of a Rubik's cube are neighbors to its kernel with the notion of 3neighborhood.

Events number

C.3 Planning Algorithm

Since vision is not considered and no a priori information on environment is provided, we decide to take advantage of OTs' force sensing ability as collision detector to incrementally construct the topological representation. For this purpose, we introduce the retreat-if-contact strategy: when the probe encounters an obstacle during search, the planner brings it back to its last non-collision configuration, updates the partially-constructed topological graph and continues searching. Here, we tolerate slight contact with obstacles and accept a weaker notion of obstacle avoidance.

C.3.1 Search Algorithm

Given a search query, the algorithm must firstly generate a path in configuration space, assuming all unknown map-cells are obstacle-free. This process is supposed to be as efficient as the heuristic A* algorithm. In each sampling period, the corresponding configuration from the generated path will be loaded, based on which a motion command is sent to positioning system. If the received force signal indicates a collision (it could be due to a dynamic obstacle or undetected obstacle), the retreat-if-contact strategy is applied and an efficient replanning must be carried out. Such process will be repeated until the goal configuration is reached or no possible solution is reported in the previously defined resolution.

Based on the logics discussed above, we combine the retreat-if-contact strategy with a modified D* Lite algorithm to answer to search query. The pseudo code, described in high abstract level, is shown in Algorithm 1. It highlights state transition (state in the sense of state machine), and sophisticated details are omitted. Compared to the original D* Lite, we apply the index priority queue as binary heap in order to favor operations on arbitrary elements of the heap in case of update procedure. The index is obtained by encoding the 3D Cartesian space. Advanced data structure as Fibonacci heap could be a better option, which reduce

Figure 1

 1 Figure 1.2: Two examples of micromanipulation systems. (a) A microrobotic system for micromanipulation and microassembly [Piat 2011]. (b) A closed-loop 3D magnetic manipulation system [Oulmas 2016].

Figure 1

 1 Figure 1.3: OTs' application in biological field. (a) OTs are used to measure the force during kinesin motor stepping and structured RNA molecule unfolding [Fazal 2011]. (b) The mouse embryonic stem cells are patterned into precise 3D structures using holographic OTs. Images were taken in bright field and fluorescent confocal microscopy (green labelled images) [Kirkham 2015].

  Figure 1.4: Automatic cell manipulation with OTs. (a) A group of yeast cells are moved to the assigned position without collisions [Sun 2012]. (b) A cell is transported with the 3-bead gripper formation controled by the OTs automatically [Chowdhury 2014].

Figure 1

 1 Figure 1.5: Teleoperation and haptic manipulation using OTs. (a) A user is using an iPad application to control the position of multi-particles through the OTs [Bowman 2011b]. (b) A user is using a haptic device during micromanipulation tasks. Haptic feedback assists him to acquire an intuitive force sensation when touching microobjects [Ni 2013b].

Figure 1

 1 Figure 1.6: Dedicated microactuators with force sensors. (a) SEM image of electrothermal three-beam microgrippers fabricated in single-crystalline silicon. The inset shows a magnified top view [Nordström Andersen 2009]. (b) Capacitive MEMS six-axis force-torque sensor which allow deflections and rotations along the x-, y-, and z-axis [Beyeler 2009].

Figure 1

 1 Figure 1.8: 3D multiple particles tracking using DIHM. (a) Experimental scheme for recording optical inline holograms. A Laser is used to illuminate the aqueous solution of 10 µm diameter polystyrene spheres and imaged digital camera. (b) A hologram of multi-particles during experiment sequence and (c) their reconstructed 3D positions [Latychevskaia 2014].

Figure 1

 1 Figure 1.9: Principle of the CLSM and the captured 3D image. (a) Scheme of a confocal microscope. The excitation and emission pinholes are imaged onto the sample to define an illuminated point in the sample. Detector detects the light from only that point. The scan mirror scans the illuminated spot across the whole sample to obtain the 3D image [Thorn 2016]. (b) A 3D image of human umbilicalvein endothelial cells from a confocal microscope (credit: Wong/Searson Lab)

Figure 1

 1 Figure 1.10: Principle of the LSFM and the captured 3D image. (a) A light sheet microscope. An illumination objective is used to form a thin sheet of light that illuminates the sample. The sheet of image is detected by camera [Thorn 2016]. (b) 3D images captured by a LSFM [Chen 2014].

Figure 1

 1 Figure 1.11: Principle of the stereo-microscopy and an example of captured image. (a) Scheme of the stereo-microscopy imaging system. Two incident lights with an angle is used to image the sample and project the left and the right image on camera. (b) A stereo image of a cube composed of 2 µm silica microspheres [Bowman 2010].

Figure 1

 1 Figure 1.12: Photos of high-speed tracking instruments. (a) A photo of quadrant photodiodes (image from FirstSensor). (b) A photo of high-speed camera (image from E1 Camera). (c) A silicon retina prototype (ETHZ, Switzerland).

  Bram van den Broek et al. reported a technique for tracking multi gold nanorods in live cells with several nanometer resolutions in 3D [van den Broek 2013]. The processing time is 50 ms in 2D and 500 ms in 3D. Multi objects of arbitrary shape could be tracked simultaneously with digital image, as shown in Fig. 1.13.

Figure 1

 1 Figure 1.13: Examples of microobjects tracking using the digital images. (a) Tracking of magnetic microrobot using bright-field microscopy [Steager 2013]. (b) The microbead formation starts pushing a cell toward the goal G [Banerjee 2014].

Figure 1 .

 1 Figure 1.14: Illustration of particle tracking based on radial symmetry. (a) A simulated CCD image of a point source with shot noise. (b) A noise-free highresolution simulated image. The red 'X' indicates the true center. (c) The gradient of the intensity (orange arrows). Circles indicate the pixel centers. Yellow lines are drawn through each midpoint, parallel to the gradient. (d) The point of minimal distance to the yellow lines in c, indicated by the orange circle, provides an accurate, analytically calculable estimate of the particle center location [Parthasarathy 2012].

Figure 1

 1 Figure1.15: The 3D position detection of the particle using the photodiode. (a) Lateral detection: the illuminated difference between the two halves of the photodiod is proportional to the particle's displacement. (b) Axial detection: when the trapped particle moving closer to the microscope objective, the light signal is decreased. On the contrary, when the particle moves away from the microscope objective, the light signal is increased[Bowman 2013].

Figure 1

 1 Figure 1.16: Event based target tracking. (a) An accumulation map of events generated by DVS. Red and blue dots represent events of different polarities. The inset shows the correspondent scene in a frame based image [Ni 2013b]. (b) An accumulation map of events generated by DVS. Black dots represent the events, the green shape shows the tracking model. The inset shows the correspondent scene in a frame based image [Ni 2012].

Figure 2

 2 Figure 2.1: (a) A photo of the ATIS circuit layout. (b) An enlarged photo of ATIS pixel array. (c) The principle scheme of ATIS pixel: the relative light intensity to certain amplitude will spike positive or negative events.

Figure 2

 2 Figure 2.2: A photo of human face and the corresponding event-based image. (a) A photo from digital camera. (b) The event-based image with a fixed accumulation time interval of 33 ms. (c) The event-based image with the fixed accumulation time interval of 1 ms.

Figure 2

 2 Figure 2.3: The Gaussian blob trackers (red ellipse) follows a pen thrown in the air. Events are also activated when the person on the left of the scene is moving. They are successfully tracking the person's features and eventually disappear in absence of motion [Lagorce 2015].

Figure 2

 2 Figure 2.5: Left: the scheme of the experimental setup to evaluate the response frequencies of ATIS under low light contrasts. Right: the table of the optimized bias parameters.

  Figure 2.6: (a) Picture of ATIS and the stimulated event-based light spot. (b) The average event numbers of each row in 1 s under different stimulative frequencies. (c) The average event numbers in 1 s of each column under different stimulative frequencies. (d) An example of determining the active area at 1 kHz stimulative frequency. A threshold of 10 is set to the curve to obtained the length and width of the active square respectively.

  Figure 2.7: (a) The active areas change with stimulative frequencies. (b) Number of events collected from the whole ATIS and active area with stimulative frequencies.

Figure 2

 2 Figure2.9: ATIS's response frequency. Inset picture: the enlarged intersection part. Black line with error bar shows the average number of events generated per modulation in active area with the standard deviation. The purple dotted line is the average events number per modulation in the optimum area of 5 pixels × 5 pixels. The blue dots indicate the maximum number of events per modulation. The red line is the threshold of 2 events: one event is generated at the brightness rising edge, the other is generated at the brightness falling edge.

Figure 2

 2 Figure2.10: (a) The maximum (green) and average (blue) response frequencies of active area with the light contrast changing from 0.5 lux to 10 lux. (b) PE stands for the positive event and NE stand for negative events. The green and red lines are the average number of events in the active area during 1 s. The dark green and dark blue lines are the maximum number of events in the active area during 1 s.

  Figure 2.11: (a) The light path of this first OTs setup. (b) A photo of the first system.

Figure 2 .

 2 Figure 2.13: First row: the digital images of probe at different z-displacement respect to the focus plane (z=0). Second row: the corresponding ATIS images of probe with 33 ms accumulation time. The displacements are in µm.

Figure 2

 2 Figure 2.14: (a) The light path of the inline holographic OTs setup. (b) The photo of one digital hologram from this system.

Figure 2

 2 Figure 2.15: The digital images of probe at different z-displacement respect to the focus plane (z=0) and the corresponding ATIS images with accumulation time of 33 ms. The displacements are in µm.

Figure 2

 2 Figure 2.16: The image of the probe touching other objects from CMOS camera and ATIS. (a) The probe is close to the obstacle. (b) The probe is in contact with the obstacle.

Figure 2

 2 Figure 2.17: (a) Light path of the optimized OTs setup. (b) A photo of the system.

Figure 2

 2 Figure 2.18: First row: the CMOS image of probe at different z-displacements respect to the focus plane (z=0). Second row: the corresponding ATIS image of probe with 33 ms accumulation time. The displacements are in µm

  Description . . . . . . . . . . . . . . . . . . . . . . . 50 3.2 Circle Tracking for the First System . . . . . . . . . . . . . . 50 3.2.1 High-Speed Robust Circle Fitting Algorithm . . . . . . . . . 50 3.2.2 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . 54 3.2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . Speed Robust Ring fitting Algorithm . . . . . . . . . . 61 3.3.2 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . 61 3.3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . Description . . . . . . . . . . . . . . . . . . . . . . 66 3.4.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.4.3 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . 70 3.4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Figure 3

 3 Figure 3.1: The image of probe under the first OTs system. The left picture is the photo from the CMOS camera. The right picture is the corresponding image captured by ATIS with the fitted circle. The event accumulation time is 33 ms.

  Figure3.2: The FIFO event buffer. When the newest inlier event is pushed in, the oldest event in the buffer is popped off.

Figure 3

 3 Figure 3.3: The force measurement working scale of the first OTs system. From (a) to (c): the circle tracking result of x-, y-position, and radius (in pixel) with the probe position (in µm). The mean of data during sinusoidal movement with STD are shown with linear regression.

Figure 3

 3 Figure 3.4: Robustness of the first OTs system in obstacles disturbance environment. (a) Image recorded by the CMOS camera. (b) Corresponding image recorded by ATIS with accumulation time of 33 ms. (c) Tracking error at different depths.

  Figure 3.5: Robustness of the first OTs system under the situation of partial occlusion. (a) Image recorded by CMOS camera. (b) Corresponding image recorded by ATIS. (c) 3D tracking error under 30% and 60% occlusions compared to the circle diameter at different depths.

  ± 5% laterally; ± 100% axially. 30% occlusion: ± 5% laterally; ± 100% axially. 60% occlusion: non robust 1.4

Figure 3

 3 Figure 3.8: The force measurement working scale of the optimized system. (a), (b) The ring center position x and y (in pixels) with the probe's position (in µm). The mean with STD during a sinusoid movement are shown with linear regression. (c), (d) The outer and inner radius (in pixels) with the probe's axial position (in µm). The mean in a sinusoid movement with STD are shown with quadratic regression.

  Figure 3.9: Robustness of the optimized system in the obstacles disturbance situation. (a) Image recorded by the CMOS camera. (b) Corresponding image recorded by the ATIS. (c) 3D tracking error at different depth.

Figure 3

 3 Figure 3.10: Robustness of the optimized system in the partial occlusion situation. (a) Image recorded by the CMOS camera. (b) Corresponding image recorded by the ATIS. (c) 3D tracking error under 30% and 60% occlusion compared to the outer radius at different depths.

Figure 3

 3 Figure 3.11: Running time of Algorithm 2 with different buffer sizes.

Figure 3

 3 Figure 3.14: (a) ATIS image of the experimental model. The events are shown in black points, and the tracking results are shown in red circles. Events are accumulated in 10 ms. (b) The experimental setup: 4 different circles are drawn on a paper and glued on a rotating motor.

  Figure 3.15: The tracking error of the circle position. (a) The target circle appears gradually in the view. (b) The target circle is moving in the image along an ellipse trajectory. (c) The target circle is moving out of the view gradually.

Figure 3

 3 Figure 3.16: The tracking error of the circle radius. (a) The target circle moves into the view gradually. (b) The target circle is moving in the image along an ellipse trajectory. (c) The target circle moves out the view gradually.

Figure 3

 3 Figure 3.17: Tracking results during one motor rotation are shown in the spatiotemporal space. The detected circle positions and radii are shown in blue lines, yet the ground truths are shown in red lines

Figure 3

 3 Figure 3.18: Running time ofAlgorithm3 with the event number.

  Description . . . . . . . . . . . . . . . . . . . . . . . . 78 4.1.2 Hard Real-Time System Realization . . . . . . . . . . . . . . 78 4.1.3 Haptic Coupling . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.1.4 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.2 Haptic Feedback in A Biological Application . . . . . . . . . 83 4.2.1 Sample Preparation . . . . . . . . . . . . . . . . . . . . . . . 83 4.2.2 Experiment 1: Touching RBC with 3D Haptic Feedback . . . 84 4.2.3 Experiment 2: Proof of Concept . . . . . . . . . . . . . . . . 89 4.2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.3 Motion Planning Using OTs Force Sensing . . . . . . . . . . 93 4.3.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . 94 4.3.2 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.3.3 Planning Method with Force Signals . . . . . . . . . . . . . . 96 4.3.4 Simulation and Result . . . . . . . . . . . . . . . . . . . . . . 97 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

  Figure 4.1: The overall system scheme. The system consists of 4 main parts: optical setup; 3D high-speed tracking; haptic coupling and real-time framework.

  Figure 4.2: The kernel approach to realize hard real-time in Linux operating system.

Figure 4 . 3 :

 43 Figure 4.3: The histograms of 1 ms time intervals of different systems. (a) Timer in the non-real time system with no disturbs. (b) Timer in the Xenomai system with no disturbs [Blaess 2015].

Figure 4

 4 Figure 4.4: The schematic diagram of the bilateral coupling of the position and force in the haptic OTs system.

Figure 4

 4 Figure 4.5: Picture of RBCs under microscopy. (a) Normal appearance of RBCs. (b) The RBCs become flat and transparent after the hemoglobin leakage.

Figure 4 . 6 :

 46 Figure 4.6: Optical forces and positions during the process of pressing a RBC from the top. (a) The 3D force applied on the probe during the experiment. The x-, y-, z-coordinates are the position of nanostage in the world coordinate. inset picture I: the optically trapped probe is not in contact with the cell; II: by increasing the position of the nanostage, the probe is coming into contact with the RBC; III: the probe moves deeper into the RBC by further increase the position of the nanostage. (b) Pictures of RBC and the probe corresponding to the three procedure of I, II, III. The scale bar is 3 µm. (c) The optical force and the position of nanostage in z-direction during the cell indention process.

Figure 4

 4 Figure 4.7: Touch the contour of a RBC using the probe. (a) The 3D path of the probe and the 3D contact forces during the RBC contour exploration. (b) Pictures of this process under microscopy. The scale bar is 3 µm. (c) 3D optical force during this process.

Figure 4 . 8 :

 48 Figure 4.8: Touch the contour of two connected transparent RBCs. (a) The 3D path of probe and 3D contact forces during the contour exploration. (b) Corresponding pictures under microscopy during this process. The scale bar is 3 µm. (c) 3D optical force.

Figure 4 . 9 :ForceFigure 4

 494 Figure 4.9: The trajectories of the probes during the experiment of RBC surface touching. (a), (b), (c) The result of three participants. The trajectories under different experiment conditions: touching with only visual feedback, only haptic feedback, and both haptic and visual feedback are shown in different colors of green , blue, and yellow respectively. The scale bar is 3 µm

Figure 4 .

 4 Figure 4.11: User experience during this experiment.

Figure 4 .

 4 Figure 4.13: Single search query pair. Left: the query pair in the CAD model. Right: 3D representation of the query pair inside the planner (unit in µm).

Figure 4 .

 4 Figure 4.14: Search process at different time stamp: (a) 0'40"; (b) 2'57" when a difficult local minimum is encountered; (c) 4'33"; (d) 5'56" when the goal is reached. Yellow points indicates occupied map-cells.

Figure 4

 4 Figure 4.15: Time spent on planning at each sampling. The peak in the middle indicates that the algorithm finds out the previously-planned path is obstructed by a difficult local minimum and plans a new path to escape.

  Figure 4.16: 3D representation of mapping result.

Figure

  Figure A.1: The tracking error when tracking a freely moved shape. (a) Stable movement and less optical changes lead to a tracking error with smaller STD. (b) Intermittent motion of hand causes bigger STD.

Figure

  Figure A.2: Running time of shape tracking method with the event number for least square calculation.

Speed Haptic Optical Tweezers Contents 1.1 Haptic Optical Tweezers

	Chapter 1
	3D High-

. . . . . . . . . . . . . . . . . . . . . 3 1.1.1 Working Principle and Properties . . . . . . . . . . . . . . . . 4 1.1.2 Manipulation Strategies . . . . . . . . . . . . . . . . . . . . . 7 1.1.3 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.2 Force Sensing through 3D Tracking . . . . . . . . . . . . . . 11 1.2.1 Realization Using Monocular View . . . . . . . . . . . . . . 11 1.2.2 Realization Using 3D Image . . . . . . . . . . . . . . . . . . . 14 1.2.3 Realization Using Stereo Vision . . . . . . . . . . . . . . . . . 16 1.2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3 High-Speed Haptic Feedback . . . . . . . . . . . . . . . . . . 18 1.3.1 Necessity of High-Speed Haptic Feedback . . . . . . . . . . . 18 1.3.2 High-Speed Force Detection . . . . . . . . . . . . . . . . . . . 19 1.3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Table 1 .

 1 ]. 2: Comparison of the high-speed tracking techniques under microscopy. *These frequencies are the information refreshing rate, rather than image frame rate.

	Method	Acquisition (KHz)	Processing (KHz)	Pros		Cons
	High speed camera + Small FOV	10	1	Multi-target; multi-feature	Constrained working space
	High speed cam-era + Simple algo-rithm	10	1	Fast		Limited simple tasks	to
	High speed camera + Hardware accel-eration	10	1	Multi-target; multi-feature	Non-efficient
	Photodiode	1000*	1000*	High accurate	speed;	Unsuitable to manipulation tasks
	DVS	1000*	30**	High low consump-speed; tion	No static in-formation

**Achieved in

[START_REF] Ni | Asynchronous event-based high speed vision for microparticle tracking[END_REF]

Speed Optical Tweezers System Construction Contents 2.1 Event-based Vision

	Chapter 2
	3D High-

. . . . . . . . . . . . . . . . . . . . . . . . 27 2.1.1 Silicon Retina . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.1.2 Principle and Properties . . . . . . . . . . . . . . . . . . . . . 28 2.1.3 State of the Art: Event based Target Tracking . . . . . . . . 30 2.2 ATIS Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.2.1 Experimental Method and Design . . . . . . . . . . . . . . . 33 2.2.2 Response Frequency under Low Light Condition . . . . . . . 34 2.2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.3 OTs for 3D High-Speed Force Sensing . . . . . . . . . . . . 39 2.3.1 The First Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.3.2 Inline Holographic Setup . . . . . . . . . . . . . . . . . . . . 42 2.3.3 Optimized Setup for Biological Application . . . . . . . . . . 45 2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

[

  Yuan 2016] Wenzhen Yuan and Srikumar Ramalingam. Fast localization and tracking using event sensors. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 4564-4571. IEEE, 2016. (Cited on page 29.) [Zhang 1997] Zhengyou Zhang. Parameter estimation techniques: A tutorial with application to conic fitting. Image and vision Computing, vol. 15, no. 1, pages 59-76, 1997. (Cited on page 52.) [Zhang 2008] Zhipeng Zhang and Chia-Hsiang Menq. Three-dimensional particle tracking with subnanometer resolution using off-focus images. Applied Optics, vol. 47, no. 13, pages 2361-2370, 2008. (Cited on page 12.)

Subject

Mean force (pN) STD (pN) Maximum force (pN) examples are shown in Fig. 4.9 (a) and (c). However, concentrating on the force is demanding and laborious, thus a longer time is needed during this manipulation.

The trap is lost in the (b) shown as a red cross. That is due to lack of the vision channel information, the users can hardly estimate the potential danger during the exploration and do not make the correct decision. By combining the haptic feedback and vision, the above shortages are overcame. The visually ambiguity objects can be detected by the haptic feedback, at the same time, the vision provides the overview information for estimation in advance. What's more, users fulfill the presented task less laboriously, faster and more repetitively. By observation, the probe paths are closer to the contour of the cell. Also, the optical trap is less frequently lost.

The data during a period from the probe touching the RBC until leaving the RBC is collected and analyzed. It is considered that the probe is always in contact with the cell during this process. The specific data of mean force with STD, and the maximum force of each user during this process is shown in the Table 4.3. In three different conditions, the average of each users maximum force, average force and STD are shown in 4.10 (a), (b) and (c) respectively. The percentages of losing trap among in these conditions are shown in (d). It is observed in (a), (b) and (c) that with only haptic feedback, the contact force is the biggest. However, the force irregularity is the smallest. On the contrary, with only vision feedback, the contact force is minimum yet the force irregularity is maximum. The reason is that with only haptic feedback, users rely on the touching sensation to guide their decision, and they prefer to have a bigger force at all the time. Thus they achieve big force and small force irregularity. However, with only visual information, the probe is not always in contact with the cell, also some unnoticed part of the cell may cause

General Shape Registration

The Chapter 3 mainly studied the event-based tracking method for the circular target. However, considering the future more general shape tracking tasks e.g. micro-actuator, biological organs, cells [START_REF] Gressin | [END_REF]]etc., arbitrary shape tracking algorithm are required. In this Chapter, the event-based general shape tracking method will be studied.

A.1 General Shape Registration

In order to wider the event based tracking method to not only track circular objects but also other shapes for the future application, an irregular shape tracking method is explored in this section. The fundamental problem of shape tracking is the shape alignment. It aims at aligning a source shape as close as possible to a target, thus to obtain a global transformation between these two poses. The most standard solution for shape tracking is the Iterative Closest Point (ICP) algorithm [START_REF] Besl | [END_REF]] . This iterative algorithm has three basic steps: 1. find closest points to each given points; 2. compute the registration that minimizes the mean square error between these two point sets; 3. transform the given points. The three above steps will be iterated until the mean square error between the data and the model fall below a threshold.

The classic ICP algorithm is an efficient and accurate approach for rigid registration. It allows only for translation and rotation. As discussed in the previous chapters, the scale of the target is a crucial hint for z-position detection. Besides, considering the possible non-rigid transformation such as shearing, squeezing etc. which are caused by the counter force, affine ICP algorithm will be used in this work [Du 2010]. Benefit from the event-based acquisition mechanism, the moving shape is encoded by the events. Thus these collected events can be directly used as feature points and conduct the alignment. The iterative steps are performed during tracking. That means the data set of the current calculation will be the model set of the next calculation and so forth. Therefore, the event-based shape tracking is much more computationally efficient than the conventional ICP due to the redundancy-free data.

A.1.1 Shape Registration Algorithm

A pre-constructed model is set initially to be near the position of the target. For each event (mt,nt) . All the data set is collected in the buffer U 1,k . After that, and make the affine transformation between the two point sets is calculated based on the following least square criterion:

Where A is the affine matrix which contain 6 parameters. At least 3 correspondence are required for solving this equation. After obtaining an estimation of the affine transformation matrix, the current point set will be transformed to the target place, and conduct the next iteration.

A.1.2 Performance Analysis

A.1.2.1 Precision

In order to track the nonlinear transformation of the target, including the scaling, translation and rotation and shearing, the experiment is conducted in the macro life. An arbitrary model is drawn on a paper and held by a person to moved freely. The tracking result is shown in the following contents.

The points alignment accuracy is evaluated by computing the mean distance between the model data and the corresponding event data every 500 µs. shows the tracking error ratio to the with respect to time. This method achieves mean errors of 2.62 %, with standard deviations of 0.43 %. It is noticed that sometimes the error shows a bigger standard deviation e.g. inset (b). That is caused by the non-uniform motion of the image, especially when rotating due to the movement of hand.

A.1.2.2 Running Time

Generally speaking, to the ICP algorithm, the most time-consuming part is determining the correspondences between the two data sets. Many methods are developed to reduce the complexity of matching procedure, such as K-d tree which decreases the complexity from O 2 to O(n log n). In our case, the most time costing parts are the determination the correspondences and the least square computation. The former is determined by the number of events and model size, while the latter is influenced by events number used for calculation. Normally, in our application, the model size will be no more than 200 events, and the number of events in 1 ms will be less than 1000. From a preliminary test, compare to the SVD computation, the time of correspondence determination is small. Thus the running problem to the event number for least square computation is studied. In testing, the model size is set to be 200, and the collected event number are set to be the worst case: 1000. The program is implemented in C++ under Windows on a 3.4 GHz i7 core desktop PC, running separately off-line. The running time with the event number for least square computation is shown in Fig. A.2 Similar to the Algorithm 1 and Algorithm 2, event based ICP has linear property. The running time increases with the number of events being processed.

A.1.3 Summary

In summary, this section developed the event-based affine ICP for arbitrary shape tracking, and tested its performance in preliminary experiments. Specifically, the capability of the tracking method is tested by tracking a freely moved model. During the experiment, the event based ICP method allows tracking with the sampling rate of more than 10 kHz with precise and the lightweight running load. The event-based data acquisition and processing method provide novel opportunities for high-speed real-time tracking. In the future, it can be used for many applications e.g. target clamping, position servoing, biological objects following etc. Considering tracking the more flexible and changeable biological livings e.g. cell division, tissue growth and so on where the affine transformation cannot cover, the event-based active contour model may be implemented in the future.

Experiment Protocol

The protocol for the user experiment of touching RBC's 3D contour in the Chapter 4 is displayed in this part. It was followed by each experiment participant. The pre-experiment questionnaire needs to be finished before the experiment, which is mainly about the basic information of participants. Besides, the post-experiment questionnaire will be finished after the experiment, which focused on the user experience.

• Experimental Protocol 1.The participant read and signed a consent form.

2.The participant completed the pre-experiment questionnaire.

3.The participant was told the goal of the experiment.

5.The haptic interface was demonstrated to the participant for several minutes.

6.The participant was allowed to practice using the haptic interface, until the participant felt they are ready to begin the task.

7.The task was finished in three modes respectively: touch the RBC with only visual information; visual and haptic feedback; and only haptic feedback.

8.The participant completed the final post-experiment questionnaire.

During each experiment, 3D position of nanostage, and optical force data are recorded.

The following questions were asked by the pre-experiment questionnaire:

• Pre-experiment Questionnaire 1. Gender (M/F/Other).

2. Handedness (Right/Left/Ambidextrous).

Profession of participate

Age and height

The following questions were asked by the post-experiment questionnaire:

• Post-experiment Questionnaire 1. Have you ever used microscopy before?

Appendix B. Experiment Protocol

2. Are you familiar with the technique of micro-manipulation?

3. Are you familiar with the haptic interface?

4. Do you think this manipulation way is intuitive?

5. Do you think the force feedback is useful?

6. How is sensation when you touch a micro-object?

7. Do you think the haptic feedback is helpful to complete micro-manipulation tasks ?

The first three question were asked to rate their expertise with the microscopy, micromanipulation and haptic. For analysis purposes, these expertise levels were assigned to the number 1-5, from not at all to expert. The question 4-6 are about the performance and applicability of system. Participants need to give the scores to these the questions from 1 (very bad) to 5 (very good).

Appendix C

Motion Planning Algorithm Based On OTs Force Sensing

C.1 Geometric Representations

Formulating and resolving planning problems requires defining geometric models of environment and related systems of bodies from the workspace [LaValle 2006]. Without the benefit of vision and any a priori information on the environment, modeling techniques such as semi-algebraic models, non-convex polyhedra, or Nonuniform rational B-splines (NURBS) can hardly bring any convenience. In samplingbased planning problems, one straight-forward method of geometric representations is to discretize the bounded workspace by using bitmaps. Though with weaker notion of completeness, bitmap often arise in real applications for its practicality. Gray-scale map or occupancy grid is one generalization of bitmaps, which is commonly applied in probabilistic robotics, such as in Simultaneous Localization And Mapping (SLAM).

With the above considerations bear in mind, uniform bitmaps are applied for geometry representation. The overall Cartesian workspace is in R 3 . For each map-cell visited by OTs, a state can be recorded to indicate if it is free path or obstacle (vessel wall). This implicitly construct the environment for the planner. The number of map-cells per each dimension depends on the required resolution. Considering the probe size and simulation model, it is assumed that setting resolution as 1 µm is resolution complete, i.e., if a solution exists to a search query, planning algorithm will not miss it due to sampling simplification. According to a different geometry condition of environment, the resolution must be carefully re-adjusted to decide the trade-off between completeness and complexity, which is fundamental to sampling-based planning problems.

C.2 Configuration Space

The configuration space (c-space) describes a set of all possible transformations that can be applied to the robot. The obstacles expressed in the workspace or other local spaces must be implicitly or explicitly transformed into obstacles in the configuration space, so that planning algorithm can solve search query in a unified space. This provides a high level of abstraction so that planning problems with various geometrics and kinematics can be handled with the same logics. The dimension of workspace and the resolution defined above determines the size of Algorithm 1 Search query algorithm. Besides query pair q start and q goal , the topological graph G is also input to the algorithm. This G could point to an empty data structure when search begins, or a partially/completely constructed graph. This algorithm communicates with positioning and sensing system for sending motion command q goal and receiving force signal state. G.replan is a flag to indicate if obstacle is encountered (retreat-if-contact is applied) and thus replanning must be carried out. The algorithm stops when q goal is reached or no possible solution can be found as indicated by the function HAS_SOLUTION(q next ). function SEARCH(q start , q goal , G) 1 : q prev , q next = q start 2 : while q next = q goal && HAS_SOLUTION(q next ) do 3 : SEND_MOTION_CMD(q next ) 4 : state = GET_HAPTICS_STATE(q next ) 5 :

if IN_COLLISION(state) then 6 : UPDATE_EDGE(q next , G) 7 :

G.replan = true 8 :

q next = q prev 9 : else 10:

q prev = q next 11:

q next = D_STAR_LITE_MODIFIED(q next , q goal , G) 12:

G.replan = false 13: end if 14: end while the complexity of insertion from Θ(lg n) to Θ(1). However, its overhead remains to be investigated, which will not be discussed in this preliminary development. Secondly, scanning edge cost is replaced by processing force signals, and related edge changes are handled in retreat-if-contact strategy.

One important invariant of the algorithm is that the variable q prev always points to the last obstacle-free configuration. So that the probe can safely retreat to this configuration without any collision checking overhead. This can be proved by induction method, as long as the initial configuration is obstacle-free.

Though described in the pseudo code with if condition for state transition, in practice, the modified D* Lite and retreat-if-contact are two different implementations (strategy pattern) for the same virtual function, which favors dynamic polymorphism.

The algorithm described above responds efficiently to any single search query. In reality, multiple-query version of the motion planning arises often. One can of course apply repeatedly the search algorithm above. But a more reasonable and efficient approach is to invest substantial time to construct a roadmap, which refers to a topological graph that contains enough paths to make any motion planning query easily solvable. This motivation leads to mapping.

C.3.2 Recursive Greedy Mapping Algorithm

For a complex structure like biological tissue, mapping could take significant amount of time. In the middle of such procedure, interruption by any exceptional events, emergency stop or malfunctions could occur. Therefore, this requires the mapping algorithm to be robust. Firstly, the algorithm must be able to take into account the partially-constructed map, instead of restart mapping procedure. Secondly, it must not rely on the previously stopped configuration or require the probe to go back to this configuration.

Based on these considerations, we hereby apply greedy algorithms [START_REF] Cormen | [END_REF]] to mapping method. The essence is to recursively move the probe from current configuration to lowest-cost neighbor configuration that is not observed yet, until the overall environment is mapped. Such method is robust, and can be easily extended to distributed search by multi-agents, i.e., cooperative mapping. Regarding the algorithm performance, the total problem solving time is completely dominated by travel distance. In the study [Koenig 2001], it has been shown that the worst-case travel distance is between Ω( log|V | loglog|V | |V |) and O(|V | 3/2 ) on strongly connected undirected graphs G = (V, E). An alternative approach is to apply depth-first search algorithm. Such method leads to less travel distance for mapping, however, it is not robust, and cannot be applied for cooperative mapping.

The recursive greedy mapping algorithm incorporates the searching method described in Algorithm 1, and its pseudo code is shown in Algorithm 2. Algorithm 2 Recursive greedy mapping algorithm. Given any arbitrary configuration q start , the algorithm firstly search the nearest (in the sense of lowest cost) unobserved neighbor configuration q goal . If distance in L ∞ metric is used as cost, then searching such neighbor can be enabled by using breadth-first-search (BFS). Then Algorithm 1 is applied to reach q goal , with G updated correspondingly. This recursive algorithm stops and returns G when q goal points to null pointer, meaning all reachable configurations are observed. function MAPPING(q start ) 1 : q goal = NEAREST_UNKNOWN_NEIGHBOR(q start ) 2 : if q goal == nullptr do 3 :

return G 4 : end if 5 : SEARCH(q start , q goal , G) 6 : return MAPPING(q goal )