Stefano Maludrottu

Alessio Dore

Hany Sallam

Carlo S Regazzoni

Performance Evaluation of Multisensor Architectures for Tracking

Keywords: Data Fusion Architecture, Tracking, Performance Evaluation, Multisensor

In recent years the development of distributed fusion systems for real-world services (tracking, surveillance...) has remarked the importance of good design choices for both logical and physical architecture. The main objective of this paper is to implement a model of a distributed fusion architecture in order to evaluate the overall performance as well as the interactions of its subparts. The advantage of the proposed architecture analysis method are: 1) to allow quantitative comparison of different fusion structures through the evaluation of performance metrics and 2) to validate the algorithms involved as well as the physical structure (communication links, computational load...). Finally example structures are compared in order to find the best solution to some benchmark problems.

Introduction

Performance evaluation of data fusion systems for tracking is a key task for the correct design of effective and robust architecture. In fact in many applications, as videosurveillance or radar flight tracking, a precise target localization is required over time. In these domains multisensor approaches are successfully employed because of the capability of resolving problems caused by misleading observations, exploiting the redundancy introduced by the possibility of having multiple measurements for the same target. The architecture of these systems is typically complex and composed by many sensors that monitor wide areas and fusion modules providing a unique scene representation. Therefore in this domain several processing units and communications links are involved in this process and their behaviors affect performances.

In this paper a model of a data fusion system for tracking is proposed that takes into account different aspects of such system architectures to assess their performances for what concerns the algorithms accuracy and also to consider communication and computational complexity issues that can arise in complex multisensor systems.

Previous Works and Motivations

This paper presents a novel modeling method for distributed fusion architecture evaluation. In particular within this domain the problem of performing multicamera tracking of moving objects using multiple of active sensors is addressed.

In the literature various approaches have been proposed for distributed fusion architecture modeling: in [START_REF] Marcenaro | Distributed architectures and logical-task decomposition in multimedia surveillance systems[END_REF] a twofold architecture description is defined on physical and logical level that we take as an inspiration: the physical architecture described as a graph constituted of nodes (processing resources as sensors or fusion nodes) and links (heterogeneous communication channels) while logical architecture is defined outlining a task decomposition (i.e. tracking) into subtasks (i.e. data acquisition, filtering etc.); thus the distribution of intelligence in a fusion system architecture can be defined as a mapping of the subtasks in the various parts of the physical architecture. In [START_REF] Micheloni | A network of cooperative cameras for visual-surveillance[END_REF], a hierarchical architecture is proposed for video surveillance and tracking; the camera network sends the data gathered to the upper level nodes in which tracking data are fused in a centralized manner. In some related works are introduced specific metrics in order to measure the reliability of each subpart involved in the data fusion. The importance of this topic is demonstrated by the relevant body of research evaluating reliability and accuracy in automatic tracking. The vast majority of the works found in literature defines metrics related to specific parts of the architecture missing a more generic analysis. However, during the design of complex data fusion based systems, it is important to consider the influence of the architectural aspects of the system on the overall performances.

The work described in [START_REF] Ellis | Performance metrics and methods for tracking in surveillance[END_REF] focuses on defining the main requirements for effective performance analysis for tracking methods. In [START_REF] Needham | Performance evaluation metrics and statistics for positional tracker evaluation[END_REF], a number of metrics is defined for trajectory comparison, in order to determine how precisely a target is tracked by a computer vision system. In [START_REF] Young | PETS metrics: On-line performance evaluation service[END_REF], an evaluation benchmark is presented to compare different visual surveillance algorithms developed for PETS (Performance Evaluation of Tracking and Surveillance) workshops. PETS metrics are specifically developed to provide an automatic mechanism to quantitatively compare similar purpose algorithms operating on the same data. In [START_REF] Ziliani | Performance evaluation of event detection solutions: the creds experience[END_REF] a complex framework is proposed for direct comparison of event detection solutions based on any image analysis and understanding module. In the work presented in [START_REF] Taylor | Ovvv: Using virtual worlds to design and evaluate surveillance systems[END_REF], the authors develop a tool to automatically generate accurate ground truth of realistic situations in multicamera videosurveillance systems based on a commercial game engine.

Data Fusion Frameworks for Tracking

Data Fusion architecture for Target Tracking

A general data fusion architecture of a system aiming at target tracking can be subdivided into several module in a hierarchic way. The lower level is constituted by sensors that gather information from the environment and try to detect the targets position within their field of view (i.e the area observable by the sensor). Target tracking can be performed in several scenarios and the typologies of sensors that can be used may change from video cameras to radars, sonars or microphones. Despite the variability of the input data the sensor can be considered as a module that perceives the variations in the environment produced by the target and it converts resulting signals to position information. However, usually in a general situation the single sensor observation of the scene is affected by noise that can produce erroneous localization and tracking.

For example typical problems arising in video data flow are due to the superimposition of targets and elements of the scene occupying different 3D in the image plane. This issue, called occlusion, causes a momentary lack of observations with consequent difficulties in the position estimation procedure. Moreover, background elements can produce misleading observations (or clutter) that are to be excluded from the measurements-target association procedures. A possible and widely exploited approach to overcome these issues and to enhance the tracking robustness consists in using more sensors monitoring with complete or partial fields of view overlapping the same area. In this way it is possible to obtain multiple observations of the same entity whose joint processing likely improves tracking performance. The fusion module can be considered as a function that associates in a proper way the data produced by the sensors with overlapped fields of views and estimate an unique scene representation.

An example of the above mentioned architecture is shown in Figure 1(a). Sensors are considered as smart sensors, i.e. with an embedded processing unit and with the capability of performing their local localization and tracking tasks. Fusion nodes gather data from two or more sensors and process them jointly to generate an unique scene representation. In the following a model for smart sensors S i together with fusion nodes F k that allows us to define a general architecture of a data fusion system for target tracking. In this work we will consider only a structure where sensors can exchange data only with the fusion node and vice-versa. This structure has been chosen since the focus of this work is on the hierarchical architecture where the fusion node overlooks the sensors and, eventually it modifies their parameters or it passes information gathered from other sensors. This model can be constituted by one or more fusion nodes according to the extension of the monitored area and the total number of sensors. Without loss of generality, we will consider the overall fields of view (FOVs) of the fusion nodes contiguous non-overlapped. In fact, if this is not the case it would be sufficient to introduce a higher fusion level performing the same tasks of the fusion nodes on their outputs.

Performance evaluation of a data fusion architecture does not pertain only on algorithms accuracy in localization but several other aspects must be considered as the data communication between sensors and fusion nodes, the computational complexity and the memory used. Then, in order to define a procedure to assess the performances of this kind of systems a general model of smart sensors and of fusion nodes has been studied that takes into account the elements involved in a multisensor tracking process.

Smart Sensor Model

Each smart sensor S i observing a certain area V Si (t) is modeled as composed by three fundamental elements (see Fig. 1(b): processing function T i : it represents a tracking algorithm that produces a certain output data D f i (t) given an input data D i (t) referring to the scene observed in V Si (t). The input noise is modeled with E i (t) and the noise on the processed data sent to the fusion node is E f i (t). The variables D i (t), D f i (t), E i (t) and E f i (t) are vectors of dimension equal to the number of targets N i (t) detected by S i at time t. The computational complexity C function of D i (t) is associated to the processing function.

The error vector in this tracking application coincides with the accuracy error that has to be evaluated through a suitable performance metric. In this paper, the considered metric is the distance between the estimated target position and the ground truth.

Another error that must be taken into account in this scenario is the detecting error E D i (t) caused by occlusions or clutter. The detection error implies that the estimated number of targets N i (t) is different to the real one N GT i (t) due to the false alarm or miss detection.

database: it represents the memory M of the smart sensor. It has been decided to address it as database since in many tracking application algorithms require to compare actual measurements with past detected targets. The algorithm T exchanges information with the database that can also send additional information to the fusion node.

communication unit: it is responsible for transmitting data to the fusion node. The communication unit has to satisfy the requirements of error free data transmission considering possible bandwidth B limitations of the link toward the fusion node. -orientation: each smart sensor is considered to be placed in a certain known position; its FOV depends on its orientation. If a smart sensor has a fixed orientation its FOV V Si (t) becomes a constant V Si .

Fusion Node Model

In this stage the data coming from each sensor are integrated to realize an accurate single model of target position. Its structure similar to the smart sensor model but some differences are present in the processing function and communication unit.

genetic optimization: genetic algorithms are population based search algorithms [START_REF] Holland | Adaptation in natural and artificial systems[END_REF] used as optimization tools in a lot of different applications. In this paper we implemented a GA-based optimization algorithm to find the best orientations of a group of smart sensors in a distributed multisensor architecture. The input to GA is the current position of moving objects in the environment as perceived by the fusion node and the output is the orientation angle of each camera. The fitness function to be optimized for each sensor separately is formulated as i w i d i , where d i is the perpendicular distance from an object position to the central line of FOV of sensor i and w is the number of objects in a given position. The camera is considered to be best oriented minimizing the sum of distances. -processing function T F : the fusion node processing function T F takes M data D f i (t) (together with the respective error measure E f i (t), i = 1, . . . , M ) as input, where M is the number of smart sensors connected to the fusion node F k . Thus, the transfer function is:

T F (D f 1 (t), . . . , D f M (t), E f 1 (t), . . . , E f M (t)). The cardinality of D F (t) and E F (t) vectors is N F (t) = T F (card(D f 1 (t) ∪ D f 2 (t) ∪ . . . D f M (t))) i.
e. the intersection of the targets detected by the sensors processed by the fusion algorithm T F that manages the overlaps and handles the false alarm and miss detection problems. Due to the overlapping fields of view the D f i (t) vectors have common elements, relative to the targets detected by multiple sensors). The output of T (E f 1 (t), . . . , E f M (t)) produces an error vector E F (t) in which each value is lower than the respective error value in E i (t) detected from the i-th sensor. Worth of note is that the computational complexity C of the fusion node processing is related to the dimension of the overlapped fields of view and the number of targets present in that zone. -database: the database performs the same tasks described for the smart sensor communication unit: it analyzes the transmission load for each communication link connecting the fusion node to the smart sensors. It can also send messages to the smart sensor communication unit in order to perform an adaptive optimization of the data transmission. For example, if one link is overused the fusion node can ask to all the smart sensor connected to that link to reduce their transmission rate by a higher compression.

3 Simulation and Results

Multisensor Tracking System Simulator

On the basis of the architectural model described in Sect. 2 a tracking simulator has been realized to demonstrate the possibility to evaluate the system performances. A context generator has been designed in order to produce a set of realistic target trajectories simulating moving objects with stochastic dynamics. The simulated smart sensor S i acquires data from the context generator only for what concerns his field of view V i (t) and processed them using a Kalman Filter. Multiple sensors with overlapped fields of view send their processed data to one fusion node F k that processes the data using a modified Kalman Filter in addition to a Nearest Neighbor data association technique [START_REF] Xu | Architecture and algorithms for tracking football players with multiple cameras[END_REF]. This experiment intends to show how it is possible to adapt a real tracking system into our proposed model.

Context Generator

The context generator produces trajectories with linear autoregressive first order model with Gaussian noise. Trajectories lies in a twodimensional space representing a map of a monitored environment. A Poissonian distribution is used to describe the probability of new target born and death. Occlusions are also simulated when two targets are aligned with respect to the camera line of sight and they are sufficiently close one to another. Clutter noise is also introduced at each scan as a Poisson process.

The context generator allows us to have a realistic but controllable environment by which simulating different situations automatically generating the correspondent ground truth.

Smart sensor

-Processing function: when a target trajectory enters in the field of view V i (t) of the i-th sensor a single-view tracking task is initialize to permit to know the position and to follow the movements of targets. For each target, in order to estimate the position of tracked objects a Kalman filter is used [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF], whose equations are:

x(k + 1) = A w x(k) + w w (k) (1)
z(k) = H w x(k) + v w (k) (2)
w l and v l are, respectively, the image plane and the measurement noise and they are independent, white Gaussian and zero mean. A l and H l respectively are the state transition matrix and the measurement matrix.

In this work the computational complexity is a linear function of the number of targets present in the FOV. When this function exceeds a threshold defined to represent the unit processing capability a random lack of data to be sent to the fusion node is simulated. -Communication unit: in this simulator the communication unit of the sensor is responsible to send data to the fusion node according to the fusion node indication, regarding the occupation of the link. A function has been implemented to simulate the optimization of data transmission -Database: in this simulator the database is not modeled since the performed tasks don't need a data storage Fusion node -Communication unit: it handles the data stream coming from the smart sensors. A maximum bandwidth value B max has been defined according to the physical constraints of the architecture. If i B i (k) > B max (where B i is the bandwith occupation of the sensor i at the time k ) a sensor subset S * (k) is chosen in order to optimize the bandwidth usage and such that i tB * i (k) <= B max . Various strategies can be implemented to select the sensor subset S * (k); first we assign to each sensor S i a priority value p i and we define the subset of selected sensors equal to an empty set S * (k) = ∅; then we will add the sensors one by one, choosing at each step the sensor between the remaining ones with the highest priority value. After the sensor selection a feasibility check is performed; if the bandwidth constraint is satisfied the sensor is added to the selected sensor subset (S * (k) = S * (k) + S j ).

-Processing function: in the processing function initially the association of measurements with targets is performed; after that these targets are updated and for unmatched measurements a target initialization procedure is applied [START_REF] Xu | Architecture and algorithms for tracking football players with multiple cameras[END_REF]. Each target x t is represented by using x t (k), state covariance P t (k) and a category estimate e t (k) for each frame k. The state vector is:

x = x y ẋ ẏ T (3)
where (x, y) are the ground plane coordinates and ( ẋ, ẏ) is the velocity.

In the measurement m t we include the ground plane position z t (k) whose covariance is R t (k), where: z = x y T . Therefore we measure the state on the ground plane using a Kalman filter. The association between tracks is evaluated by using the Mahalanobis distance [START_REF] Mahalanobis | On the generalized distance in statistics[END_REF] e define:

S jt (k) = H w P t (k/k -1)H T w + R (i) j (k) (4)
thus, defining

I(k) = z (i) j -H w x(k/k -1) : d 2 jt = I(k) T S jt (k) -1 I(k) (5)
where x(k/k -1) is the target prediction. To determine the association this distance is compared against a threshold and the Nearest Neighbour Algorithm is applied in order to establish the association. We can make the hypothesis that for each target, in one view, there is at most one measurement, i.e. j β (i) jt ≤ 1; naturally if the measurement misses then j β (i) jt = 0. The overall measurement for each target is the result of single camera measurements weighted with their covariance R (i) j as follows:

R t =   i j β (i) jt R (i) j -1   -1 (6)
z t = R t   i j β (i) jt R (i) j -1 z (i) t   (7)
These equations express the accuracy improvement obtained thanks to the fusion of all the views; as a matter of fact Equation 6implies that the overall uncertainty is lower than the one provided by a single camera. Moreover Equation 7 ensures that the fused measurement is biased to the most accurate measurement from one camera. The target updating is, then, attained in the following way:

xt(k/k) = xt(k/k -1) + Kt(k) [zt(k) -H w xt(k/k -1)] (8)
Kt(k) = P t(k/k -1)H T w h H w P t(k/k -1)H T w + Rt(k) i -1 (9)
its covariance is:

P t (k/k) = [I -K t (k)H w ] P t (k/k -1) with 0 < η < 1.
For those measurements not matched to any existing targets a target initialization procedure has to be applied. Hence, every unmatched measurement from a camera must be checked against the other unmatched measurements from other cameras to find new targets. We call z (i * ) j * the unmatched measurement from the i * th camera and x n the new target associated with it. All the association matrices β (i) are extended by one column, β (i) jn , for the new target x n ; its elements represent the association between the jth measurement from ith camera and the new target. For the i * th camera the elements of the columns nth, referring to the new target, are:

β (i * ) jn = 1 if j = j * 0 otherwise (10)
Then the Mahalanobis distance between z (i * ) j * and each unmatched measurement z (i) j from ith camera is calculated:

d 2 i * j = h z (i) j (k) -z (i * ) j * i T h R (i) j -R (i * ) j * i -1 h z (i) j (k) -z (i * ) j * i ( 11
)
where R (i) j and R (i * ) j * are the measurement covariances. Then, the new target is initialised as follows:

xn (k/k) = z n (k) T 0 0 T (12)
P n (k/k) = R n (k) O 2 O 2 σ 2 v I 2 (13)
where O 2 and I 2 are 2 × 2 zero and identity matrices, respectively. The processing capability of the fusion node is limited due to the hardware specific processing resources (CPU, memory...). Similarly to the communication unit we define a maximum CPU load C max ; each of the S i ⊂ S * (k) sensors (output of the communication unit at a given time k) produces a CPU usage C i (k). If i C i (k) >= C max we define a priority-based selection of the data streams in order to obtain a feasible subset S ′ (k) such that C ′ i (S ′ i (k)) <= C max . This selection will cause a loss of input data in order to preserve the computational feasibility; thus the data fusion will be performed using only a subset of the whole input data to the fusion node.

-Database: in this simulator the database is not modeled since the performed tasks don't need a data storage

Experimental results

Three scenarios have been investigated in order to demonstrate the possibility of fusion architecture evaluation using the proposed method. The performance metric is defined as

perf = i |f i -g i | + w • m i T OT i ( 14
)
where |f ig i | is the euclidean distance between the target position f i and its corresponding ground truth g i at timestep i. The term w is a penalty value set equal to 10, m i the number of missing associations between ground truth data and fused tracks at timestep i and T OT i is the total number of timesteps of the generated context (see Figure 2). First scenario Two alternative fusion architectures are defined to be evaluated using the proposed method in order to find which one has better performances. The overall structure (5 smart sensors, 1 fusion node and wireless communications) is maintained fixed and only the sensors position has been changed (always ensuring the complete coverage of the context area) to find the best configuration. In the first structure the field of view of the smart sensors does not change over time, while in the second structure the field of view dynamically change according to the genetic-based optimization technique defined in Sect. 2.3. Ten different scenes with a fixed number of trajectories [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF] have been provided by the context generator as input for the model. Parameters of GA are inizialized as follows: population size of 20, number of generations equal to 100 and crossover and mutation rate respectively 0.8 and 0.04. There are many parameters that influence the context generator algorithm behavior: the total number of trajectories, the generation rate, the Gaussian noise (i.e. measurement error in smart sensors), etc. Those parameters have been randomly modified for each context generation in order to provide a ground truth as diversified as possible. Table 3 presents the different results (obtained using the metric defined in Eq. 14 )of the sample architectures and shows the connection between different camera placement strategies and performances: dynamic changing camera orientation outperforms a fixed orientation solution up to 20%. Second scenario Focusing on the architectures defined in the previous test, the overall result of the data fusion has been compared with the tracks produced by each smart sensor in the architecture in order to point out the contribution of each one to the performance of the fusion architecture. As shown in Table 1 sensor 3 has the worst performance: it acts like a bottleneck for the entire structure; a better placement or perhaps a more efficient sensor could improve Table 1. (a) Performance comparison of subparts of Architecture 1(fixed camera orientation); (b) Performance comparison of subparts of Architecture 2 (genetic-based algorithm for optimization of camera orientations). Cont refers to the context used as input. Avg is the average performance over ten experiments. Performances are evaluated using the metric defined in Eq. 14.

the performance of the entire structure.

Third scenario Two different fusion structures have been compared: architecture A has five sensors, an average communication link and average computational resources while architecture B has only three sensors with wider fields of view (60 • instead of 45 • ) and a more accurate target detection and tracking, a better communication link and better fusion node hardware. 10 different scenes with a fixed number of trajectories [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF] have been provided as input for the model. In Table 2 is shown that architecture A, even with worse hardware obtains a better overall result thanks to the higher number of sensor placed on the area.

Conclusions and Future Works

In this paper a model of a system architecture for multisensor tracking has been proposed. The aim of this model is to represent basic tasks performed by such a system in order to address the problem of performance evaluation. This description allows one to validate algorithms and system design with respect to the monitored scenario. A simulator of a multisensor hierarchic tracking system has been presented to show the possibility of using the model as a tool to assess system architectures and algorithms. Future works will be devoted to apply the model to decentralised fusion architectures and to realise a more accurate model of different tipologies of smart sensors.

Fig. 1 .

 1 Fig. 1. (a) Example of a data fusion system for target tracking; (b) Representation of the context generator(scene) and intelligent sensor model

Fig. 2 .

 2 Fig. 2. Representation of the ground truth ('o') and fused target tracking ('*')

Fig. 3 .

 3 Fig. 3. Performance comparison of Architecture 1 (fixed camera orientation -in blue) and Architecture 2 (genetic-based algorithm for optimization of camera orientationsin violet)

Context

Arch


Yuyu Liu

Yoichi Sato

Finding Speaker Face Region by Audiovisual Correlation

The ability to find the speaker face region in a video is important in various application areas. In this work, we develop a novel technique to find this region robustly against different views and complex backgrounds using gray images only. The main thrust of this technique is to integrate audiovisual correlation analysis into an image segmentation framework to extract the speaker face region. We first analyze the video in a time window and evaluate the audiovisual correlation locally at each pixel position using a novel statistical measure based on Quadratic Mutual Information. As only local visual information is adopted in this stage, the analysis is robust against the view change of the human face. Analyzed correlation is then incorporated into Graph Cut-based image segmentation, which optimizes an energy function defined over multiple video frames. As this process can find the global optimum segmentation with image information balanced, we thus can extract a reliable region aligned to real visual boundaries. Experimental results demonstrate the effectiveness and robustness of our method.

Introduction

The ability to detect a speaker accurately from multiple persons is important for various applications in video processing and content analysis systems. For example, a video-teleconferencing system may need to focus on a speaker, or a video analysis system may have to associate uttered words with the right speaker. Being able to identify the speaker face region is furthermore preferred because this makes various effects possible, such as to automatically emphasize the speaker by blurring all other persons and background, or, on the contrary, to impose mosaic over the speaker in an interview to protect privacy. An example of them is shown in Figure . 1 based on the results of our method.

However, regardless of the great progresses in face and human detection (e.g., [START_REF] Viola | Robust Real-Time Face Detection[END_REF] and [START_REF] Dalal | Histograms of Oriented Gradients for Human Detection[END_REF] respectively) made in recent years, speaker detection is still under development. As the purpose is to distinguish a person from others, face or human detection fails in this area. One solution [START_REF] Luettin | Speaker identification by lipreading[END_REF] is to detect the face, locate the mouth, and check its movement. The weakness of this method is the requirement of a frontal view. In fact, the problem of view dependency is also a challenge even in face and human detection systems [START_REF] Viola | Robust Real-Time Face Detection[END_REF] [START_REF] Dalal | Histograms of Oriented Gradients for Human Detection[END_REF]. Additionally, as the correlation between the mouth movement and audio signal is not well analyzed, it is prone to be disturbed by unconscious movements from other persons.

In this work, we develop a novel technique to find the speaker face region from gray images of a video in a time window, which is robust against different views and complex backgrounds. This technique is based on the recent developments in audio source localization by audiovisual correlation analysis and an image segmentation technique over multiple video frames.

To localize an audio source by audiovisual correlation analysis is a relatively new research topic and has drawn much attention in recent years. Psychological research [START_REF] Driver | Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading[END_REF] discovered that audiovisual correlation mainly lies in the synchrony. Initiated from it, Hershey and Movellan [START_REF] Hershey | Audio vision: Using audiovisual synchrony to locate sounds[END_REF] first introduced a method to localize the audio source by audiovisual correlation analysis. They assumed that audio energy and pixel intensity obey joint normal distribution and measured their Mutual Information (MI) to decide the audio source. Yet, this assumption is too strong and contradicts with the real observation. To avoid this problem, following researches tried another way, where they optimize an objective function to find the audio source. Darrell and Fisher [START_REF] Darrell | Speaker association with signal-level audiovisual fusion[END_REF] searched the optimum projection vector that can maximize the lower bound of MI between the projected audiovisual signals. Kidron and Schechner [8] searched a projection vector which can maximize the Canonical Correlation Analysis (CCA) between audiovisual signals and has minimum L1-norm. In 2005, Monaci et al. [START_REF] Monaci | Analysis of multimodal signals using redundant representations[END_REF] claimed that the movements of the photographed objects convey better audiovisual correlation than the pixel values. They employed Matching Pursuit (MP) to extract local objects, tracked their translations and rotations in the video and computed the correlation with audio to decide which one should be the audio source.

Unfortunately, all methods mentioned above suffer a common problem: the estimated audio source is highly fragmental. Therefore, they in fact experience difficulties in designating a correct speaker position, much less identifying a reliable speaker region. Most of them can detect only pixels that are supposed to be the audio source. Only Casanovas [START_REF] Casanovas | Blind audiovisual source separation using sparse redundant representations[END_REF] clustered the estimated pixels and adopted the cluster center as the speaker position. Yet clustering is vulnerable to the outliers that appear often.

To be able to detect a reliable speaker face region, we consider a novel technique, whose key idea is to integrate audiovisual correlation analysis into image segmentation framework to extract out the region. Our current system requires that the speaker must stay nearly at the same position in the estimation time window for the sake of audiovisual correlation analysis, as was assumed in previous methods [START_REF] Darrell | Speaker association with signal-level audiovisual fusion[END_REF] [6] [START_REF] Kidron | Pixels that sound[END_REF]. The time window is generally within 2 -4 seconds. We extract the audiovisual features inside this time window and analyze the statistical audiovisual correlation at each pixel position using a novel measure -Quadratic Mutual Information (QMI). We then do a global optimization to extract a reliable speaker region from the pixel-based correlation values, which is based on the Graph Cut-based image segmentation technique. Note that the speaker region here does not necessarily mean his/her face region, although our method can detect the whole face region in most cases. Sometimes only the mouth region is detected, as discussed in Sec. 4. This will not be a problem if only the speaker position is required. Yet, for the applications requiring the face region, we can use the face detection rectangle to supply a coarse mask because we now know who the speaker is.

In contrast to audio source localization, image segmentation has been studied for decades. Recently, Boykov and Funka-Lea made an important progress step [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF], in which a globally optimum segmentation, which balances pixel likelihood and image region information, can be found efficiently using Graph Cut. The method works for not only a single image, but also for multiple video frames with inter-frame continuity considered [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF]. The global optimization framework and the ability to process multiple video frames well fit our requirements. Additionally, although the method in [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF] requires the user to designate seeds of foreground and background manually, which limits its usefulness, our method successfully eliminates this requirement by integrating audio information. Audio information also gives our method better robustness against backgrounds than [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF].

The main contribution of this work can be summarized as follows: 1) to find the speaker face region robustly against different views and complex backgrounds by incorporating audiovisual correlation analysis into image segmentation, 2) to firstly use the QMI and kernel density estimation to analyze audiovisual correlation, and 3) to adopt audio information to eliminate the manual operations in Graph Cut-based segmentation and improve its robustness against complex backgrounds.

The rest of this work is organized as follows. In Section 2, we propose our audiovisual features and the computation of pixel-based audiovisual correlation using QMI. In Section 3, we explain how we find the speaker face region by performing image segmentation based on the results of audiovisual correlation analysis. In Section 4, we present and discuss our experimental results. In Section 5 we present our conclusions.

Audiovisual correlation analysis

If audio and visual signals are originated from one source, they should have strong correlation. To measure such correlation, some methods have been proposed; as reviewed in Sec. 1. However, none of the previous measures satisfies our requirements: to analyze the pixel-based audiovisual correlation under arbitrary distributions. Consequently, we introduce a novel measure -QMI, which is based on kernel density estimation. For continuous random variables, QMI can be computed much more efficiently than MI and is adopted to analyze the pixelbased audiovisual correlation. Moreover, kernel density estimation can estimate arbitrary probability density function (p.d.f.) [START_REF] Parzen | On the estimation of probability density function and the mode[END_REF]. Below we first introduce our audiovisual feature. Then we introduce the computation of audiovisual correlation by QMI.

Our audiovisual feature

As synchrony is the key to compute audiovisual correlation, we require the audio and visual signals to be recorded synchronously in the video. Based on this prerequisite, we extract the audio and visual features.

Audio feature. Input audio data are first divided into frames. The duration T a of each audio frame is set to be the same as that of each visual frame. In order to keep the continuity between frames, it is set such that each pair of two successive frames have an overlap of the duration of T a /2. Additionally, to reduce the boundary effect, a Hamming window is multiplied [START_REF] Rabiner | Fundamentals of Speech Recognition[END_REF]. Coefficients of a standard Hamming window can be calculated through h(m) = 0.54 -0.46 cos( πm M ), m = 1, • • • , M where M is the number of the samples in an audio frame. The logarithm energy of each frame is then computed by a(t) = log 1 M M m=1 s 2 (t, m) , where s(t, m) refers to the processed audio sample m in frame t. The audio feature is defined to be the differential energy between the current and next frames as f a t = a(t + 1) -a(t). Adopting differential energy is closely related to our adopted visual feature: the pixel-based intra-frame movement. Differential audio energy obviously demonstrates much higher correlation with this feature. The reason for this fact can be explained as follows: when the uttering movement is fast, audio energy should change quickly also.

We perform a verification of the existence of speech, although none of the previous methods has mentioned this process. Such process is important not only in the meaning that it checks whether audio conveys information, but also for its function to act as trigger to tell our system when it should work. The verification by now is to check whether or not the audio energy a(t) is beyond a pre-defined threshold. It can be extended to check whether or not speech exists in the audio by using more complex algorithm. Frames failing this test are regarded as silence and dropped, together with their corresponding image frames. Note that, if a(t + 1) fails the test, f a t will not be extracted either. From here on, if we talk about N video frames with speech, this refers to the frames that pass this test, and not to the actual number of video frames, which may exceed N .

Visual feature. Similar to the considerations in [START_REF] Monaci | Analysis of multimodal signals using redundant representations[END_REF], we believe that better audiovisual correlation exists not in the pixel values but between the movements of the photographed objects and the audio. However, to accurately extract all objects and track their movements in a video sequence is difficult and not robust. Instead, we adopt the intra-frame pixel-based movement, which is described by optical flow and extracted between every two consecutive video frames.

To adopt optical flow as the visual feature has three advantages for our system. First, it helps our system to be background robust. Movement is invariant to the complexity of a static background. For a moving background, as its movements usually correlate marginally with audio, the influence can be suppressed in the subsequent correlation analysis. Second, it helps our system to be view robust. Since optical flow is extracted locally at each pixel position, view difference of the human face has only minor influence on it. No matter how different a face looks in different view, its local parts move similarly when speaking. Third, the locality of the optical flow also makes it possible for our method to achieve good segmentation boundary.

Note that our method is different from motion segmentation technique. Although both adopt visual movements as feature, the inside assumptions differ largely. Motion segmentation assumes that regions of pixels move with coherent motion and evaluate the coherence to do segmentation [START_REF] Kanatani | Motion segmentation by subspace separation and model selection[END_REF]. Yet the speakingrelated actions are generally not coherent, e.g., the two lips of a people usually move in opposite directions. Motion segmentation cannot distinguish which motion correlates with audio either.

We compute the optical flow on gray images using the Lucas-Kanade method [START_REF] Lucas | An Iterative Image Registration Technique with an Application to Stereo Vision[END_REF]. The window size we used is 9 × 9. Since the optical flow cannot be estimated stably for windows with low texture, we verify the pixel intensity variation inside the window. If it is below a threshold, we set its flow value to zero.

Optical flow has two elements: horizontal and vertical movement. We need to transfer it into a scalar value to compute the correlation with the audio feature. We tested three derived values: amplitude, horizontal element, and vertical element. Results are shown in Figure . 2. The vertical element obviously has much higher correlation with the audio feature. This matches our intuition since speaking movements generally happen vertically. Consequently, we adopt its vertical element to be our visual feature, i.e., the visual feature f v t (u, v) at pixel position (u, v) in frame t is defined as the vertical part of the pixel's optical flow extracted between frame t and t + 1.

Audiovisual correlation by QMI

We regard the audio and visual features as two random variables and compute the correlation statistic using their temporal samples extracted from each video frame. Note that the correlation will be computed independently at each pixel position (u, v).

Firstly we adopt kernel density estimation to estimate their joint probability density function (p.d.f.). Kernel density estimation [START_REF] Parzen | On the estimation of probability density function and the mode[END_REF] (also known as Parzen window estimation) is a method to estimate arbitrary p.d.f. of a random variable.

p(x) = 1 N N i=1 K Σ (x -x i ), (1)
where K Σ (•) is the specified kernel function. In practice, Gaussian function with the diagonal bandwidth matrix

Σ = diag(σ 2 1 , • • • , σ 2 d
) is often adopted. Based on the estimated p.d.f., we compute the statistical correlation between the audio and visual feature. However, the computation of the Shannon MI by Eq. ( 1) is difficult since it does not yield a closed-form solution. Thus we adopt QMI proposed by Xu et al. [START_REF] Xu | A Novel Measure for Independent Component Analysis (ICA)[END_REF], which is based on the quadratic form of Renyi entropy.

If one of the four constraints defined by Shannon is weakened, Renyi [START_REF] Renyi | On measures of Entropy and Information[END_REF] showed that the entropy of a random variable can be evaluated by a group of functions defined as H α (x) = 1 1-α log p α (x)dx , where α > 0, α = 1. As α → 1, H α (x) approaches the Shannon entropy H(x). In practice, the one most often used is its quadratic form α = 2, which can be easily computed based on the p.d.f. estimated by kernel density estimation using a diagonal Gaussian kernel because we have [START_REF] Xu | A Novel Measure for Independent Component Analysis (ICA)[END_REF]

K Σ (x -x i )K Σ (x -x j )dx = K 2Σ (x i -x j ). (2)
Unfortunately, Renyi MI is left undefined. Although Shannon MI between two random variables x 1 and x 2 can be directly computed from their entropies as M I(x 1 ; x 2 ) = H(x 1 )+H(x 2 )-H(x 1 x 2 ), this is not true for the Renyi entropy. In 1998, Xu et al. [START_REF] Xu | A Novel Measure for Independent Component Analysis (ICA)[END_REF] proposed a form to compute QMI based on Renyi entropy, which is defined as

C(x 1 , x 2 ) = log p 2 (x 1 , x 2 )dx 1 dx 2 p 2 (x 1 )p 2 (x 2 )dx 1 dx 2 ( p(x 1 , x 2 )p(x 1 )p(x 2 )dx 1 dx 2 ) 2 . ( 3
)
It is easy to show that C(x 1 , x 2 ) ≥ 0 and the equality holds true if and only if p(x 1 ) = p(x 2 ) using Cauchy-Schwartz inequality.

In our case, using the temporal samples of the audiovisual feature {(x 1t , x 2t ) = (f a t , f v t (u, v)), t = 1, • • • , N }, we can analyze their QMI at each pixel position (u, v). Following Eq. ( 2), QMI can be directly computed from the sample set in closed-form as [START_REF] Xu | A Novel Measure for Independent Component Analysis (ICA)[END_REF]

               C(x 1 , x 2 |{x t }) = log Vc({xt})Vm({x1t})Vm({x2t}) V 2 nc ({xt}) V c ({x t }) = 1 N 2 N i=1 N j=1 2 k=1 G(x ki -x kj , 2σ 2 k ) V s (x kj , {x ki }) = 1 N N i=1 G(x kj -x ki , 2σ 2 k ) k = 1, 2 V m ({x ki }) = 1 N N j=1 V s (x kj , {x ki }) k = 1, 2 V nc ({x t }) = 1 N N j=1 2 k=1 V s (x kj , {x ki }) (4)
where G(x, σ 2 ) represents a 1D Gaussian p.d.f. and

K Σ (x) = 2 k=1 G(x k , σ 2 k
). An example of the analyzed correlation can be seen in Figure . 4.

Speaker region segmentation

Based on the analyzed pixel-based audiovisual correlation, we take advantage of the image segmentation technique to extract the speaker region.

Again using the retrieved N video frames, we build the N-D image as defined in [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF] and perform the video segmentation. The distance of each pixel to the foreground (speaker) and the background is computed from the analyzed audiovisual correlation. Since there is only one scalar correlation value at each image position (u, v), the computed likelihood is same for all pixels at (u, v), regardless of in which frame t they are. On the other hand, as image information, like edge, pixel similarity and intra-frame continuity, is related to both (u, v) and t, segmentation results can still be different in each frame and capture the face deformation when speaking.

Graph Cut-based segmentation

Segmentation of video frames by optimizing a global energy function was proposed in [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF]. The global energy function is composed of two important terms: the sum of the data cost to assign a pixel to be speaker, and the sum of the smoothness penalty between every two neighboring pixels in both temporal and spatial domains. Its definition is as Eq. ( 5),

E(l) = p D p (l p ) + λ • {p,q}∈N e S pq (l p , l q ), (5)
where l represents the segmentation labels of all pixels in the N-D image. l p = 1 means pixel p is labeled as the speaker; while l p = 0 means the background. N e defines the neighborhood relationship between two pixels, which is discussed in detail in Sec. 3.3. λ is a constant adjusting the balance between the two terms. It was shown that the energy function defined in Eq. ( 5) can be efficiently optimized by calculating the minimum cut of a graph using a maximum flow algorithm [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF]. Moreover, the optimization result is guaranteed to be the global minimum solution of the energy function [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF].

Data cost by audiovisual correlation

Since QMI is not a normalized correlation measure, its dynamic range changes largely for different retrieved N video frames. Furthermore, as shown in Figure . 4, although speaker face region obviously has higher audiovisual correlation values, their variation is large. Thus, it is difficult to directly adopt the correlation values as the data cost.

In order to solve these difficulties, we apply a clustering process to the correlation values from all pixels and take the Mahalanobis distance to each cluster center as the data cost. First, the highest and lowest analyzed audiovisual correlation values are selected as two seeds. Then, applying the Expectation-Maximization (EM) algorithm to all correlation values, we can train two onedimensional Gaussian distributions. The EM training generally converges after 6 -10 iterations. The one trained from the seed of the lowest correlation is regarded as the background correlation distribution, denoted as G(µ 0 , σ 2 0 ). The other is regarded as the foreground (speaker) distribution, denoted as G(µ 1 , σ 2 1 ). Finally, the pixel's data cost is computed by the Mahalanobis distance to each center, as defined in Eq. [START_REF] Hershey | Audio vision: Using audiovisual synchrony to locate sounds[END_REF].

D p (l p ) = (C(f a, f v(u, v)) -µ 1 ) 2 /σ 2 1 l p = 1 (C(f a, f v(u, v)) -µ 0 ) 2 /σ 2 0 l p = 0 (6)

Smoothness penalties by image information

Smoothness penalty is forced between every two neighboring pixels in both spatial and temporal domains. In the N-D image, the spatial and temporal neighborhoods are defined as shown in Figure . 3. Each pixel can maximally have 26 neighbors.

The value of smoothness penalty is computed by

S pq (l p , l q ) = exp -β(I p -I q ) 2 • (d(p, q)) -1 • T [l p = l q ], (7)
where p and q are two neighboring pixels. I p and I q are their intensity values. The constant β is chosen as in [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF] to be β = 2 (I p -I q ) 2 -1 , where • denotes the expectation over the N-D image sample. This choice of β ensures that the exponential term in Eq. [START_REF] Kanatani | Motion segmentation by subspace separation and model selection[END_REF] switches appropriately between high and low contrast. d(p, q) calculates Euclidean distance between p and q in the three-dimensional grid, which may be 1, √ 2 or √ 3 in our neighborhood model. T [•] is a boolean function returning 1 when the condition inside is true and 0 otherwise.

Experimental results

We used the CUAVE audiovisual database [START_REF] Patterson | Moving-talker, speakerindependent feature study and baseline results using the cuave multimodal speech corpus[END_REF] in all our experiments, where 17 females and 19 males uttered English numbers in front of a green background with frontal, lateral, and moving views. Video sequences were filmed at 29.97 fps, while the audio signals were sampled at 44 kHz stereo. An advantage of CUAVE database was that we could remove the green background by color and replace it with other complex backgrounds to test the performance of our algorithm. Color images were then converted into gray images and down-sampled from 720 × 480 to 240 × 160 for memory saving. For all our experiments, the kernel bandwidths are set as (σ 1 , σ 2 ) = (0.4, 0.3). The balance constant in Graph Cut is set as λ = 20.

Since we computed the audiovisual correlation by temporal statistics, we first tested the computed correlation values using different numbers of the video frames. The experimental results are shown in Figure. [START_REF] Darrell | Speaker association with signal-level audiovisual fusion[END_REF]. We can see that using more frames helps to remove the ambiguity of another person besides the speaker. However, using more frames tends to break our assumption that the speaker stays at the same position. Thus, we generally adopt 40 frames to compute our experimental results. On our laptop whose CPU is Intel Core2 1.83G with 1G RAM, it takes about 31 seconds.

We tested the performance of our algorithm for different backgrounds, with results shown in Figure . 5 (a-c). 40 frames were used for computation, although only one of them is shown. The results of our algorithm show minor changes for different backgrounds. We also implemented the method in [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF], whose manually designated seeds and the segmentation results over the same 40 frames are shown in Figure . 5 (d-g). Note that this does not mean a comparison of performance, as the results by [START_REF] Boykov | Graph Cuts and Efficient N-D Image Segmentation[END_REF] can be improved iteratively by adding more seeds. However, the results demonstrated that, through using audio information, our method achieved better robustness to the different complex stationary or non-stationary backgrounds. As both frontal and lateral views of the speaker face were photographed, we applied our method to the data using completely same parameters. The results are shown in Figure . 6. Since frontal and lateral views are two extreme cases of the view change, the success of our method to process them elegantly in the same framework demonstrated its robustness against different views.

We also applied our method to several other sequences with same parameters, where single or multiple persons are photographed. The experimental results are shown in Figure . 7. Sequences (a-c) demonstrated the results of other single speakers under different backgrounds. For (c), our method could segment out the mouth region of the speaker only. As the man in (c) moved his mouth only when speaking and demonstrated none of other speaking-related actions, our method can segment his mouth region only. Sequences (d-f) demonstrate the results for multiple persons. Our method correctly located the current speaker and detected their face regions in most cases.

To give a quantitative evaluation of our detection result, we have manually labeled the face regions for the first frame of four video sequences. The ground truth and the detection rate of our method were shown in Figure . 8. In most cases, our method can extract the face region with high detection rate.

Conclusions and future works

In this work, we have developed a method to find speaker face region in gray images robustly against views and backgrounds by integrating audiovisual correlation analysis into Graph Cut-based image segmentation framework. We have shown that our method is capable of finding less fragmented speaker face regions than previous methods for both single and multiple persons under different views and different complex backgrounds.

Our current evaluation of audiovisual correlation is sensitive to the noise. Visual noise may yield incorrect optical flow in untextured regions. Audio noise may disturb the frame energy estimation and decrease the audiovisual correlation. We plan to try other reliable methods to compute optical flow and other robust audio features, especially the audio features in frequency domain. Additionally, our method requires the speaker to stay at relatively the same position in the statistical time span. We will consider methods to eliminate this constraint.

Fig. 1 .

 1 Fig. 1. Localized speaker face regions and application demonstrations. (a) is the original image. (b-c) show the speaker face region localized by our method. (d-e) show the imposed special effects based on our results.

Fig. 2 .

 2 Fig. 2. Audiovisual correlation using different optical flow elements. The whiter a pixel, the higher its correlation. (a-d) show the original image, correlation by horizontal element, vertical element and mode, respectively.

Fig. 3 .

 3 Fig. 3. A demonstration of the N-D image and the neighborhood.

Fig. 4 .

 4 Fig. 4. Statistical audiovisual correlation using different frame numbers. (b-d) demonstrate the analyzed correlation values using 20, 40 and 80 video frames with speech. Correlation values are normalized independently in (b-d).

Fig. 5 .

 5 Fig. 5. Estimated results by the method in [1] and ours. Areas blended with blue mean estimated background. Boundary pixels are shown as white. (a-c) show our segmentation results of original video, video merged with a static background and video merged with a non-stationary background, respectively. (d) shows the user designated seeds. (e-f) are the segmentation results by the method in [1].

Fig. 6 .

 6 Fig. 6. Estimation results for different views. (a) and (c) show the detected speaker face region for a frontal view. (b) and (d) show the results for a lateral view.

Fig. 7 .

 7 Fig. 7. The experimental results for other persons. Sequences (a-c) are the results when only a single person exists. Sequences (d-f) are results when multiple persons exist.

Fig. 8 .

 8 Fig. 8. Ground truth and the detection rate of our method. The ground truth in the first row shows the manually labeled face region superimposed over the original image.


Laurent Chatre
email: chatre@cyceron.fr

Miria Ricchetti
email: miria.ricchetti@pasteur.fr

mTRIP, an Imaging Tool to Investigate Mitochondrial DNA Dynamics in Physiology and Disease at the Single-Cell Resolution

Keywords:

come

Introduction

Mitochondria are highly dynamic organelles that can fuse and divide to produce a variety of morphologies ranging from individual entities to interconnected tubular networks, which are functional to cell growth, cell physiology, and disease [START_REF] Chan | Mitochondrial dynamics and its involvement in disease[END_REF]2]. In eukaryotes, mitochondria play a central role in the energy metabolism that is regulated by the nuclear genome as well as the organelle genome. In mammals, individual mitochondria carry multiple copies of double-stranded circular mtDNA, packed into nucleoids structures and autonomously replicated and transcribed [3,4]. The human mtDNA is a 16.5 kbp molecule coding for two ribosomal RNAs, 22 transfer RNAs, and 13 protein-coding genes that are transcribed into polycistronic precursor RNAs and then processed to mature mRNAs, rRNAs, and tRNAs ( [START_REF] Ojala | tRNA punctuation model of RNA processing in human mitochondria[END_REF]) (Fig. 1). The dynamics of mtDNA replication are still debated, and distinct models are proposed ( [6][7][8], for reviews see [9][10][11]). Nevertheless, in these models initiation of replication takes place at the O origin, and the nascent heavy (H) strand may subsequently pause about 700 nucleotides downstream generating the 7S DNA, which produces a characteristic triple displacement loop or D-loop [12]. mtDNA replication and transcription have been extensively characterized by biochemical approaches, which assess values within large cell populations. In addition, superresolution imaging recently revealed, at the single cell level, that mtDNA replication occurs only in a subset of nucleoids [13,14]. These findings underscore the urgency of combined protein and DNA analysis of mitochondrial substructures. However, classic FISH procedures may damage protein epitopes and are therefore not suitable also for immunofluorescence analysis, thereby limiting the simultaneous track of mitochondrial nucleic acids and proteins. stranded mitochondrial genome that is not engaged in replication initiation is not detected by mREP. mTRANS, a mix of three DNA probes located on different regions of the mitochondrial genome, detects specific mtRNAs which are considered representative of global mtRNA levels (Fig. 1).

Fig. 2

Scheme of mTRIP labeling. Summary and chronology of steps of mTRIP labeling described in detail in this document. After labeling, 3D confocal acquisition and the subsequent treatment of acquisitions are shown. 3D acquisition is necessary for fluorescence quantification of the entire cell volume, but also 2D acquisition of a single plan (and thereby fluorescence quantification of this single plan) can be performed (not shown here)

We generally associate mTRIP analysis of mitochondrial nucleic acids with immunofluorescence of a mitochondrial outer membrane protein (TOM22 or TOMM22) to identify the mitochondrial network. These labeling allow for the detection and quantification of at least three classes of mitochondrial subpopulations: (1) replication initiation active and transcript-positive (Ia-Tp); (2) replication initiation silent and transcriptpositive (Is-Tp); and (3) replication initiation silent and transcriptnegative (Is-Tn) [15], revealing that individual mitochondria are strongly heterogeneous within human cells (Fig. 3a-d). This heterogeneity is functional to physiological changes, as during the cell cycle [16], and can be also affected during stress and disease [15]. mTRIP can be also associated with MitoTracker analysis, which detects mitochondria in living cells, based on its binding to thiol groups present in the organelle (Fig. 3e-h). Other probes than mREP and mTRANS can be produced to target specific regions of interest in the mitochondrial genome (see Ref. 15). For instance, mTRIP labeling with either a LSP (Light Strand Promoter)-specific probe or an ND6 (NADH-ubiquinone oxidoreductase chain 6)-specific probe was combined with immunostaining of the mitochondrial transcription factor and DNA binding protein TFAM for colocalization experiments. ND6 is the only gene located in the L-strand of the mitochondrial genome (see Fig. 1), and is a component of the respiratory complex I. mTRIP contributed to show that the mitochondrial fraction of the protooncogene MDM2 directly impacts on TFAM binding to the LSP, thereby affecting ND6 transcription of and thereby the respiratory complex I function in cancer cells [17]. Described and novel mTRIP probes, combined with a variety of immunofluorescence and mitochondrial markers (Fig. 4) should help decipher mtDNA dynamics and their impact under physiological changes, including stress, and in disease. Notably, the mREP and mTRANS probes alone can act as markers of functional mtDNA processing. The investigation of functional mtDNA processing in single cells has relevance beyond classical mitochondrial activity (oxidative phosphorylation). For instance, mTRIP labeling with these two probes contributed to demonstrate that mtDNA acquired from the host mouse by mtDNA-depleted grafted cancer cells was not only present but also fully functional in individual cells a few days after grafting [START_REF] Bajzikova | Reactivation of dihydroorotate dehydrogenase-driven pyrimidine biosynthesis restores tumor growth of respiration-deficient cancer cells[END_REF]. This work demonstrated that acquisition and processing of host mtDNA, and thereby restoration of mitochondrial respiration, are essential for tumorigenesis, although not for ATP production but rather pyrimidine biosynthesis through dihydroorotate dehydrogenase (DHODH) reactivation [START_REF] Bajzikova | Reactivation of dihydroorotate dehydrogenase-driven pyrimidine biosynthesis restores tumor growth of respiration-deficient cancer cells[END_REF]. In another study, mTRIP detection of mtDNA initiation of replication using the mREP probe helped understanding the dynamics of mtDNA synthesis/depletion. Indeed, a novel function of the endonuclease G (EndoG) both as a stimulator of mtDNA replication initiation and mtDNA depletion was identified [START_REF] Wiehe | Correction: endonuclease G promotes mitochondrial genome cleavage and replication[END_REF][START_REF] Wiehe | Endonuclease G promotes mitochondrial genome cleavage and replication[END_REF]. Further In this study, experiments that included the mTRANS probe contributed to demonstrate that removal of damaged mtDNA by EndoG is compensated by mtDNA replication to maintain mtDNA transcription and mitochondria homeostasis [START_REF] Wiehe | Correction: endonuclease G promotes mitochondrial genome cleavage and replication[END_REF][START_REF] Wiehe | Endonuclease G promotes mitochondrial genome cleavage and replication[END_REF].

mTRIP is therefore a versatile and powerful tool to investigate single cell dynamics of mitochondrial DNA processing in physiology and disease.

Materials

Prepare all solutions using ultrapure water when needed. Follow all waste disposal regulations when disposing of waste materials.

2.1. Probe Preparation and Labeling 3. Denaturation: incubate the reaction mixture at 80 °C for 10 min in the dark.

4. Precooling step: transfer the denatured reaction mixture containing tube at 37 °C for at least 30 min in the dark. This time can be extended to a maximum of 1 h at 37 °C in the dark. This step helps lowering unspecific binding of salmon sperm DNA versus specific binding of DNA probes. Keep denatured reaction mixture under these conditions until the end of Subheading 3.4. 12. The favorite IF procedure, if different form the one described here, can alternatively be performed.

Preparation of Cells on Coverslip

13. Reliable quantification requires fluorescence measurement of the cell volume. Cell surface (2D-acquisition) can also be acquired instead of cell volume, but the corresponding quantification will not measure the total cell fluorescence.

14. mTRIP tool is covered by patent applications: EP2500436 and WO2012123588 "Method, probe and kit for DNA in situ hybridization and use thereof."

15. mTRIP can be used with paraformaldehyde-fixed human cells and paraffin-embedded human tissue sections (method not described here).

Fig. 1

 1 Fig. 1 Schematic representation of the human mitochondrial DNA and probes used in mTRIP. The H (heavy) and light (L) strands of the mitochondrial genome are indicated in the external and inner circle, respectively, with the position and name of single genes within. tRNA genes are indicated with a black dot and the corresponding letter. All genes are located on the H-strand, with the exception of ND6 and several tRNAs, located on the L-strand. Magnification of the D-loop region (which contains H-strand origin of replication (O ) and promoters of both H (PH1 and PH2) and L (PL) strands) is shown on top. Blue arrows indicate the direction of DNA replication from O and O . The position of probe mREP and probes 1, 6, and 11 is indicated with a red box. mTRANS is a mix of probes 1, 6, and 11, which targets rRNA as well as several coding genes and tRNA on both strands

Fig. 3 mTRIPFig. 4

 34 Fig. 3 mTRIP colabeling with either immunofluorescence mitochondrial proteins or Mitotracker. Panels a-d: Colabeling of TOMM22 immunofluorescence (green, panel a), mTRANS (red, panel b), and mREP (blue, panel c) probes, shows mitochondrial initiation of replication, transcription, and mitochondrial network, respectively. In merge, mREP essentially colocalizes with mTRANS (purple; arrow; Ia-Tp, replication Initiation active and Transcript positive), and independent mTRANS labeling (orange; arrow; Is-Tp, replication Initiation silent and Transcript active), as well as replication Initiation silent and Transcript negative mitochondrial structures (green; arrow, Is-Tn) are also observed. Panels (e-h) Colabeling of MitoTracker Deep Red (purple; panel e), mTRANS (green; panel f), mREP (red, panel g), and merge (panel h, which was also counterstained with Hoechst, blue, for nucleus detection). MitoTracker labeling results in a more diffused signal than immunofluorescence and mTRIP, therefore colocalization with other markers appears less defined in panel h than in panel d. Ia-Tp, Is-Tp, and Is-Tn mitochondrial structures are detected also with this colabeling (presence or absence of labeling in panels e-g is shown

Be careful as the following steps are dedicated to DNA probes fluorescent labeling by nick translation and are crucial for mTRIP. Consider that no other technique of DNA labeling than nick translation has been tested in this context.

	e.Proofing | Springer	https://eproofing.springer.com/books_v3/printpage.php?token=...
	4. 2% PFA can be stored at 4 °C for a couple of weeks. 2. Amplify the mTRANS component probe 1 by PCR using at least 8. Preparation of the mREP DNA probe:
	5. Permeabilization buffer: 0.5% Triton X-100 in 1× PBS (mix 50 µl of 100% Triton X-100 with 9550 µl 1× PBS). 100 ng/µl of total genomic DNA and LA Taq DNA polymerase (a) Resuspend 1 µg of pure PCR product in a maximum (recommendation: annealing temperature: 56 °C; extension: volume of 14 µl in a 0.2 ml thin-wall 8-tube strip with caps 68 °C). for regular thermal cycler (PCR machine).
	6. 20× saline sodium-citrate (SSC) buffer: 150 mM NaCl, 15 mM Na Citrate × 2H O, pH 7.0). For SSC preparation, dissolve 175.3 g of NaCl and 88.2 g of sodium citrate in 800 ml of ultrapure water. 3. Amplify the mTRANS component probe 6 by PCR using at least If the volume is smaller, adjust to 14 µl with PCR-grade 100 ng/µl of total genomic DNA and LA Taq DNA polymerase (recommendation: annealing temperature: 56 °C; extension: water from the kit.
	Adjust the pH to 7.0 with 14 N solution of HCl. Adjust the volume to 1 l with additional ultrapure water. Sterilize by autoclaving. 68 °C). 9. Preparation of the mTRANS DNA mix probes.
	Store at room temperature up to 6 months. 4. Amplify the mTRANS component probe 11 by PCR using at least (a) Mix 333 ng of mTRANS probe 1, 333 ng of mTRANS
	100 ng/µl of total genomic DNA and LA Taq DNA polymerase probe 6, and 333 ng of mTRANS probe 11 (equimolar
	7. Pretreatment buffer for permeabilized cells: 50% formamide-2× (recommendation: annealing temperature: 56 °C; extension: amounts of the three probes) for a final amount of 1 µg of
	SSC in 1× PBS. Store the solution at room temperature up to 68 °C). DNA mix, in maximum volume of 14 µl in a 0.2 ml thin-1. Lysis buffer to extract total genomic DNA: 0.2% SDS, 5 mM EDTA, 0.2 mg/ml Proteinase K in 1× phosphate buffered saline (PBS) buffer. 6 months in the dark. 8. Denaturation buffer for permeabilized cells: 70% formamide-2× wall 8-tube strip with caps for regular thermal cycler (PCR 5. Purify the PCR products using 2% agarose electrophoresis gel and conventional PCR products cleanup kit. machine).
	2. Sodium acetate 3 M, pH 5.2, cold ultrapure isopropanol and SSC in 1× PBS.
	ultrapure water. Store the solution at room temperature up to 6 months in the dark.
	3. Taq DNA polymerase (we recommend LA Taq DNA polymerase
	TaKaRa). 2.3. mTRIP Hybridization
	4. Primers for PCR amplification of the mREP probe from total 1. Parafilm.
	genomic DNA:
	2. Hybridization buffer: 10% dextran sulfate-50% formamide-2× Coordinates of all primers are according to NC_012920 GenBank. SSC in 1× PBS).
	Store the solution at room temperature up to 6 months in the dark mREP (98 nucleotides, nt): coordinates 446-544. (see Note 3).
	Forward 5′-ACATTATTTTCCCCTCCC-3′. 3. 100 ng/µl salmon sperm DNA: dilute stock salmon sperm DNA
	Reverse 5′-GGGGTATGGGGTTAGCAG-3′. with 1× PBS.
	7. Choose the fluorescence label for mREP and mTRANS DNA
	Store the solution at -20 °C. probes (see Note 8). If mTRIP is coupled to immunofluorescence 5. Primers for PCR amplification of the mTRANS probe from total genomic DNA: and Hoechst 33342, resulting in four-color channels, the 4. Washing buffer A: 2× SSC in 1× PBS. recommended choice is mREP labeled with Atto550 NT kit,
	mTRANS is an equimolar mix of three DNA probes: probe 1, 5. Washing buffer B: 1× SSC in 1× PBS. mTRANS labeled with Atto488 NT kit, immunofluorescence with
	a Cy5-conjugated secondary antibody, and Hoechst.

6. Estimate the amount of the purified PCR product. It is essential to get 1 µg of PCR product at a concentration of 75 ng/µl or higher. If this amount of DNA product is not reached, one can concentrate the DNA by sodium acetate and isopropanol precipitation, then resuspend in the appropriate volume of ultrapure water to reach the required concentration. Note that precipitation of highly diluted DNA is not recommended at it may result in salt concentration higher than necessary, in spite of intense washing.

If the volume is smaller adjust to 14 µl with PCR-grade water from the kit.

The following steps are identical for mREP and mTRANS fluorescence labeling. 2

  µl of 20 ng/µl of fluorescent DNA probe (40 ng DNA probe are used per coverslip at a final concentration of 1.6 ng/µl) with 4 µl of 100 ng/µl salmon sperm DNA (400 ng of salmon sperm DNA are used for 40 ng of DNA probe, and per coverslip) and 19 µl of hybridization buffer. If two probes (mREP and mTRANS) are used, the reaction mix is done in a final volume 25 µl per coverslip. Gently mix 2 µl of 20 ng/µl fluorescent mREP probe (40 ng of DNA probe are used per coverslip; final concentration 1.6 ng/µl) and 2 µl of 20 ng/µl fluorescent mTRANS probe (40 ng DNA probe are used per coverslip; final concentration 1.6 ng/µl) with 8 µl of 100 ng/µl of salmon sperm DNA (800 ng of salmon sperm DNA are used for 2× 40 ng of DNA probes and per cove slip) and 13 µl of hybridization buffer.

	2. Choice 2: TWO probes, preparation of the reaction mixture (in a
	1.5 ml tube):

At this point, either label the sample by immunofluorescence, and in this case go to Subheading 3.7 (mTRIP coupled to immunofluorescence) or do not label the sample by immunofluorescence (this is also the case for MitoTracker labeling), and in this case:

  Springer https://eproofing.springer.com/books_v3/printpage.php?token=... coverslip with pretreated permeabilized cells) on the top of a metal block heater at 75 °C for 5 min to denature the sample. 18. Immediately transfer the 6-well culture plate on ice (4 °C) and keep on ice at least 1 min and not longer than 10 min (see Note 11). Seal the slide with varnish and keep the mounted slide in a dark and clean box either at room temperature if confocal analysis is done the done the same day, or at 4 °C until confocal analysis is done (it is recommended to analyze the slide not later than 2 weeks from labeling). The fluorescence color of the DNA probes can be different from the ones used here, and for this, appropriate fluorophores must be selected. Proofing | Springer https://eproofing.springer.com/books_v3/printpage.php?token=... 23 of 27 18/01/2021, 14:36 9. In this case, use mREP-Atto550 and mTRANS-Atto-488 for mTRIP labeling, and Hoechst. 10. Treatment with DNaseI (recommended 100 U/ml) or RNaseA (recommended 100 µg/ml) allows identification of the DNA and RNA components of the mTRIP signal. Treatment with RNaseH (recommended 100 U/ml) allows identification of RNA/DNA structures, usually at replication origins and in transcription bubbles. For combined use of nucleases and the respective readouts on mitochondrial nucleic acids by mTRIP, see Ref. 15. 11. Do not keep denatured samples at room temperature, as this condition will interfere with the hybridization step. Keep the samples always on ice until hybridization.

	e.Proofing | Springer	https://eproofing.springer.com/books_v3/printpage.php?token=...
	3.5. mTRIP Hybridization 1. Put a 25 µl drop of denatured and precooled reaction mixture which contains DNA probe(s) on an appropriate surface of Parafilm (at least 3.6 × 3.6 cm for 18 mm diameter coverslip). For multiple labeling, use a single surface of Parafilm with enough room for multiple and well separated drops. 2. Put the coverslip upside down, with the face containing cells (samples previously undergone denaturation) directly on the drop. 3. Incubate at 37 °C for 15 h either at the top of a heating metal block in the dark (cover with plastic top to set dark) or on a humid chamber at 37 °C in the dark. 3.6. mTRIP Washing 6. Incubate the coverslip for 1 h in the dark with 10 µg/ml Hoechst 33342 in 1× PBS. 7. Wash 5× with 1× PBS. 8. Mount the coverslip on a clean and dry glass slide using the favorite mounting buffer or a drop of 50% glycerol in 1× PBS. 9. 3.7. mTRIP Coupled to Immunofluorescence 3 (panels b, c, f, g). The mTRIP procedure is covered by a patent application for commercial use (Note 14), which also includes tissues sections (Note 15). 4. Notes AQ3 1. Wizard SV Gel, PCR clean-Up system (Promega), and NucleoSpin Extract II (Macherey-Nagel) worked well in our hands. 2. Labeling of DNA probes with other techniques than Nick Translation has not been tested. We do not exclude that other DNA probes labeling techniques are compatible with mTRIP. 3. It is strongly recommended to prepare the hybridization buffer several days (at least 2 days) before the experiment as dextran sulfate takes a long time to dissolve. 4. Alternative blocking buffers can also be used. Follow the immunofluorescence (IF) protocol suggested below, (see Note A 3D-reconstructed human cell stained by mTRIP is shown in Fig. 5. Unconjugated rabbit polyclonal anti-TOMM22 is used to
		illuminate the mitochondrial network. However, you can use other
		1. After 15 h incubation, remove the coverslip from Parafilm and put primary antibodies for detecting the mitochondrial network, or
		it back into a clean cell culture plate (6-well plate). other cellular structures or proteins of interest.
		The following steps must be performed in the dark: 6. The choice of the secondary antibody-conjugated fluorescence
		depends on the available spectra after exclusion of the
		2. Wash 2× with 2× SSC in 1× PBS for 2 min at room temperature fluorophore(s) used for DNA probes (mTRIP). All combinations
		(RT) with gentle shaking. compatible with available fluorophores and the equipment
		1. Culture cells under appropriate culture conditions in (6-wells or 12-wells) culture plates containing a clean coverslip at the bottom 3. Wash 2× with 1× SSC in 1× PBS for 2 min at RT with gentle (fluorescence filters) of your microscope can be used.
		of each well. Ideally cells should be grown at about 50-60% shaking. 7. Cancer and immortalized cell lines may have mutated and highly
		confluence to avoid plans with packed cell during microscope heteroplasmic mitochondrial genomes. 4. Wash 2× with 0.1× SSC in 1× PBS for 2 min at RT with gentle analysis. shaking. 8.
		5. Wash 2× with 1× PBS for 2 min at RT with gentle shaking.
	20 of 27	18/01/2021, 14:36

If mTRIP is coupled to MitoTracker labeling, incubate live cells with MitoTracker (e.g., MitoTracker Deep Red) for 1 h in ® ® e.Proofing | ® ® e.

of 27 18/01/2021, 14:36

e.Proofing | Springer https://eproofing.springer.com/books_v3/printpage.php?token=... 9 of 27 18/01/2021, 14:36

e.Proofing | Springer https://eproofing.springer.com/books_v3/printpage.php?token=...

18/01/2021, 14:36

Acknowledgments

This work was supported by Association Nationale contre le Cancer (ARC 4022 and SFI20111204038), PTR-Institut Pasteur (PTR217), DARRI-Institut Pasteur (projects P790319 and PasteurInnov 14/152), and Agence Nationale pour la Recherche (ANR 11BSV202502).

6. Incubate at RT for 1 h in the dark. The incubation time depends on the primary antibody.

7. Wash 3× in 1× PBS for 2 min at RT with gentle shaking. 8. Dilute secondary antibody (1:1000) in 1% BSA and 1× PBS, and add 10 µg/ml of Hoechst 33342. 9. Put a 50 µl drop of diluted secondary antibody mix on an appropriate surface of Parafilm.

10. Put the coverslip upside down, with the face containing cells directly on the drop.

11. Incubate at RT for 1 h in the dark.

12. Wash 5× in 1× PBS for 2 min at RT with gentle shaking.

13. Mount the coverslip on a clean and dry slide using the favorite mounting buffer or a drop of 50% glycerol in 1× PBS.

14. Seal the slide with varnish and keep the mounted slide in a dark and clean box either at room temperature if confocal analysis is done the same day, or at 4 °C until confocal analysis is done (it is recommended to analyze the slide not later than 2 weeks from labeling).

Imaging Acquisition and Fluorescence Intensity Quantification

1. Acquisition for subsequent quantification is done with confocal microscope (see Note 13).

2. One can do either two-dimensional (2D) or three-dimensional (3D) acquisition. In the last case 3D reconstruction of the image should follow 3D acquisition.

3. With image format (.tiff) of the acquired image, one can quantify the fluorescence intensity per per cell using the Image J software.