Member, IEEE Effrosyni Doutsi

Lionel Fillatre
email: lionel.fillatre@i3s.unice.fr

Member, IEEE Marc Antonini

Julien Gaulmin

Retina-inspired Filter

Keywords: Human retina, OPL layer, Spatiotemporal filtering, Weighted difference of gaussian

This paper introduces a novel filter which is inspired by the human retina. The human retina consists of three different layers: the Outer Plexiform Layer (OPL), the inner plexiform layer and the ganglionic layer. Our inspiration is the linear transform which takes place in the OPL and has been mathematically described by the neuroscientific model "virtual retina". This model is the cornerstone to derive the non-separable spatiotemporal OPL retina-inspired filter, briefly renamed retinainspired filter, studied in this paper. This filter is connected to the dynamic behavior of the retina, which enables the retina to increase the sharpness of the visual stimulus during filtering before its transmission to the brain. We establish that this retinainspired transform forms a group of spatiotemporal Weighted Difference of Gaussian (WDoG) filters when it is applied to a still image visible for a given time. We analyze the spatial frequency bandwidth of the retina-inspired filter with respect to time. It is shown that the WDoG spectrum varies from a lowpass filter to a bandpass filter. Therefore, while time increases, the retinainspired filter enables to extract different kind of information from the input image. Finally, we discuss the benefits of using the retina-inspired filter in image processing.

I. INTRODUCTION

T HE decryption of the Human Visual System (HVS) is of a great interest. Studying its physiological and functional properties is helpful to many scientific fields, including signal and image processing. There are several bio-inspired filters for image analysis which have been built in order to mimic the HVS and the visual perception. Many filters are used as an approximation of the early vision system like static Difference of Gaussian (DoG), Gabor or foveating filters. This paper is focused on DoG filters which have been extensively studied in the past [START_REF] Field | What is the goal of sensory coding?[END_REF]- [START_REF] Winnemöller | XDoG: An eXtended difference-of-Gaussians compendium included advanced image stylization[END_REF]. Most of the DoGs are static spatial filters which have been proposed for instance for perceptual visual quality metrics like the image quality assessment [START_REF] Pei | Image quality human visual DOG model fused with random forest[END_REF], for edge detection [START_REF] Marr | Theory of edge detection[END_REF]- [START_REF] Basu | Gaussian-based edge-detection methods-a survey[END_REF], and for object recognition [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF]. Some works have studied multiscale DoG filters, introducing time as the scale level, in order to approximate the spatiotemporal behavior of the visual system. Laplacian pyramid [START_REF] Burt | The Laplacian pyramid as a compact image code[END_REF], [START_REF] Andelson | Pyramid methods in image processing[END_REF], visual saliency [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF], [START_REF] Frintrop | Traditional saliency reloaded: A good old model in new shape[END_REF], speed and direction estimation [START_REF] Young | The gaussian derivative model for spatial-temporal vision: I. cortical model[END_REF],

image stylization [START_REF] Winnemöller | XDoG: An eXtended difference-of-Gaussians compendium included advanced image stylization[END_REF] and bio-inspired codecs [START_REF] Masmoudi | Frames for exact inversion of the rank order coder[END_REF], [START_REF] Masmoudi | Streaming an image through the eye: The retina seen as a dithered scalable image coder[END_REF] are some of these attempts. However, the spatiotemporal receptive fields of the early HVS are more complex and above all dynamic. Wohrer et al [START_REF] Wohrer | Virtual retina: A biological retina model and simulator, with constrast gain control[END_REF] have proposed a more realistic model of the retina based on neuroscientific experiments. This paper proposes and studies a dynamic spatiotemporal DoG filter which is derived from [START_REF] Wohrer | Virtual retina: A biological retina model and simulator, with constrast gain control[END_REF] and adapted to the image processing field.

Our filter, which is inspired from a realistic model of the Outer Plexiform Layer (OPL), is named the non-separable spatiotemporal OPL retina-inspired filter but it is briefly called the retina-inspired filter. We have already briefly discussed about it in our previous works [START_REF] Doutsi | Retina-inspired filtering for dynamic image coding[END_REF], [START_REF] Doutsi | Event-based coding of images using a bio-inspired frame[END_REF]. The contributions of this paper are the followings. We first review the main neuroscientific models which have been proposed in order to mimic the linear transform which takes place in the OPL layer and the first group of cells of the retinal tissue [START_REF] Deangelis | Spatiotemporal receptive field organization in the Lateral Geniculate Nucleus of cats and kittens[END_REF], [START_REF] Wohrer | Virtual retina: A biological retina model and simulator, with constrast gain control[END_REF], [START_REF] Kuffler | Neurons in the retina: Organization, inhibition and excitation problems[END_REF]- [START_REF] Fleet | Spatiotemporal inseparability in early visual processing[END_REF]. This review underlines the importance of time for describing the way the retina cells are structured and work. Then, we derive from these previous works a realistic but simple retina-inspired filter. We show that this can be written as a Weighted DoG (WDoG) filter whose weights evolve in time when it is applied to a still image. Finally, we propose a full analysis of the WDoG filter in space and in frequency domains. We prove that its spatial frequency bandwidth depends on time, which means that while time increases the retina-inspired filter extracts different kind of information. More precisely, in the beginning, our filter is lowpass and it turns into bandpass while time increases. We also show that the filter stops to evolve asymptotically. As a result, there exists a certain time when the filter has already extracted all the necessary data.

In section II, we introduce the structure of the retina and we focus on the OPL on which the retina-inspired transform is based. In section III, we build the retina-inspired filtering model and study how it is applied to a still image. In section IV, we study the spatial and frequential behaviors of the retina-inspired filter considered as a group of WDoG filters. In section V, we illustrate the behavior of the filter with some numerical results. Last but not least, in section VI, we discuss the applications of this paper and some possible extensions of this work.

II. NEUROSCIENTIFIC BACKGROUND: OPL MODEL

The visual system [START_REF] Hubel | The visual cortex of the brain[END_REF] is part of the Central Nervous System (CNS) and consists of many different areas which participate to the coding of a visual stimulus. The most important and better studied areas are the retina, the optic nerve, the Lateral Geniculate Nucleus (LGN) and the visual cortex as depicted in Fig. 1. The retina is a layer of tissue, lining the inner surface of the eye, which is responsible to capture the visual stimuli and transform it into a sequence of electrical impulses (spike trains). This code of spikes is transmitted through the optic nerve to the LGN cells and finally to the primary visual cortex. The cells in visual cortex are more complex [START_REF] Livingstone | Anatomy and physiology of a color system in the primate visual cortex[END_REF], [START_REF] Desimone | Contour, color and shape analysis beyond the striate cortex[END_REF] and their functions are related to edge and orientation detection, motion estimation, discrimination of the shape or the color, etc.

I(X, t)

Fig. 1: The visual system pathway: the retina captures the visual stimuli I(X, t) and transmits it to the visual cortex via the optic nerve and the LGN.

The input of the HVS is the luminance of light which is modeled as a 3D spatiotemporally varying visual stimulus I(X, t) with X ∈ R 3 and t ∈ R. The luminance is transformed inside the retina [START_REF] Masland | The fundamental plan of the retina[END_REF]- [START_REF] Vanessen | Information processing in the primate visual system: An integrated systems perspective[END_REF] to fit the human brain. This transformation is serial due to the retina's structure. The retinal neurons can be separated into three groups: the Outer Plexiform Layer (OPL), the Inner Plexiform Layer (IPL) and the Ganglionic Layer (GL) as depicted in Fig. 2. The OPL layer is structured by photo-receptors, horizontal and bipolar cells. Bipolar and amacrine cells build the IPL layer. The GL layer is constructed by the amacrine and the ganglion cells [START_REF] Wohrer | Retinal filtering and image reconstruction[END_REF], [START_REF] Masland | Cell populations of the retina: The proctor lecture[END_REF]. The overlapping of the layers is due to the feedforward and feedback network of the retinal cells. The OPL layer captures the incoming light (photons) and linearly transforms it into an electrical current. The IPL layer is responsible for a nonlinear rectification of the electrical signal. The GL produces the spike trains which is a kind of code based on electrical impulses [START_REF] Wohrer | Virtual retina: A biological retina model and simulator, with constrast gain control[END_REF].

This paper is dedicated to the study and analysis of the OPL layer in order to build a novel retina-inspired filter. According to the law of optics, the 3D visual stimuli I(X, t) is projected onto the retina via the lens (the optics of the eye is detailled in [START_REF] Ögmen | The geometry of visual perception: Retinotopic and nonretinotopic representations in the human visual system[END_REF]). Hence, the 3D luminance I(X, t) is simplified into a 2D luminance f (x, t) where x ∈ R 2 . We assume that the visual stimulus is flashed for a given time T > 0 and it is constant during this time interval. This involves that the 2D visual stimulus is written as f (x, t) = f (x)½ [0,T ] (t) where It is important to note that cells of the same kind lie within a common layer.

f (x) ∈ L 1 (R 2 ) is a still image and ½ is the indicator function such that ½ [0,T ] (t) = 1 if 0 ≤ t ≤ T , otherwise 0. The space L 1 (R 2 ) is the set of the Lebesgue integrable functions from R 2 to R.

A. OPL Layer

The OPL cells receive as an input the visual stimulus f (x, t) which is spatiotemporally transformed into an electrical signal. This transformation takes place inside the receptive field (RF) of each cell. Kuffler [START_REF] Kuffler | Neurons in the retina: Organization, inhibition and excitation problems[END_REF] proposed to shape the RF by two concentric nested circles or ellipses, which are termed centersurround. The smaller circle (or ellipse) corresponds to the center and the larger one is the surround. Let Ω i ⊆ R 2 be the RF of a bipolar cell centered in x i ∈ Ω i . Let A(x i , t) be the electrical signal produced by the RF Ω i when the input signal is f (x, t). Considering the bipolar cell is a time invariant linear system, the following linear approximation of the OPL retinaltransform has been introduced in [START_REF] Wohrer | Retinal filtering and image reconstruction[END_REF]:

A(x i , t) = +∞ t ′ =0 x ′ ∈Ωi K(x i -x ′ , t-t ′ )f (x ′ , t ′ )dx ′ dt ′ (1)
where K(x, t) is the spatiotemporal transform of a single bipolar cell, also known as the Point Spread Function (PSF) at time t. The above equation indicates that the electrical signal A(x i , t) depends linearly on the spatial neighborhood and the past values of the input stimuli located in the RF Ω i of the single bipolar cell centered in x i . Assuming that i) the number of cells is very large, ii) all the cells obey to the same spatiotemporal model (spatial invariance) and iii) the temporal point spread function K(x, t) is not restricted to the domain Ω i , the spatiotemporal transform (1) is approximated by the following spatiotemporal convolution denoted x,t * :

A(x, t) = t ′ ∈R x ′ ∈R 2 K(x -x ′ , t -t ′ )f (x ′ , t ′ )dx ′ dt ′ = (K x,t * f )(x, t), (2)
where x ∈ R 2 is located on the retina. Many models of the temporal point spread function K(x, t) have been proposed.

B. Approximations of K(x, t)

1) Spatial Receptive Field: The first mathematical approximation of the OPL was proposed by Kuffler [START_REF] Kuffler | Neurons in the retina: Organization, inhibition and excitation problems[END_REF]. A bipolar cell receives its input signal directly from a group of photoreceptors and/or a group of horizontal cells (Fig. 3). On the one hand, the output of two or more photoreceptors is averaged and transmitted to the center of the RF of the bipolar cell in order to excite it. This is approximated by a Gaussian filter G σc (x) given by:

G σc (x) = 1 2πσ 2 c exp - x 2 2σ 2 c = G σc ( x ), (3)
where x ∈ R 2 , x is the Euclidean norm of x and σ c is the standard deviation which tunes the spread of the Gaussian filter. On the other hand, the same or higher number of photoreceptors is linked to horizontal cells. An horizontal cell is strongly connected to neighbor horizontal cells averaging twice the initial input stimulus. The output of one or more horizontal cells is then propagated to the surround of the RF of the bipolar cell in order to inhibit it [START_REF] Hérault | Modeling Visual Perception for Image Processing[END_REF]. This signal is modeled by the Gaussian filter G σs (x) with σ c < σ s . As a result, the bipolar cell receives two signals of opposite signs. Finally, the center-surround activity K(x, t) of the RF of a bipolar cell is modeled as a DoG filter:

DoG(x) = G σc (x) -G σs (x).
Kuffler assumed that all these processes happen instantaneously. Hence, the PSF K(x, t) is constant for all time t.

2) Separable Spatiotemporal Receptive Field:

The first attempt to improve the static DoG filter was done by Marr [START_REF] Marr | Vision[END_REF]. Marr's theory contains an assumption of spatiotemporal separability adapting a temporal impulse response for each one of the decomposition layers which are formed by a static DoG:

K(x, t) = H(t)DoG(x), (4)
where H(t) is usually low-pass for "sustained" units, and band-pass for "transient" units.

3) Non-Separable Spatiotemporal Receptive Field: The models above were not accurate with time-varying stimuli f (x, t). Hence, Fleet [START_REF] Fleet | Spatiotemporal inseparability in early visual processing[END_REF] proposed a non-separable spatiotemporal center-surround model as an extension of the DoG. Electrophysiological studies have shown that the center and the surround have different time courses of response. In addition, the temporal delay between the response of the center and surround areas of the cells receptive field should also be considered. This model was interpreted by Fleet as a precursor to the extraction of velocity specific information [START_REF] Fleet | Spatiotemporal inseparability in early visual processing[END_REF]. Fig. 4: Propagation of the electrical signal to the receptive field of a bipolar cell according to Fleet [START_REF] Fleet | Spatiotemporal inseparability in early visual processing[END_REF].

The inseparability of space and time was highlighted and confirmed while studying the spectrum of the center-surround model. The dynamics of the model are due to the sensitivity changes of the response of the cells and the different phases of the center-surround areas [START_REF] Fleet | Spatiotemporal inseparability in early visual processing[END_REF]. Similar models have been proposed in order to describe how does the receptive field of neurons work in areas which come after the retina like the Lateral Geniculate Nucleus (LGN) or cortical neurons [START_REF] Deangelis | Spatiotemporal receptive field organization in the Lateral Geniculate Nucleus of cats and kittens[END_REF], [START_REF] Wohrer | Retinal filtering and image reconstruction[END_REF], [START_REF] Deangelis | Spatiotemporal organization of simple-cell receptive fields in the cat's striate cortex. i. general characteristics and postnatal development[END_REF]. The common point of all these models is the spatiotemporal inseparability, which confirms its importance.

A non-separable spatiotemporal retinal filtering as part of the visual system coding process has been mathematically introduced in virtual retina [START_REF] Wohrer | Retinal filtering and image reconstruction[END_REF]. This filter K(x, t), given in [START_REF] Pei | Image quality human visual DOG model fused with random forest[END_REF], is built by the combination of Gaussian spatial filters and Difference of Exponential (DoE) temporal filters, where the center C(x, t) and surround S(x, t) areas are structured according to ( 6) and ( 7) respectively:

K(x, t) = C(x, t) -S(x, t), (5)
C(x, t) = w c G σc (x)V (t), (6)
S(x, t) = w s G σs (x) V t * E τS (t), (7)
where w c and w s are constant parameters, G σc (x) and G σs (x) are spatial Gaussian filters standing for the center and surround areas respectively, V (t) is a low-pass filter and E τS (t) is an exponential temporal filter. The temporal filter V (t) is given by ( 8) and describes the DoE which stands for the spatial variation with respect to time. It is modeled with temporal low-pass filters [START_REF] Wohrer | Virtual retina: A biological retina model and simulator, with constrast gain control[END_REF]:

V (t) = E τG,n t * (δ 0 -w c E τC ) (t) if t ≥ 0, 0 otherwise, (8)
where the gamma temporal filter E τG,n (t) is defined by:

E τ,n (t) = t n exp(-t/τ ) τ n+1 if t ≥ 0, 0 otherwise, (9)
with n ∈ N, τ > 0, δ 0 (t) is the dirac function, E τC (t) is an exponential temporal filter, and t * stands for the temporal convolution. The exponential temporal filter is given by [START_REF] Masland | The fundamental plan of the retina[END_REF] for n = 0.

The mathematical model of C(x, t) and S(x, t) is totally linked to the way the retinal cells are connected and propagate the stimuli. To explain the non-separable spatiotemporal retina-inspired filter, we need to focus again on bipolar cells. We have already mentioned how they receive two opposite signals in their RF. In this model the key is the hierarchy of the retinal cells and their connectivity. The temporal behavior of the retina cells is described in Fig. 4. The horizontal cells are the first ones which receive an average signal by a group of photoreceptors at time t 1 . A smaller group of photoreceptors propagates an excitatory signal to the center of bipolar cells C(x, t) at time t 2 while horizontal cells receive this signal and they communicate and exchange information with adjacent horizontal cells. This causes a small delay E τs (t) until time t 3 in the propagation of the inhibitory signal S(x, t) coming from horizontal cells to bipolar cells.

III. RETINA-INSPIRED FILTERING OF A STILL IMAGE

The goal of this section is to simplify the retinal spatiotemporal filtering [START_REF] Deangelis | Spatiotemporal receptive field organization in the Lateral Geniculate Nucleus of cats and kittens[END_REF], based on the model ( 5), when it is applied to a still image. We show that the retina-inspired filter could be interpreted as a group of time varying WDoG filters. That means that, at each time, a new and different spatial WDoG filter arises. Consequently, while time increases, the retinainspired filter is able to extract different kinds of information from its input signal.

A. Equivalence with a Group of Time-varying WDoG Filters

A still image does not vary in time when it is flashed. This time invariance enables us to simplify the spatiotemporal convolution (2) as established in Proposition 1.

Proposition 1. Assume f (x, t) = f (x)½ [0,T ] (t) for all x ∈ R 2
and all t ∈ R. Then, the spatiotemporal convolution [START_REF] Deangelis | Spatiotemporal receptive field organization in the Lateral Geniculate Nucleus of cats and kittens[END_REF] turns into the spatial convolution:

A(x, t) = φ(x, t) x * f (x), ( 10
)
where φ(x, t) is a spatiotemporal WDoG filter weighted by two temporal filters R c (t) and R s (t):

φ(x, t) = w c R c (t)G σc (x) -w s R s (t)G σs (x) if t ≥ 0, 0 otherwise, (11) R c (t) = ½ [0,+∞) (t) t max{0,t-T } V (u)du, (12)
R s (t) = ½ [0,+∞) (t) t max{0,t-T } (V t * E τS )(u)du, (13)
for all x ∈ R 2 and for all t ∈ R.

Proof. The proof consists in calculating the convolution:

A(x, t) = K(x, t) x,t * f (x, t) = (C(x, t) -S(x, t)) x,t * f (x)½ [0,T ] (t) = w c G σc (x) x * f (x) V t * ½ [0,T ] (t) -w s G σs (x) x * f (x) V t * E τS t * ½ [0,T ] (t) .
For an integrable function U (t), a short calculation shows that

U t * ½ [0,T ] (t) =    0 if t < 0, t max{0,t-T } U (u)du otherwise. ( 14
)
Then, it is straighforward to derive [START_REF] Burt | The Laplacian pyramid as a compact image code[END_REF] with φ(x, t) defined in [START_REF] Andelson | Pyramid methods in image processing[END_REF].

This proposition is fundamental because it turns spatiotemporal filtering of the still image f (x) with the retinal filter K(x, t) into a simpler spatial convolution with the retina-inspired filter φ(x, t) which is a group of time-varying WDoGs. The temporal functions R c (t) and R s (t) act like weights and modify the WDoG spatial spectrum with respect to time (see Section IV). Since a DoG filter is a 2D spatially symmetric real function, the filter φ(x, t) can be simplified into φ(r, t) where the radius r is the norm of x. Fig. 5 plots the retinal filter φ(r, t) as a function of r = x 2 for two different cases related to the temporal filters R c (t) and R s (t) which are studied in the following subsection. For simplicity, it is assumed that r ∈ R and φ(r, t) is symmetric around r = 0 for all t. The parameters have been tuned according to neuroscientific results [START_REF] Masmoudi | Frames for exact inversion of the rank order coder[END_REF] which approximate the retinal spectrum and the speed of the retinal processing. Since the retina-inspired filter is a 2D spatially symmetric function, its spectrum is also symmetric. This spectrum, denoted φ(ω, t) where ω is the spatial angular frequency related to r, is shown in Fig. 5.

B. Spatiotemporal Behavior and Convergence

We are interested in studying the two temporal filters R c (t) and R s (t) which are responsible for the spatiotemporal behavior of the retina-inspired filter. First of all, we calculate their closed-form model in Proposition 2. This proposition is based on the following lemma. where P n (t) is a polynomial function in t of order n and α c and γ c are two reals, and

Lemma 1. Assume t ≥ 0, then J c (t) = t 0 V (u)du = P n (t) exp -t τ G + α c exp -t τ C + γ c , (15)
J s (t) = t 0 (V t * E τS )(u)du = Q n (t) exp -t τ G + α s exp -t τ S +β s exp -t τ C + γ s , ( 16
)
where Q n (t) is a polynomial function in t of order n and α s , β s and γ s are some reals.

Proof. See the Appendix.

Proposition 2. The temporal weights R c (t) in [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF] and R s (t) in ( 13) satisfy:

R c (t) = J c (t) if 0 ≤ t ≤ T, J c (t) -J c (t -T ) if T < t, (17)
R s (t) = J s (t) if 0 ≤ t ≤ T, J s (t) -J s (t -T ) if T < t, ( 18
)
where J c (t) is given in [START_REF] Masmoudi | Streaming an image through the eye: The retina seen as a dithered scalable image coder[END_REF] and J s (t) is given in [START_REF] Wohrer | Virtual retina: A biological retina model and simulator, with constrast gain control[END_REF].

Proof. The proof is based on the fact that R c (t) = J c (t) for 0 ≤ t ≤ T and

R c (t) = t t-T V (u)du = t 0 V (u)du - t-T 0 V (u)du
for t > T . Lemma 1 is used to deduce [START_REF] Doutsi | Retina-inspired filtering for dynamic image coding[END_REF]. The same equalities hold for R s (t).

The temporal weights R c (t) and R s (t) are illustrated in Fig. 6. The parameters in (a) and (b) have been tuned according to the parameters of Fig. 5 for T = +∞ and T < +∞ respectively. We should note that their shapes are very similar except that the surround temporal filter R s (t) appears with a small delay E τS (t) with respect to the center one. There is a high impact of the above characteristic on the spatiotemporal evolution of the filter. The delay E τS (t) is crucially important because for the very first few milliseconds, while R s (t) does not yet exist, the second term of the WDoG is zero. As a result, at the very beginning, the retina-inspired filter is a pure Gaussian with a very low amplitude since it is weighted by R c (t). Finally, we can note that R c (t) and R s (t) converge to a constant value. Hence, φ(x, t) also converges as t → +∞. This convergence is established in the following proposition.

Proposition 3. The filter φ(x, t) is a continuous and infinitely differential function over R 2 × R such that φ(x, 0) = 0 for all x ∈ R 2 . If T = +∞, then φ(x, t) converges uniformly toward φ(x) where φ(x) is the WDoG filter:

φ(x) = w c γ c G σc (x) -w s γ s G σs (x), (19)
with γ c and γ s defined in ( 15)-( 16). If T < +∞, the filter vanishes uniformly as t → +∞:

lim t→+∞ sup x∈R 2 | φ(x, t) |= 0. ( 20
)
Proof. The uniform convergence [START_REF] Marr | Vision[END_REF] results from the definition of φ(x, t) in [START_REF] Andelson | Pyramid methods in image processing[END_REF] since G σc (x) and G σs (x) are bounded and R c (t) and R s (t) converges to 0 according to Proposition 2 and Lemma 1. When T = +∞, R c (t) and R s (t) converge, respectively, to γ c and γ s as t → +∞. Hence, φ(x, t) converges uniformly toward φ(x).

Figure 5 depicts the two different cases of Proposition 3. A brief discussion about this proposition is that, while time T increases to infinity, the retina-inspired filter turns into a static WDoG. This is totally concurrent with the neuroscientific assumptions about the time limits of the visual system. Neuroscientists have proposed that the objects categorization of a single image which is propagated from the retina to the brain needs approximately 100 ms [START_REF] Liu | Timing, timing, timing: Fast decoding of object information intracranial field ptentials in human visual cortex[END_REF] before the next image is processed. Recent studies prove that a simple comprehension lasts approximately 13 ms [START_REF] Potter | Detecting meaning in RSVP at 13ms per picture[END_REF]. In any case, there exists a time t c when, even if the photoreceptors will continue capturing the same signal, all the necessary information which need to be processed has already been transmitted to the brain. From a theoretical point of view, the existence of t c is deduced from the uniform convergence established in Proposition 3. In fact, given ε > 0, the time t c = t c (ε) can be defined when the uniform convergence is achieved up to ε.

IV. WEIGHTED DOG ANALYSIS

This section aims to study the WDoG filter, i.e., to approximate its spatial response and its frequency response. Without any loss of generality, a WDoG is defined by:

ϕ(x) = aG σa (x) -bG σ b (x) (21)
where a, b > 0 and σ 2 b > σ 2 a . The retina-inspired filter φ(x, t) consists of a group of WDoG with the coefficients

a = w c R c (t) = a(t), b = w s R s (t) = b(t), (22)
that are time dependent, σ a = σ c and σ b = σ s . Since the WDoG is symmetric, we define the WDoG according to the radial coordinate r = x 2 :

ϕ(r) = aG σa (r) -bG σ b (r). (23)
Since ϕ(r) is symmetric around 0, we assume that r ∈ R (and not only to R + ) to ensure a better legibility of the results and to study more easily the spectrum of the WDoG. According to the couple (a, b), the WDoG has eigth shapes which are depicted in Fig. 7. The parameter b = 1 is fixed. According to the value of a ∈ {0.1, 0.3, 0.4, 0.7, 0.9, 1.1, 2, 4}, the WDoG is either a lowpass filter (cases L1, L2 or L3) or a bandpass filter (case BP) or a mixed lowpass/bandpass filter (case LB). The conventional DoG filter (a = b = 1) is a special case of BP. When a is changing, the bandwidth is also changing but the WDoG is always constrained to one of these eight shapes, corresponding to five behaviors. This section studies these behaviors in the space domain and in the frequency domain.

A. WDoG in Space Domain

Let us study the variations of ϕ(r). The first derivative of the WDoG, which is differentiable for all r ∈ R, is given by A short analysis of the roots of ϕ ′ (r) shows that two cases occur: i) if γ ≥ 1, there is only one root r = 0 , ii) if γ < 1, there are three roots r = 0, r 1 > 0 and -r 1 with

ϕ ′ (r) = r - a σ 2 a G σa (r) + b σ 2 b G σ b (r) . ( 24
) Let γ = γ(a, b) = bσ 4 a aσ 4 b . (25)
r 1 = σ a σ b 2 ln(γ) σ 2 b -σ 2 a . (26)
As an illustration, case i) corresponds to the first left curve in Fig. 7 and case ii) corresponds to all the other curves. It is then easy to determine the positive intervals and negative intervals of ϕ ′ (r) and, hence, to determine when ϕ(r) is increasing or decreasing.

Proposition 4. If γ ≥ 1, ϕ(r) is negative for all r ∈ R and its minimum is

ϕ 0 = min r∈R ϕ(r) = ϕ(0) = aσ 2 b -bσ 2 a 2πσ 2 c σ 2 s < 0. ( 27
)
Otherwise, if γ < 1, then ϕ(r) is negative and decreasing over [-∞, -r 1 ], increasing over [-r 1 , 0], decreasing over [0, r 1 ] and finally increasing and negative over [r 1 , +∞]. The global minima are -r 1 and r 1 .

Proof. The proof is straightforward by studying the sign of ϕ ′ (r) over the intervals given in the proposition.

The Full Width Half Maximum (FWHM) response of the symmetric WDoG is given by the interval [-r C , r C ] where r C satisfies:

ϕ(r C ) = max r∈R ϕ(r) 2 . ( 28
)
The FWHM is not relavant for case L1 (see Fig. 7) because the shape of the WDoG is significantly different from a peak. The FWHM is relevant for the other cases when the peak at r = 0 is sufficiently large, i.e., when ϕ(0) > -2ϕ(r 1 ). The exact calculation of FWHM is difficult. Hence, we approximate r C in (28) using the following method which is proposed in [START_REF] Birch | Approximate bandpass and frequency response models of the difference of gaussian filter[END_REF].

We first calculate the maximum response. Then, we compute a straight line from each maximum response to the first intercept with the r-axis (see Fig. 8 red solid line). We compute half of this line and we approximate its correspondent value. The resulting FWHM is illustrated in Fig. 8. A straightforward calculation shows that the first intercept of ϕ(r) with the raxis is

r 0 = σ a σ b 2 ln( aσ 2 b bσ 2 a ) σ 2 b -σ 2 a . ( 29
)
where bσ 2 a > aσ 2 b since ϕ(0) > 0. It follows that r C ≈ r 0 2 by using the triangle approximation.

B. WDoG in Frequency Domain

The Fourier transform of the WDoG in [START_REF] Livingstone | Anatomy and physiology of a color system in the primate visual cortex[END_REF] is

φ(ω) = a Ĝσa (ω) -b Ĝσ b (ω) (30)
where Ĝσ

(ω) = 1 2π exp - ω 2 σ 2 2 , (31)
and ω ∈ R denotes the spatial angular frequency associated to r. The WDoG bandwidth B = B(a, b) refers to the frequency range in which the spectrum φ(ω) is above a threshold value. The threshold value is defined relative to the maximum value and the points where the spectrum is half its maximum value, i.e., we need to find all the solutions ω of

φ(ω) = max ω∈R φ(ω) 2 . ( 32
)
For each case identified in Fig. 7, the derivation of a closed form expression of the bandwidth is very tricky. Hence, we propose some simple approximations. For cases L1, LB, BP, each part of the bandwidth is approximated by a triangle. A triangle is formed by taking the zero position, the maximum position and double the turning point as already illustrated in Fig. 8. For cases L2 and L3, a better approximation is obtained by considering that the WDoG is almost equivalent to a Gaussian function.

Let us determine the zero position and the maximum position of the WDoG. The solutions of φ(ω) = 0 are the two opposite roots ω 0 and -ω 0 with

ω 0 = 2 σ 2 b -σ 2 a ln b a ( 33
)
when b > a. Otherwise, when b ≤ a, the WDoG is always positive with the single root ω = 0 in the special case a = b. Since the WDoG is differentiable, the extrema are the solutions of φ′ (ω) = 0 where φ′ (ω) is the first derivative of φ(ω). A short calculation shows that

φ′ (ω) = ω 2π -aσ 2 a exp - ω 2 σ 2 a 2 + bσ 2 b exp - ω 2 σ 2 b 2 .
A first extrema is ω = 0. The other extrema are the solutions of exp

ω 2 (σ 2 b -σ 2 a ) 2 = bσ 2 b aσ 2 a = ̺(a, b). (34)
If ̺(a, b) ≤ 1, there is no other extrema. Otherwise, there are two opposite extrema ω 1 > 0 and -ω 1 where

ω 1 = 2 σ 2 b -σ 2 a ln(̺(a, b)). (35)
From [START_REF] Liu | Timing, timing, timing: Fast decoding of object information intracranial field ptentials in human visual cortex[END_REF], it is clear that φ′ (ω) > 0 for 0 < ω < ω 1 and φ′ (ω) > 0 for ω 1 < ω. Since lim ω→±∞ φ(ω) = 0, the frequency ω 1 , if it exists, is a global maximum. We can also show that -ω 1 is a global maximum. We obtain the following proposition.

Proposition 5. If ̺(a, b) > 1, then the maximum is given by

max ω∈R φ(ω) = φ1 = φ(ω 1 ) = φ(-ω 1 ) = a(σ 2 b -σ 2 a ) 2π 1 ̺(a, b) σ 2 a σ 2 b -σ 2 a ( 36
)
with ω 1 given in [START_REF] Potter | Detecting meaning in RSVP at 13ms per picture[END_REF]. Otherwise, if ̺(a, b) ≤ 1, then

max ω∈R φ(ω) = φ0 = φ(0) = a -b 2π > 0.
Proof. If ̺(a, b) ≤ 1, there is only one extrema ω = 0. Since φ′ (ω) has the sign of -ω, φ is increasing when ω < 0 and decreasing when ω > 0. Hence, 0 is a positive global maximum. If ̺(a, b) > 1, it has been already shown that the maximum are located at ω 1 and -ω 1 .

The maximum value in [START_REF] Birch | Approximate bandpass and frequency response models of the difference of gaussian filter[END_REF] is obtained by inserting ω 1 , given in [START_REF] Potter | Detecting meaning in RSVP at 13ms per picture[END_REF], in [START_REF] Masland | Cell populations of the retina: The proctor lecture[END_REF].

According to the couple (a, b), the WDoG has three possible behaviors: lowpass, bandpass or lowpass/bandpass. Hence, the total bandwidth B = B(a, b), including negative and positive frequencies, can have one of the three following forms:

• Lowpass: B = [-ω H , ω H ] with ω H > 0, • Bandpass: B = [-ω H , -ω L ] ∪ [ω L , ω H ] with 0 < ω L < ω H , • Lowpass/bandpass: B = [-ω C , ω C ] ∪ [-ω H , -ω L ] ∪ [ω L , ω H ] with 0 < ω C < ω L < ω H .
The following proposition gives the bandwidth B(a, b) with respect to the couple (a, b).

Proposition 6. According to the value of (a, b), the WDoG φ(ω) satisfies one of the following cases:

1) If ̺ > 1 and | φ0 | ≥ φ1 a) Case L1: if | φ0 | ≥ 2 φ1 , then φ(ω) is lowpass with ω H ≃ ω 0 2 , b) Case LB: if | φ0 | < 2 φ1 , then φ(ω) is low- pass/bandpass with ω C ≃ ω 0 2 , ω L ≃ ω 0 + ω 1 2 , ω H ≃ ω 0 + 3ω 1 2 , 2) If ̺ > 1 and | φ0 | < φ1 a) Case BP: if | φ0 | ≤ φ1 2 , then φ(ω) is bandpass with ω L ≃ ω 0 + ω 1 2 , ω H ≃ ω 0 + 3ω 1 2 , b) Case L2: if | φ0 | > φ1 2
, then φ(ω) is lowpass with

ω H ≃ ω 1 + 2 ln(2) σ a , 3) If ̺ ≤ 1, which corresponds to Case L3, then φ(ω) is lowpass with ω H ≃ 2 ln(2) σ a .
Proof. The behavior of the filter depends on the maximum values φ0 or φ1 , the maximum position and the zero position.

In case of L1, LB and BP, each part of the bandwidth is approximated by a triangle. A short calculation gives the bounds of the triangle. In case of L2, we use two Gaussians to approximate the central part of the spectrum. The bandwidth of a Gaussian function with zero mean and standard deviation σ is given by

B σ = -σ 2 ln(2), σ 2 ln(2) .
In case of L3, we use a single Gaussian approximation.

The retina-inspired filter is filtering the input image f (x) by using a WDoG varying in time. By controling the trajectory (a(t), b(t)), this filter is able to explore the frequency spectrum of the input image.

ω t ω H -ω H ω L -ω L ω H -ω H ω L -ω L ω H -ω H ω L -ω L ω H -ω H ω L -ω L

V. NUMERICAL RESULTS

This section aims to compute the retina-inspired decomposition tuned with respect to bio-plausible parameters given in [START_REF] Wohrer | Virtual retina: A biological retina model and simulator, with constrast gain control[END_REF] and described in Fig. 5. One could adjust these parameters and achieve non bio-plausible WDoG filters which are maybe more adaptable for different research topics. The tuning of these parameters is out of the scope of this work but we are going to test some non bio-plausible parameters and discuss the results. In addition, concerning the two different case when T = +∞ and T < +∞, one could notice according to Fig. 6 that the second one is separated into three groups for t ∈ [0, T ], t ∈ (T, 2T ] and t > 2T . Concerning the first group, the retinainspired filter would evolve exactly in the case was as in case T = +∞. The second group is the inverse of the first group and the third group corresponds to the zero value because of the zero convergence. According to Fig. 5 it is obvious that at least for bio-plausible parameters, the two first groups have exactly the same spectrum. As a result, we concentrate the analysis on the first group which is identical to T = +∞.

We are able now to compute the bandwidth B = B(a, b) of the filter as a function of time using Proposition 6 which is illustrated in Fig. 9 (a). The approximation consists of 3 different case which appear progressively in time: for 1 ≤ t ≤ 5 ms the bandwidth is approximated according to L3, for 5 < t ≤ 23 ms it follows L2 and finally, for t > 23 ms it belongs to BP. The exact bandwidth is computed by solving the equation (32) using the Matlab T M fsolve function and it is depicted in Fig. 9 (b). Concerning the exact solution one should notice that the bandwidth evolves in time in a very similar way depicted that the filter behaves as a lowpass for 1 ≤ t ≤ 23 ms while it becomes bandpass for t > 23 ms. Comparing these two plots, we confirm the accuracy of Proposition 6 is not perfect because of the triangle approximation. This approximation of the DoG filter has already been used in previous works getting similar accuracy results [START_REF] Birch | Approximate bandpass and frequency response models of the difference of gaussian filter[END_REF]. It would be also important to be mentioned that the approximation is not continuous but it has been proposed for each case separately. This is the reason why some discontinuities appear. The second row of Fig. 9 comes to remind the reader that if one chooses different set of parameters, not only the behavior of the filter but also the evolution of the bandwidth with respect to time will change.

In Fig. 10, we represent the filtering results while the retinainspired filter is applied to an image for different parameters of w c and w s . We used a gray-scale image of size n = 512 × 512 pixels. We have decided to illustrate only 5 decomposition layers for each experiment. The evolution of the retina-inspired filter is according to the bandwidth of Fig. 9 (b). The first column of Fig. 10 corresponds to the bio-plausible parameters where w c = w s = 1. The middle and the left columns correspond to non bio-plausible parameters but very close to the ones of the left column. It is interesting that while w c decreases there is a stronger categorization of the L2, L3, LB and BP cases. Another important remark is related to the scale of the top pictures which correspond to the first decomposition layer. Apparently, this layer seems to be identical to the original. However, its scale is very low (i.e. 10 -13 ) which means that in the presence of noise or after quantization all this information will be completely lost.

VI. CONCLUSION

This paper has introduced a non-separable spatiotemporal retina-inspired filter based on a realistic model of the OPL. This filter is a groundbreaking analysis of neuroscience for image processing. It has been established that this filter is a group of time-varying WDoGs when it is applied to a still image flashed for a given time. Hence, it enables the extraction of different kind of data while time increases. As underlined in the introduction, the retina-inspired filter is a great improvement of other bio-inspired filters which are simpler and not much as accurate neuroscientific approximation of the retina. Hence, this is certainly of a great interest for the image processing field. There are numerous applications for our retinal-inspired filter, the most important of which is image compression. We believe that the retina-inspired transform is the first step to build novel retina-inspired codecs. It seems that even if image and video codecs are still two of the most attractive research topics, they have reached their limits: even if the bitrate efficiently decreases, the computational cost dramatically increases [START_REF] Grois | Performance comparison of h.265/mpeg-hevc, vp9, and h.264/mpeg-avc encoders[END_REF]. The HVS has low power consumption while at the same time its visual perception is very high. Some models have been already proposed to mimic the retinal coding like the Rank Order Coding (ROC) [START_REF] Masmoudi | Frames for exact inversion of the rank order coder[END_REF], [START_REF] Thorpe | Spike arrivel times: A highly efficient coding scheme for neural network[END_REF], [START_REF] Thorpe | Rank Order Coding: A new coding scheme for rapid processing in neural network[END_REF]. However, the filtering bank used by ROC was very simple. As a result, it would be interesting to extend these works by using our retinainspired model. In addition, it would be also relevant to study a full bio-plausible neural coding architecture like [START_REF] Masmoudi | Streaming an image through the eye: The retina seen as a dithered scalable image coder[END_REF], [START_REF] Lazar | Video time encoding machines[END_REF] based on our model.

APPENDIX

This appendix proves Lemma 1. It is shown that both J c (t) and J s (t) can be defined in a closed form as polynomial functions which are attenuated by exponential ones. These functions will be essential to calculate R c (t) and R s (t). The calculation of J c (t) and J s (t) are based on the following lemma whose proof is straightforward. where P n (t) is a polynomial function in t of order n and α c and γ c are two reals.

B. Closed-form of J s (t)

The method used to calculate J c (t) can be applied to J s (t). This leads to:

J s (t) =

Fig. 2 :

 2 Fig.2: The retinal layers according to[START_REF] Wohrer | Virtual retina: A biological retina model and simulator, with constrast gain control[END_REF]. This figure shows the connectivity and hierarchical structure of the retinal cells. It is important to note that cells of the same kind lie within a common layer.

Fig. 3 :

 3 Fig. 3: Propagation of the electrical signal to the receptive field of a bipolar cell according to [19]. The left figure is part of the retinal structure which corresponds to the OPL layer. The right figure is focused on a single bipolar cell and how the RF of this cell works.

Fig. 5 :

 5 Fig. 5: (Left to right) The retina-inspired filter φ(r, t) as a function of r ∈ R and t ∈ R. The top view of φ(r, t). The retina-inspired filter spectrum φ(ω, t). The top-view of | φ(ω, t)|. (a) The top line corresponds to the case T = +∞. (b) The bottom line stands for the case T < +∞ (Parameters: T = 150 ms, τ C = 20 ms, τ S = 4 ms, τ G = 5 ms, n = 5, w S = 1, w C = 1, σ c = 0.5 and σ s = 3σ c ).

Fig. 6 :

 6 Fig.6: Temporal filters R c (t) and R s (t). The convergence of the filters depends on the value of time T . Subplot (a) corresponds to the case T = +∞, while subplot (b) corresponds to the case T < +∞.

Fig. 7 :

 7 Fig. 7: Eight typical shapes of the WDoG ϕ(r) (first row) and the WDoG spectrum modulus | φ(ω)| (second row) w.r.t. some values of the couple (a, b). The dotted line represents the half of the maximum value of the spectrum modulus. The rectangles in dashed line represent the approximated bandwidth of the spectrum. There are five specific cases: 1) lowpass L1, 2) lowpass/bandpass LB, 3) bandpass BP, 4) lowpass L2 and 5) lowpass L3.

Fig. 8 :

 8 Fig. 8: Approximate FWHM interval of a WDoG based on a triangle approximation.

Fig. 9 :

 9 Fig. 9: For bio-plausible parameters described in Fig. 5 we compute in (a) the approximation of the evolution of the bandwidth with respect to time which is computed according to Proposition 6 and in (b) the exact evolution of the bandwidth with respect to time which is numerically computed. (c). The exact solution of the bandwidth which behaves as lowpass for longer time (τ C = 30 ms, τ S = 20 ms, τ G = 15 ms, n = 5). (b) The exact solution of the bandwidth which behaves as bandpass (τ C = 40 ms, τ S = 0.5 ms, τ G = 9 ms).

Fig. 10 :

 10 Fig. 10: Decomposition of an image using the retina-inspired non-separable spatiotemporal filter. Left: w c = w s = 1. Middle: w c = 0.9 and w s = 1. Right: w c = 0.7 and w s = 1.

Lemma 2 .

 2 Let ω a real value, t ≥ 0 and n a positive integer. Using an integration by parts, we obtain the following equality:t 0 u n exp (-ωu) du = P n (t) exp (-ωt) + c, where P n (t) = n k=0 -n! (nk)! ω k+1 t n-kis a polynomial function in t of order n whose coefficients depend on ω and c is a constant value.A. Closed-form of J c (t)Assume that 0 ≤ t ≤ T . It follows that:J c (t) = t u=0 V (u)du, which yields J c (t) = t u=0 E τG,n t * (δ 0w C E τC ) (u)du = t u=0 E τG,n (u)duw C t u=0 E τG,n t * E τC (u)du.The definition of the gamma and exponential filters yields:c (ml)!b k+1 a l+1 τ n+1 G c b k+1 a m+1 τ n+1 G τ C -n!w c b n+1 τ n+1 G + n! where m = nk. Finally, J c (t) = P n (t) exp -t τ G + α c exp -t τ C + γ c

E

  τG,n * (δ 0w C E τC ) * E τS (u)du = t u=0 E τG,n * E τS (u)du -w c t u=0 E τG,n * E τC * E τS (u)du. m-l exp (-at) g k+1 a l+1 (ml)! p-r exp (-at) (pr)!b k+1 g l+1 a r+1 + n k=0 n! b k+1 g l+1 a p+1 -n k=0 n!τ S b k+1 g m+1

  where Q n (t) is a polynomial function in t of order n and α s , β s and γ s are some reals. This ends the proof.

	+	n!τ S b n+1 φ	1 -exp	-t τ S
	-	n!τ C b n+1 φ	1 -exp	-t τ C	,
	with the variables
	g =	τ S -τ G τ G τ S	, φ =	τ S -τ C τ C τ S	,
	p = m -l and m = n -k. It follows that:
	J s (t) = Q n (t) exp	-t τ G	+ α s exp	-t τ S
		+β s exp			-t τ C	+ γ s
						-t
						τ S

1 -exp

Thanks to 4G-TECHNOLOGY and ANRT.


Emmanuel Maggiori
email: emmanuel.maggiori@inria.fr

Yuliya Tarabalka

Guillaume Charpiat

Pierre Alliez

FULLY CONVOLUTIONAL NEURAL NETWORKS FOR REMOTE SENSING IMAGE CLASSIFICATION

Keywords: Remote sensing images, classification, deep learning, convolutional neural networks

We propose a convolutional neural network (CNN) model for remote sensing image classification. Using CNNs provides us with a means of learning contextual features for large-scale image labeling. Our network consists of four stacked convolutional layers that downsample the image and extract relevant features. On top of these, a deconvolutional layer upsamples the data back to the initial resolution, producing a final dense image labeling. Contrary to previous frameworks, our network contains only convolution and deconvolution operations. Experiments on aerial images show that our network produces more accurate classifications in lower computational time.

INTRODUCTION

Image classification is a recurrent problem in remote sensing, aimed at assigning a label to every pixel of an image. Contrary to the image categorization problem (i.e., assigning an entire image to a category such as 'residential' or 'agricultural' area), we conduct a dense pixel-wise labeling. A challenging aspect of the dense classification problem is the design of algorithms that can deal with the large scale of remote sensing data. Besides the execution time constraints, obtaining an accurate classification is substantially more difficult when dealing with large amounts of heterogeneous data.

Most state-of-the-art dense classification approaches label every pixel individually by taking into account its spectrum and possibly some constraints with respect to its close neighbors [START_REF] Tarabalka | Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques[END_REF]. In this work we are dealing with large-scale satellite images that do not have a large spectral resolution, making it difficult to distinguish object classes just by their spectrum. We must thus infer the class of a pixel from its context and from the shape of the surrounding objects.

Convolutional neural networks (CNNs) learn contextual features at different scales. While initially devised for image categorization [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF], we show that they are also effective at All Pléiades images are c CNES (2012 and 2013), distribution Airbus DS / SpotImage. The authors would like to thank the CNES for initializing and funding the study and providing Pléiades data.

dealing with the dense classification of satellite imagery.

In remote sensing, CNNs have been used to classify the pixels of hyperspectral images. Instead of convolving in the spatial domain, convolutions are performed in the 1D domain of the spectrum of each pixel [START_REF] Li | Classification of hyperspectral image based on deep belief networks[END_REF], or in the 1D flattened spectrum vector of a group of adjacent pixels [START_REF] Chen | Spectralspatial classification of hyperspectral data based on deep belief network[END_REF]. We convolve in the 2D spatial domain instead, in order to automatically infer the contextual spatial features required to classify satellite imagery. Penatti et al. [START_REF] Otávio | Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?[END_REF] show that the CNNs used to recognize everyday objects generalize well to categorize remote sensing scenes. One of the biggest challenges however is to turn the categorization networks, that generate a single category for the whole scene, into dense labeling networks. In remote sensing, Mnih [START_REF] Mnih | Machine learning for aerial image labeling[END_REF] performs dense labeling through CNNs. The typical single-output final layer was replaced by a fullyconnected layer that outputs entire classification patches.

We discuss Mnih's approach next, and point out some limitations that hamper its accuracy and efficiency (Sec. 2-3). We then propose a new CNN architecture that carries out dense labeling by relying solely upon convolutional layers (Sec. 4). We show in various experiments (Sec. 5) that it outperforms the previous approaches and offers a solid framework for remote sensing image classification.

PATCH-BASED NETWORK

State-of-the-art CNNs for image categorization tend to follow a similar pattern: a series of convolution and subsampling operations to extract features of the image, followed by a fully connected layer to carry out the final labeling into categories. Typical CNNs produce as many outputs as number of categories, or a single output for binary labeling. For details on image categorization with CNNs we refer the reader to [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF].

In our problem we must generate as output a dense classification, i.e., not just one categorization for the entire image, but a full pixel-wise labeling into the different categories.

To this end, Mnih proposed a patch-based convolutional neural network [START_REF] Mnih | Machine learning for aerial image labeling[END_REF]. Given the sheer size of remote sensing images, training and inference are performed patch-wise. The network takes as input a patch of an aerial image, and generates as output a classified patch. The output patch is smaller, and centered in the input patch, to take into account the surrounding context for more accurate predictions. The way to Fig. 1: Convolutional neural network architectures (e.g., "64@14 × 14" means 64 feature maps of size 14 × 14).

create dense predictions is to increase the number of outputs of the fully connected classification layer, in order to match the size of the target patch. Fig. 1(a) illustrates such patchbased architecture. The network takes 64 × 64 patches (on color pansharpened images of 1m 2 spatial resolution) and predicts 16 × 16 centered patches of the same resolution. Three convolutional layers learn 64, 112 and 80 convolution kernels, of 12×12, 4×4 and 3×3 spatial dimensions, respectively. The convolution kernels are three-dimensional, i.e., the two spatial dimensions plus a third dimension that goes through all the feature maps convolved. The first convolution is not applied to every pixel of the input, but at every other forth pixel, feature referred to as a stride. Without strides (or other sort of subsampling) the number of parameters becomes too large for a network to learn effectively. After the three convolutional layers, a fully connected layer transforms the features into a classification map of 256 elements, matching the required 16 × 16 output patch. Each convolutional layers is followed by a (ReLU) activation function [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] to add non-linearities and increase the space of functions that the network can learn. The final classification probabilities are computed by applying a sigmoid function to the output of the last layer. The loss function used for training is the cross-entropy on the sigmoid values [START_REF] Christopher | Neural networks for pattern recognition[END_REF].

Training is carried out by an stochastic gradient descent applied to random patches of a dataset. In every iteration, the patches are grouped into mini-batches to estimate the gradient of the loss function with respect to the network's parameters, and the parameters are updated accordingly.

LIMITATIONS OF THE PATCH-BASED SCHEME

We now point out some limitations of the patch-based approach discussed above, that motivate the design of an improved architecture. We first discuss the role of the fully connected layer. The size of the feature maps of its previous layer is 9 × 9 and have 1/4 of the resolution of the input, given the stride in the first convolution. The fully connected layer outputs a 16 × 16 map instead. This means that the fully connected layer does not only carry out a classification, but it also learns how to upsample the feature maps from the previous layer to the initial image resolution. In addition, the fullyconnected layer allows every output to have different weights with respect to the previous feature map. For example, the activation of an output pixel at the top-left corner of the patch might not be the same as the one at the bottom-right. This makes it possible for the network to learn priors on the position inside a patch, in order to carry out the final classification. In our context, the partition of an image into patches is arbitrary, hence the "in-patch location" prior is not necessary. Otherwise, e.g., two patches that are similar but rotated by 90 degrees might yield different classification maps.

When training the patch-based network of Fig. 1a we expect that, after processing many training cases, the fully connected layer will learn a location-invariant function to classify and upsample the features of the previous layer. The experiments carried out on patch-based networks presented by Mnih [START_REF] Mnih | Machine learning for aerial image labeling[END_REF] show the existence of discontinuities at the border of the patches in the output probability maps (see Fig. 4). This implies that the networks do not succeed in learning to classify pixels independently of their location inside the patch.

FULLY CONVOLUTIONAL APPROACH

We propose a fully convolutional neural network architecture (FCN) to produce dense predictions. This architecture explicitly restricts the outputs of the patches to be locationindependent, which means that they should be the result of a series of convolutions only.

A classification network can be "convolutionalized" [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] as follows. First, we rewrite the fully connected layer that carries out the classification as a convolutional layer. If we choose a convolution kernel whose dimensions coincide with the previous layer, the connections are equivalent to a fully connected layer. The difference now is that if we enlarge the input image, the output size is also increased, but the number of parameters remains constant. This can be seen as convolving the whole original network around a larger image to evaluate the output at different locations.

To increase the resolution of the output map, we then add a "deconvolutional" layer [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] that learns filters to upsample the resolution. A deconvolutional layer takes a single input and multiplies it by a learned filter, to produce an output patch. If these patches overlap in the output, they are simply added to create the final result. This can be seen as a convolutional layer with backward and forward passes inverted.

As compared to a patch-based approach, our fully convolutional network exhibits the following advantages:

• Elimination of discontinuities due to patch borders;

• Improved accuracy due to a simplified learning process, with a smaller number of parameters; • Lower execution time at inference, given that convolutions can enjoy the benefits of GPU processing.

We convolutionalize the patch-based network depicted by Fig. 1. We choose an existing framework to benefit from a mature architecture and to carry out a rigorous comparison.

Fig. 1b depicts such conversion into a FCN. In the patchbased network, every output inside the output patch is located at a different position with respect to its so-called receptive field (the portion of the input to which it is connected). This behavior is hard to justify, so we will here pretend that the output patch is of size 1 × 1 instead of 16 × 16, thus just concentrating on a single centered output. We then rewrite the fully connected layer as a convolutional layer with one feature map and with the spatial dimensions of the previous layer (9 × 9). Finally, we add a deconvolutional layer that upsamples its input by a factor of 4, in order to recover the input resolution. Notice that the tasks of classification and upsampling are now separated.

This new network takes as input images of different sizes, with the output size varying accordingly. During the training stage, we emulate the learning as performed by the patchbased networks by taking an input of size 80 × 80 in order to produce maps of size 16 × 16 as before (see Fig. 1c). The input patch is larger than the one of patch-based networks, not because we are dealing with more context, but because every output is now centered in its context. At inference time we take inputs of arbitrary sizes to construct classification maps. For arbitrary size of input the FCN structure is always similar, with the same number of parameters.

Fig. 1c depicts the role of deconvolution: The output value of each neuron from the previous layer is multiplied by a learned filter, which is "pasted" in the output with a stride of 4. Overlapping occurs due to the size 8 × 8 of the filter. The overlapping areas are added (gray) while the excess areas beyond border are excluded (white).

EXPERIMENTAL RESULTS

The CNNs were implemented using the Caffe deep learning framework [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF]. In a first experiment we apply our approach to the Massachusetts Buildings Dataset [START_REF] Mnih | Machine learning for aerial image labeling[END_REF]. The dataset consists of color images over Boston with 1m 2 spatial resolution, covering an area of 340km 2 for training and 22.5km 2 for testing. The images are labeled into two-classes: building and non-building. A portion of an image with its corresponding reference is depicted in Figs. 2a-b.

We train the patch-based and fully convolutional networks (Figs. 1a and1c respectively) for 30k stochastic gradient descent iterations on randomly sampled patches, with mini-batches of size 64, a learning rate of 0.0001, momentum 0.9 and a weight regularization of 0.0002. The parameters and rationale for selecting them are detailed by Mnih [START_REF] Mnih | Machine learning for aerial image labeling[END_REF].

To evaluate the accuracy of the classification we used two different measures: pixel-wise accuracy (proportion of correctly classified pixels, obtained though binary classification of the output probabilities with threshold 0.5) and the area under the receiver operating characteristics (ROC) curve [START_REF] Ferri | A coherent interpretation of AUC as a measure of aggregated classification performance[END_REF]. The latter measures the overall quality of the fuzzy maps, being 1 the area value corresponding to an ideal classifier.

Fig. 3a plots the evolution of area under ROC curve and pixel-wise accuracy, through the iterations. The FCN consistently outperforms the patch-based network. Fig. 3b shows ROC curves for the final networks after training, the FCN exhibiting a better relation between true and false positive rates (areas under ROC curves are 0.9922 for FCN and 0.9899 for the patch-based network). Fig. 2c-d depicts visual fragments.

To further evaluate the benefits of neural networks over other previous learning approaches we trained a support vector machine (SVM) with Gaussian kernel on 1k randomly selected pixels of each class. Such SVM-based approach is common for remote sensing image classification [START_REF] Tarabalka | Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques[END_REF].

As shown by Fig. 2e, such pixel-wise SVM classification often confuses roads with buildings as their colors are similar, while neural networks better infer and separate the classes by taking into account the geometry of the context. The accuracy on the Boston test dataset is 0.6229 and its area under ROC curve is 0.5193 (lower than with CNNs, as shown in Fig. 3).

In terms of efficiency the FCN also outperforms the patchbased CNN. Instead of carrying out the prediction in a small patch basis, the input of the FCN is simply increased to output larger predictions, better benefiting from the GPU parallelization of convolutions. The execution time to classify the whole Boston 22.5km 2 dataset (run on an Intel I7 CPU @ 2.7Ghz with a Quadro K3100M GPU) is 82.21s with the patch-based In a second experiment we visually demonstrate the effectiveness of FCNs for the classification of a Pleiades image covering the area of Forez, France, at a 0.5m 2 spatial resolution. We use a pansharpened color version of the image, which we train for 30k iterations on patches extracted from a subset of the surface that covers 24.75km 2 . The classes and the parameters used for training are the same as for the Boston network. To preserve the amount of spatial context in this higher resolution image, we first subsample the image and then linearly upsample the output patches. The reference data used for training are extracted from OpenStreetMap project. Fig. 5 shows a test subset and its corresponding classification map. The time to infer a tile of 2.25km 2 is 10.6 seconds for a patch-based CNN and 1.8 for the FCN.

As shown in the amplified fragments from both images (Fig. 4), the border discontinuities present in the patch-based scheme are absent in our fully convolutional setting.

CONCLUDING REMARKS

This work addresses the problem of remote sensing image classification with convolutional neural networks. CNNs are mostly used to categorize images, hence new architectures must be designed for dense pixel-wise classification. To this end, we proposed a fully convolutional neural network. By imposing the restriction that all layers must be convolutional or deconvolutional, the learning process is enhanced and the execution time reduced. Our experimental results show that such networks outperform previous approaches both in accuracy and in the computational time required for inference.

  Fully convolutional (16 × 16 output)

Fig. 2 :

 2 Fig. 2: Experimental results on a fragment of the Boston dataset.

Fig. 3 :Fig. 4 :

 34 Fig. 3: Evaluation on the Boston test set.

Fig. 5 :

 5 Fig. 5: Classification of a Pleiades image, with an FCN.


I Sadeh

I Oya

J Schwarz

E Pietriga

Prototyping the graphical user interface for the operator of the Cherenkov Telescope Array

Keywords: The Cherenkov Telescope Array, graphical user interface, Web technology

The Cherenkov Telescope Array (CTA) is a planned gamma-ray observatory. CTA will incorporate about 100 imaging atmospheric Cherenkov telescopes (IACTs) at a Southern site, and about 20 in the North. Previous IACT experiments have used up to five telescopes. Subsequently, the design of a graphical user interface (GUI) for the operator of CTA involves new challenges. We present a GUI prototype, the concept for which is being developed in collaboration with experts from the field of Human-Computer Interaction (HCI). The prototype is based on Web technology; it incorporates a Python web server, Web Sockets and graphics generated with the d3.js Javascript library.

INTRODUCTION

The Cherenkov Telescope Array (CTA) [START_REF] Actis | Design concepts for the Cherenkov Telescope Array CTA: an advanced facility for groundbased high-energy gamma-ray astronomy[END_REF][START_REF] Fuessling | Status of the array control and data acquisition system for the cherenkov telescope array[END_REF][START_REF] Oya | The software architecture to control the cherenkov telescope array[END_REF] is a planned observatory for very high-energy (> 100 GeV) gammarays. Gamma-rays induce particle cascades in the atmosphere. These are accompanied by Cherenkov radiation, which is emitted by the charged particles in the cascade. The Cherenkov light may be detected by imaging atmospheric Cherenkov telescopes (IACTs). [START_REF] Hillas | Evolution of ground-based gamma-ray astronomy from the early days to the cherenkov telescope arrays[END_REF] Using multiple telescopes in concert, Cherenkov showers can stereoscopically be sampled, allowing to reconstruct the properties of the primary gamma-ray.

CTA will include three different telescope types, sensitive to different gamma-ray energy ranges. The telescopes will be deployed in two sites, Northern and Southern, which will respectively include about 100 and about 20 telescopes. Currently running IACT experiments such as H.E.S.S., [START_REF] Aharonian | Observations of the Crab nebula with H[END_REF] MAGIC [START_REF] Albert | VHE Gamma-Ray Observation of the Crab Nebula and Pulsar with MAGIC[END_REF] and VERITAS, [START_REF] Holder | The first VERITAS telescope[END_REF] are restricted to up to five telescopes. Compared to these instruments, the large number of CTA telescopes will improve the sensitivity and the energy coverage of gamma-ray measurements by at least an order of magnitude.

The large number of telescopes has implications for the development of a graphical user interface (GUI) for the operator of a CTA site. The complexity of the system presents new and interesting challenges, requiring innovative design. In the following, we detail the development process of the operator GUI. We discuss the requirements we have derived for the GUI, and describe our prototype implementation.

DEVELOPMENT PROCESS AND REQUIREMENTS FOR THE GUI

In order to develop an effective user interface, we have drawn upon the lessons learnt from the Atacama Large Millimeter/submillimeter Array (ALMA). [START_REF] Wootten | The Atacama Large Millimeter/Submillimeter Array[END_REF] ALMA is an astronomical interferometer of radio telescopes in the Atacama desert of northern Chile. It will be comprised of up to 66 radio antennas, and so will be comparable to CTA in complexity.

During the early stages of ALMA, conventional user interfaces were developed. Early experience operating the ALMA array with only a few antennas, indicated that the latter were not adequate. The initial interfaces required many unnecessary interactions to access relevant information. This resulted in extraneous cognitive

The e-mail of I.S., iftach.sadeh@desy.de . load, and was not efficient for quick diagnosis of system problems. The implementation was therefore improved, taking into account advances in the field of Human-Computer Interaction (HCI). [START_REF] Pietriga | Interaction design challenges and solutions for ALMA operations monitoring and control[END_REF] For creating the GUI for the operator of CTA, we are following the design process used for ALMA. The GUI is being developed by involving experienced telescope operators and astroparticle physicists on the one hand, and experts from the field of HCI on the other. In contrast to ALMA, we have been able to adopt this model at the start of the design process for CTA. We will therefore be able to apply HCI input to all aspects of the operator interface. The development process includes practical face-to-face meetings, in tangent with brainstorming workshops with representatives of the relevant stakeholders. To date, we have held two such participatory design workshops. The outcome of the meetings is twofold. For one, the workshops helped us to refine the scope of the operator GUI, i.e., answer the question, "what should the GUI enable users to do?".

Additionally, we have defined a preliminary set of panels for the GUI. That is, we have began to address the question, "how should the GUI be designed?".

With regards to the first question, we first note that on-site operations related to observing with CTA will normally be automated. Consequently, the GUI for the operator of a CTA site will nominally be used to perform the following tasks:

1. initiate and end observations; 2. override the automated scheduled operations in order to perform a specific task, or for safety reasons, which might require manual control over a given telescope or group of telescopes; 3. monitor the state of the array during data acquisition, which includes monitoring of low-level hardware components, of software processes, and of the output of a near real-time data analysis; 4. identify and diagnose problems with specific sub-systems or processes, in order to solve minor problems or notify technical experts, as needed.

With regards to the second question, we have defined the following categories, which will be represented by different GUI panels (some of which may overlap):

1. Telescope: monitor the status of telescopes and their respective sub-systems; manually control single telescopes and sub-arrays, etc.

2. Process: monitor and modify predefined operation sequences, such as starting up the array at the beginning of the night, and performing a scheduled observing task.

3. Science: monitor the output of a physics analysis on the level of a single telescope or of a sub-array, including trigger rates, event-reconstruction metrics, science summary reports, etc. 4. Infrastructure: monitor the status of auxiliary systems, such as the power grid, as well as other indicators, such as alarms and data transfer rates.

5.

Environment: monitor environmental systems, such as weather monitors, all-sky cameras, etc. 6. Miscellaneous: provide access to terminals, shift logs, expert call sheets, etc. With these guidelines in mind, we have identified the following set of initial requirements for the GUI:

1. the GUI will be able to convey information for various levels of telescope multiplicity; on the level of the entire array (∼ 100 telescopes); the level of sub arrays (between 1 and 32 groups); or the level of a single telescope and the associated sub-systems. Multi-telescope views for a given sub-system will also be available, which will take into account differences between telescope types. The various levels of information will be integrated using semantic zooming, as defined below; 2. the GUI will need to be highly responsive, where the different panels of the GUI will have the option to be synchronized; 3. the GUI will integrate different interfaces to CTA software (databases, live feeds etc.) with various latencies; 4. the different components of the GUI will be modular, allowing for quick updates and for collaborative development.

In the next section, we illustrate the requirements for the operator GUI using a prototype design. At this stage of development, the prototype by no means addresses the complete scope of the GUI. Instead, it acts as a proof of concept for those features which were identified as most important.

PROTOTYPING ACTIVITIES

Our prototype is based on Web technologies. A Web-based framework has the advantage of being lightweight and modular. It also naturally allows for remote access, which will enable the GUI to be used for remote monitoring, and for remote operation. These will be useful both for physicists and for technical experts, who the operator of the array will need to consult.

The back-end of the prototype is a Python server called Pyramid. * A Python framework has the advantage of being very versatile, incorporating the ability to use many off-the-shelf libraries for computation, communication, multi-threading and multi-processing. The front-end is a web browser. The design is responsive, i.e., adjustable according to the width of the display. It is implemented using Polymer, † a Web Component application programming interface (API), developed by Google, based on the Material Design ‡ concept. Data are displayed using an open-source Javascript library, called d3.js. § The latter is a data-driven framework, with integrated mechanisms for displaying, updating and animating vector graphics. Communication between the back-end and the front-end of the GUI is performed using Web Sockets. These allow asynchronous communication, facilitating quick and robust GUI behaviour. For plotting, we also use dc.js, ¶ a library based on d3.js and on

Crossfilter. The latter is a Javascript library, which facilitates exploring large multivariate datasets. Zoom factor 1: Figure 1(a) shows a global view of all telescopes, emphasizing physical positions, as well as a general health metric. In this example, health corresponds to the combined health of all sub-systems of the telescope. For illustration, one of these sub-system, the camera, reports reduced health as the percentage of dead pixels. Zoom factor 7: Given this zoom level, Fig. 1(b) shows how the display changes for a given element which is hovered over. The hover action triggers a transition, where a circle representing a single telescope is replaced by a more detailed view of the associated sub-systems. The context from the previous zoom level is preserved during the transition. The health metric, which was previously represented by the full circle, has morphed into the red circle surrounding the four wedges. Each of the wedges represents one of the sub-systems of a telescope: the data-acquisition system (DAQ); the camera; the mirror system; and the mount (the structure of the telescope).

Each of the latter now also reports its individual health metric. These are represented numerically, by colour and by the percentage of filled area within the corresponding wedge. components, which is exposed for each of the four sub-systems. As before, the context from the previous zoom level is preserved; the health metric of each of the sub-systems is now represented by a thin outer arc. The inner wedges now each have different layouts of components. These are arranged in sunburst diagrams, [START_REF] Stasko | An evaluation of space-filling information visualizations for depicting hierarchical structures[END_REF] which are a hierarchical variation on a pie chart. The change from the view shown in Fig. 1(c) to that shown in Fig. 1(d) occurs as the user clicks on the wedge of the mount. The sunburst diagram of the latter then opens up to a full 360 • , and the wedges of the DAQ, camera and mirror disappear. The generic names (mount 0, mount 1, etc.) are temporary place-holders for different elements of the mount sub-system, such as drivers and PLC (programmable logic controller) status indicators. The panel on the right shows a logical grouping of telescopes, as opposed to the pseudo-geographic display of Fig. 1. In this case, telescope positions indicate association to a sub-array, which here stands for a group of telescopes which all point at the same target on the sky. This type of visualization, called an enclosure diagram, is a hierarchical nested layout of elements. In this case, grey-shaded circles represent sub-arrays, with coloured circles standing for telescopes.

Both the panel on the left and that on the right incorporate semantic zooming, as indicated by comparing Figs. 2(a), 2(b), and 2(c). For the left panel, the behaviour is a simple scaling of elements. As one zooms in or out, the size of telescopes and targets changes very slowly with respect to the changing scale of the coordinate system. For the panel on the right, the semantic zoom includes a threshold transition at zoom factor ∼ 2, as shown in Fig. 2(b). In this example, each telescope is represented by a circle and an arc; these respectively indicate the ϕ and δ coordinates of the telescope and its target, using the same visual language as for the panel on the left. Another threshold transition occurs at zoom factor ∼ 14, as presented in Fig. 2(c). In this instance, the purpose of the visualization is to provide detailed information regarding the coordinates of the telescope.

The two panels are synchronized using brushing and linking, following the principles of coordinated multiple views. [START_REF] North | Snap-together visualization: a user interface for coordinating visualizations via relational schemata[END_REF] In Fig. 2(b), a sub-array is selected on the right panel; this causes the associated elements in the panel on the left to become highlighted. Such behaviour allows to quickly identify in the display on the left, which telescopes are assigned to which sub-array. In Fig. 2(c), the single selected telescope is similarly the only element in focus, both on the right panel and on the left. The interplay between the two panels illustrated here also serves to demonstrate an important feature of panel-synchronization. Namely, that synchronization need not necessarily work both ways. As in this case, user interactions with the panel on the right affect the panel on the left by focusing specific elements, but the reverse does not hold.

Figure 3 shows three views of a panel featuring monitoring plots. In this example, the data are extracted from the database of a weather monitoring station. They include measurements of the temperature inside and outside of the weather station, of the speed of the wind near the station, and of the humidity inside of it.

The monitoring plots are placed inside a container, designated as a Dashboard. Plots may be resized and dragged within the Dashboard, or placed in separate Dashboards. The plots are generated using dc.js. They are correlated (a feature of Crossfilter), as shown in, e.g., Fig. 3(b). In this case, the user has selected a range of values of the wind speed parameter (marked by the blue brush). The other plots reflect the selection, by showing only those measurements which have corresponding wind speeds within the chosen range. A side-menu may be overlaid on top of the plots, as shown in Fig. 3(c). Several control options exist. To name a few, the user may add new plots; the user may select new dates for querying the database; and the user may switch to a live-mode, where real-time data are updated every few seconds. The main purpose for implementing this visualization, was to create a practical example for a monitoring panel. In addition, it was also used as a platform to conduct tests of our prototype. These included the following:

testing the performance of the dc.js library for generating synchronized plots; testing the performance of an interface between the Python web-server and an external database; splitting data processing tasks between the Python server and the web-browser client. The results of these tests have so far been encouraging; they have indicated that a web-based framework is suitable for the next stages of development of the operator GUI.

SUMMARY

The planned Cherenkov Telescope Array will be comparably more complex than existing IACT experiments.

This poses new challenges for creating an effective graphical user interface for the operator of the array.

The development of the GUI for CTA follows the successful working model of the ALMA experiment. The design process brings together experienced telescope operators and astroparticle physicists on the one hand, and experts from the field of Human-Computer Interaction on the other. The relevant stakeholders have so far conducted two participatory design workshops, in addition to other face-to-face meetings. The outcome of the work to date is: a refined definition of the scope of the GUI and of the requirements on its performance; a preliminary list of GUI panels; identification of a set of relevant data visualization techniques.

A preliminary prototype of several GUI panels has been implemented. It is based on Web technologies, incorporating a Python web server, Web Sockets and graphics generated with the d3.js Javascript library.

The prototype illustrates semantic zooming and coordinated multiple views, and serves for performance testing of the proposed technology.

Figure 1

 1 Figure 1 shows a representation of the layout of a CTA array, where each circle corresponds to a single telescope. The relative position of telescopes corresponds to a scaled physical layout on-site, constituting a pseudo-geographic display of the array. The attached numerical values and the colour scheme of elements correspond to a health metric. This metric may represent different properties according to the visualization context. The panel incorporates semantic zooming, 10 which defines the context. Semantic zooming is a technique of assigning different layers of information to a given visual element. This behaviour complements the usual geometric zooming, where the size of elements changes with the zoom factor. For the type of semantic zoom implemented in this example, the level of detail associated with an element increases as one zooms-in. The figure shows the evolution of the display for different zoom factors, as follows.

Figure 1 .

 1 Figure 1. Pseudo-geographic display of a CTA array, showing different semantic zoom levels, as described in the text. The zoom factors are 1 in (a), 7 in (b), and ∼ 14 in (c) and (d).

12

 12

Figure 2

 2 Figure 2 presents another telescope-centric view. In this case, two synchronized panels are provided side by side. The panel on the left shows the positions on the sky, at which telescopes point. The polar plot has two coordinates. The angular coordinate, denoted by ϕ, represents the azimuth on the sky; it spans the range from 0 • (at the top-most position of the circle) to 180 • (for the right-hand side), and 0 • to -180 • (for the left-hand side). The radial coordinate, denoted by δ, indicates the zenith, with values between 0 • (at the centre) and 90 • (on the edge of the figure). Telescopes are represented by full circles. Each one has an intended target position, indicated by rings, where faint dashed lines connect telescopes and targets. The display is updated in real-time, showing the movement of telescopes from their initial pointing, towards their respective target positions.

Figure 2 .

 2 Figure 2. Synchronized panels, showing the positions on the sky at which telescopes point (left), and the grouping of telescopes into sub-arrays (right), as described in the text. The zoom factors for the panel on the right are 1 in (a), 2 in (b), and ∼ 14 in (c).

Figure 3 .

 3 Figure 3. Example of a monitoring panel, featuring data extracted from the database of a weather monitoring station. (a) : A selection of data monitoring plots, including measurements of the temperature inside and outside of the weather station, of the speed of the wind near the station, and of the humidity inside of it. (b) : Illustration of the synchronization of plots. A selected range of values of the wind speed parameter (marked in blue), affects the other plots, by excluding data outside of the chosen scope. (c) : A side-menu with various control options may be overlaid on top of the plots, upon clicking the purple button on the top right corner. The option also exists (not shown), of pinning the menu alongside the plots instead.

ACKNOWLEDGMENTS

We would like to thank Caroline Appert, Antonio Cabrera, Francesco Dazzi, Valentina Fioretti, Stafano Gabici, Markus Garczarczyk, Rosa Macias and the members of the ACTL team for their helpful comments and insights.