Martin Westerholt-Raum

Hyper-Algebras of Vector-Valued Modular Forms

almost holomorphic Siegel modular forms. In this work, we further develop the theory of hyperalgebras of modular forms.

Hyper-algebras mimic algebras but they have multivalued multiplication. In particular, they are natural analogues of hyper-groups [START_REF] Wall | Hypergroups[END_REF]. Note that we suppress Wall's assumption of dimensionality of hyper-groups, which does not hold for examples that we treat. Hyper-groups arise naturally in the context of double cosets in group theory. The definition of a group G includes the product a • b ∈ G of a, b ∈ G. Given a sufficiently nice subgroup H -that is, if (G, H ) is a Gelfand pair-one can define commutative Hecke algebras, making thus sense of the product of double cosets H aH • H bH . For instance, if G = GSp g (Q) and K = GSp g (Z), then the Hecke algebra with integral coefficients is free as a Z-module and has basis H c H , c ∈ G. Every product of double cosets can be written a sum c∈G m c H c H , where 0 ≤ m c ∈ Z and m c = 0 for only finitely many c. We rephrase this to arrive at a notion of multivalued multiplication of double cosets. One can decompose H aH = c∈L ( H ,c) occurs with multiplicity m c that occurred before. A more detailed explanation can be found in [Kri90]. Summarizing, we obtain a multiplication of double cosets that takes values in multisets of double cosets:

H \G/H × H \G/H -→ Multiset(H \G/H ).
We thus obtain a commutative hyper-group in this way. Note that every group can be naturally viewed a hyper-group. Vice versa, a hyper-group whose multiplication takes values in multisets of size 1 is a group.

It is possible to extend the definition of hyper-groups to hyper-algebras. Addition is axiomatized as in the case of algebras. Multiplication takes values in the set of all subspaces and compatibility relations that mimic those for algebras are imposed on it. We give a precise definition of hyper-algebras in Section 2. Given an algebra A we obtain a hyper-algebra by assigning to two elements the module spanned by their product. In general, it is not possible to recover the algebra form this, because the hyper-algebra product a • b associated with a, b ∈ A and (r a) • (r b) is the same for all units r, r in the base ring R.

Siegel modular forms are assigned to a weight σ and a type ρ, which are representations of GL g (C) and Sp g (Z), respectively. Given Siegel modular forms f and g of weights and types σ f , ρ f , σ g , and ρ g , then their tensor product f ⊗ g has weight σ f ⊗ σ g and type ρ f ⊗ ρ g . Even if the weights and types of f and g are irreducible, their tensor products can be reducible. On the other hand, to avoid redundancies, graded modules of Siegel modular forms are preferably build from irreducible weights and types. As a consequence, tensor products do not yield an algebra structure of such modules.

Since weights and types behave similarly with respect to the construction that we discuss now, we focus on the former. To remedy the lack of algebra structures on the graded module of Siegel modular forms, we have previously suggested to attach to f and g the span

f ⊗ g = span φ • ( f ⊗ g ) : φ ∈ Hom σ f ⊗ σ g , σ ⊂ σ M (g ) σ ,
where σ runs through a fixed set of representatives of isomorphism classes of irreducible weights, and M (g ) (σ) denotes the corresponding space of Siegel modular forms. In [Kle+15], we identified the resulting structure as a hyper-algebra. It is a special case of hyper-algebras of Siegel modular forms, defined in Section 2.1.

In [Kle+15], we also defined the concept of differential hyper-algebras to subsume the action of covariant differential operators. In Section 2.4, we revisit this definition and reinterpret it in terms of a Hecke operator T ∞ at the infinite place of Q. Returning to the case of general weights and types, we also study vector-valued Hecke operators T v at all places v of Q. If v = p is a prime, then T p generalizes classical Hecke operators for Siegel modular forms. It extends the vectorvalued Hecke operators for elliptic modular forms that we have already defined in [START_REF] Westerholt-Raum | Products of Vector Valued Eisenstein Series[END_REF]. The polynomial algebra T generated by formal elements T v acts by means of these Hecke operators on the hyper-algebra of Siegel modular forms. Specifically, at the infinite place, the Hecke operator acts as a hyper-derivation. At the finite places, Hecke operators respect the hyper-product structure. A precise description can be found in Section 2.6.

Section 3 contains a more detailed study of tensor products and the action of T on modular forms is studied in more detail in the genus 1 case. Let S (1) ( k ) be the space of genus 1 cusp forms of weight k and level 1. Write E (1) (l ) for the level 1 Eisenstein series of weight l . Theorem I. Let l , l ≥ 4 be even integers and ρ a finite dimensional representation of SL 2 (Z) whose kernel is a congruence subgroup. Then for every k ≥ l + l , we have

S (1) k ⊗ ρ ⊂ T T E (1) (l ) ⊗ T E (1) (l ) . (0.1)
Vector-valued Hecke operators defined in this paper can be related to adelic automorphic representations. Covariant differential operators correspond to the action of an appropriate Lie algebra on Harish-Chandra modules. Hecke operators at finite places arise directly from a representation of a group over Q p . We make these relations precise and determine some constituents in the tensor product of principal series. Specifically, we consider automorphic representations for PGL 2 . We let (k -1, 1) be the principal series that is unramified at the finite places and has Harish-Chandra parameter k -1 at infinity. Let ∞ (k -1) be the discrete series with Harish-Chandra parameter k -1.

Theorem II. Given even integers l

, l ≥ 4, then -→ (l -1, 1) ⊗ (l -1, 1)
for every cuspidal automorphic representation = ∞ (k -1) ⊗ f with k ≥ l + l and f a representation of PSL 2 (A f ).

In Section 1, we collect preliminaries on Siegel modular forms. In Section 2, we define hyperalgebras of Siegel modular forms. In Sectoin 3, we show that products of certain Eisenstein series yield cusp forms. In Section 4, we give an interpretation of our results in terms of adelic automorphic representations.

1 Modular forms §1.1 The classical setup. The Siegel upper half space τ ∈ Mat g (C) : t τ = τ, Imτ > 0 of genus g is denoted by H (g ) . It carries an action of the real symplectic group

Sp g (R) = γ ∈ Mat 2g (R) : t γJ (g ) γ = J (g ) , J (g ) = 0 -I (g ) I (g ) 0
, where I (g ) is the g × g identity matrix. This action is explicitly given by γτ = (aτ + b)(cτ + d ) -1 , where we employ the block decomposition of γ = a b c d . The subgroup Γ (g ) = Sp g (Z) of symplectic transformation matrices with integral entries is called the Siegel modular group of genus g .

Generally, a complex representation σ of GL g (C) is called a weight. For the purpose of this paper, we restrict to representation that factor through GL g (C)/{±I (g ) }. A finite dimensional representation ρ of Sp g (Z) is called a type. Throughout this note, we focus on types whose kernel is a congruence subgroup that contains -I (g ) . The representation spaces of σ and ρ are denoted by V (σ) and V (ρ). A weight and a type together determine a slash action

f | σ,ρ γ (τ) = σ(cτ + d ) -1 ρ(γ) -1 f (γτ) on functions f : H (g ) → V (σ) ⊗ V (ρ).
The space of genus g Siegel modular forms of weight σ and type ρ is defined as the space of holomorphic functions f : H (g ) → V (σ) ⊗ V (ρ) such that f | σ,ρ γ = f for all γ ∈ Γ (g ) and which satisfy f (x + i y) = O(1) as y → ∞, if g = 1. We write

M (g ) σ ⊗ ρ = E (g ) σ ⊗ ρ ⊕ S (g ) σ ⊗ ρ
for this space, the space of Eisenstein series, and the space of cusp forms, respectively. §1.2 The symplectic group and its Lie algebra. We write G = PGSp g for the Q-split algebraic group of projective symplectic similitudes. Throughout we work with the model

PGSp g (Z) = γ ∈ Mat 2g (Z) : t γJ (g ) γ = s(γ)J (g ) , s(γ) ∈ Z sI (g ) : s ∈ Z . Compact subgroups K (g ) ∞ and K (g ) p of G(R) and G(Q p ) are U g (R)
/{±I (g ) } and PGSp g (Z p ), where the former is embedded into G(R) by ai +b → a -b b a . Weights correspond to complex representations of K (g ) ∞ by means of the restriction along U g (R)/{±I (g ) } → GL g (C)/{±I (g ) }. We let Rep(K (g ) ∞ ) be a fixed set of representatives of isomorphism classes of finite dimensional, complex representations of K (g ) ∞ . For all g , the representation det k : U

g (R) → C, k → det(k) with k ∈ Z, 2 | g k yields an irreducible representation of K (g ) ∞ via the above isomorphism of K (g ) ∞ and
U g (R)/{±I (g ) }. In the context of spaces of modular forms, we will occasionally write k instead of det k to match more closely the classical notation. For example,

M (g ) k ⊗ ρ = M (g ) det k ⊗ ρ
We denote the l -th symmetric power of the standard representation by sym l . The dual of a representation is indicated by a superscript ∨: σ ∨ is the dual of σ.

We let Rep(K (g ) p ) be a fixed set of representatives of isomorphism classes of finite dimensional, complex representations of K (g ) p . To simplify notation, we let Rep(K (g ) f ) be the set of restricted tensor products of the representations in Rep(K (g ) p ). That is, it consists of exterior tensor products of representations of K (g ) p which are trivial for all but finitely many p. There is a correspondence between Rep(K (g ) f ) and finite dimensional, complex representations of Γ (g ) with a congruence subgroup and the matrix -I 2g in their kernel.

Subsets of irreducible representations are denoted by

IrrRep K (g ) ∞ ⊂ Rep K (g ) ∞ and IrrRep K (g ) f ⊂ Rep K (g ) f . §1.3 Covariant differential operators. We call a differential operator D covariant from | σ,ρ to | σ ,ρ if for all g ∈ G(R) and all smooth functions f : H (g ) → V (σ) ⊗ V (ρ) , we have D f | σ,ρ g = D( f )| σ ,ρ g .
If D is covariant for the trivial type then it yields differential operator that is covariant for all types.

A theorem of Helgason [Hel62], allows us to classify order 1 differential operators. For each σ there is a lowering operator L = L σ that is covariant from σ to Lσ = sym 2 ∨ σ and a raising operator R = R σ that is covariant form σ to Rσ = sym 2 σ. We only fix normalization in the case g = 1, setting

L = -2i y 2 ∂ τ and R = 2i ∂ τ + k y -1 .
In [Kle+15], we gave explicit expressions for all g . The normalization employed there, however, does not coincide with the one we choose here.

Covariant differential operators allow us to define almost holomorphic Siegel modular forms. A function f : H (g ) → V (σ) ⊗V (ρ) that vanishes under the (d + 1)-th tensor power of the lowering operator, i.e. L d f = 0, that is invariant of weight σ and type ρ, i.e. f | σ,ρ γ = f for all γ ∈ Γ (g ) , and that satisfies f (τ) = O(1) as y → ∞, if g = 1, is called an almost holomorphic Siegel modular form of weight σ, type ρ, and depth d . The space of such functions is denoted by

[d ] M
(g ) σ ⊗ ρ . §1.4 Hecke operators. We extend the definition of vector-valued Hecke operators in [START_REF] Westerholt-Raum | Products of Vector Valued Eisenstein Series[END_REF] for elliptic modular forms to Siegel modular forms. Many of the proofs in op. cit. apply to the general case word by word. For this reason, we skip several arguments in this subsection.

Hecke operators on representations. For a positive integer M , we let

∆ M = a b 0 d ∈ GSp g (Q) : t ad = M I (g ) , a, b, d ∈ Mat g (Z), d upper triagonal, ∀i < j : 0 ≤ d i , j < d j , j , ∀i : 0 ≤ b i , j , < d j , j ,
where GSp g is the group of symplectic similitude transformations. We have a right action of Sp g (Z) on ∆ M defined by (m, γ) → mγ with γ mγ = mγ for some γ ∈ Sp g (Z). This, in particular, defines a cocycle

I m (γ) = γ ; That is, we have I m (γ 1 γ 2 ) = I m (γ 1 )I mγ 1 (γ 2 ).
To every type ρ we associated the type T M ρ defined by

V (T M ρ) := V (ρ) ⊗ C ∆ M and (T M ρ)(γ) (v ⊗ e m ) := ρ I -1 m (γ -1 ) (v) ⊗ e mγ -1
(1.1)

The cocycle property of I m (γ) implies that it is a representation of Sp g (Z). Given a scalar prod-

uct 〈 • , • 〉 ρ on V (ρ), we obtain one on V (T M ρ) by 〈v ⊗ e m v , w ⊗ e m w 〉 =    〈v, w〉 ρ , if m v = m w ; 0, otherwise.
Hecke operators on representations are compatible with homomorphisms between types and with tensor products. Specifically, the following are homomorphism of Sp g (Z)-representations:

T M φ : T M ρ -→ T M ρ , v ⊗ e m -→ φ(v) ⊗ e m ;
(1.2)

1 -→ T M 1, c -→ c m∈∆ M e m ; (1.3) (T M ρ) ⊗ (T M σ) -T M (ρ ⊗ σ), (v ⊗ e m ) ⊗ (w ⊗ e m ) -→    (v ⊗ w) ⊗ e m , if m = m ; 0, otherwise;
(1.4)

where φ ∈ Hom(ρ, ρ ). If ψ : ρ → ρ is a further homomorphism, then T M (ψ•φ) = (T M ψ)•(T M φ).
For later reference, we denote the morphism in (1.4) by π M , ρ,ρ .

Hecke operators on modular forms. For m = a b 0 d ∈ GSp g (R) of similitude M , and for f :

H → V (σ), we define f | σ m (τ) = σ d / M -1 f (aτ + b)d -1 .
Fixing a type ρ and a positive integer M , we define the vector-valued Hecke operator T M acting on f ∈ M (g ) σ ⊗ ρ by

T M f (τ) = m∈∆ M f | σ m (τ) ⊗ e m ∈ M (g ) σ ⊗ T M ρ .
(1.5)

The above homomorphism π M , ρ,ρ is compatible with Hecke operators. Specifically, we have

π M , ρ,ρ (T M f ) ⊗ (T M g ) = T M f ⊗ g .

Hyper-algebras

We start with the formal definition of hyper-groups, which is the blueprint to our Definition 2.2 of hyper-algebras. Given a set S, let Multiset(S) ∼ = { f : S → Z ≥0 } be the set of all multisubsets of S. Multisets are throughout denoted by double curly brackets {{• • • }}. The union of multisets corresponds to the sum of functions S → Z ≥0 . As a shorthand notation, for the binary operator appearing in the next definition, we set a

• {{b i }} i = b i a • b i and similarly {{a i }} i • b = a i a i • b. Definition 2.1. A pair (G, • ) of a set G and a binary operator • : G × G → Multiset(G) is called a hyper-group if the following axioms are satisfied: (i) (Finite image) For any a, b ∈ G, the multiset a • b is finite. (ii) (Associativity) For any a, b, c ∈ G, we have (a • b) • c = a • (b • c), that is, h∈a • b h • c = h∈b • c a • h.
(iii) (Identity) There is an element e ∈ G such that for every a ∈ G we have a ∈ a • e and a ∈ e • a.

(iv) (Inverse) For every element a ∈ G there is element a -1 ∈ G such that e ∈ a • a -1 and e ∈ a -1 • a.
Extending this notion to hyper-algebras is straightforward. However, the reader should be warned that the word "hyper-algebra" is used in the context of algebraic groups [START_REF] Sullivan | Representations of the hyperalgebra of an algebraic group[END_REF], too, and these two notations should not be confused.

Given a commutative ring R (with identity), and an R-module M , we let SubMod R (M ) be the set of all R-submodules of M . In analogy with the definition of multisets, we extend the binary operator • in the following definition to submodules by a

• N = b∈N a • b and N • b = a∈N a • b for N ∈ SubMod R (M ).
Definition 2.2. Let R be a ring. A triple (A, +, • ) with (A, +) an R-module and binary operator

• : A × A -→ SubMod R (A) is called a hyper-algebra (with identity) if (i) (Linearity) Given a, b ∈ A and r ∈ R, we have (r a) • b = r a • b = a • (r b) (ii) (Identity) There is e ∈ A such that we have e • a = a • e = span R a for all a ∈ A. (iii) (Associativity) Given a, b, c ∈ A, we have a • (b • c) = (a • b) • c. (iv) (Distributivity) for a, b, c ∈ A, we have a • (b + c) ⊆ a • b + a • c.
Concepts like commutativity, grading, and derivation extended to hyper-algebras. Fixing a hyper-algebra A, we call it commutative if for all a, b ∈ A, we have ab = ba. We say that it is graded by a hyper-group G if A = g A g as an R-module and

A g A g ⊆ h∈g g A h . A hyper-derivation d on A is an R-module endomorphism such that d (a • b) ⊆ (d a) • b + a • (d b).
Note that for a hyper-algebra A, we can define a (left) hyper-module M as an R-module with binary operator • : A ×M → SubMod R (M ) satisfying the analogue of the axioms in Definition 2.2. §2.1 Hyper-algebras of modular forms. Recall the various sets of representations defined in Section 1.2. The hyper-algebra of holomorphic Siegel modular forms as a vector space is

M (g ) • ⊗ • = σ,ρ M (g ) σ ⊗ ρ , σ ∈ IrrRep K (g ) ∞ , ρ ∈ IrrRep K (g ) f (2.1) with product f g = f ⊗ g ⊂ M (g ) • ⊗ • defined as span (φ σ ⊗ φ ρ ) • ( f ⊗ g ) : σ ∈ IrrRep K (g ) ∞ , φ σ : σ f ⊗ σ g → σ, ρ ∈ IrrRep K (g ) f , φ ρ : ρ f ⊗ ρ g → ρ . (2.2) Note that both IrrRep K (g ) ∞ and IrrRep K (g ) f
are hyper-groups, and the hyper-algebra of Siegel modular forms is graded by both. Our notation for submodules, for example M (g ) • ⊗ρ , has obvious meaning.

We will also work with the hyper-algebras of almost holomorphic Siegel modular forms

[•] M (g ) • ⊗ • = 0≤d σ,ρ [d ] M (g ) σ ⊗ ρ , σ ∈ IrrRep K (g ) ∞ , ρ ∈ IrrRep K (g ) f (2.3)
with product as in (2.2). §2.2 Computing the hyper-algebra product in Sage. In the case of elliptic modular forms, all irreducible weights are 1-dimensional. Types can, however, be arbitrary dimensional. We adopt an example from [START_REF] Westerholt-Raum | Products of Vector Valued Eisenstein Series[END_REF], which illustrates how to compute products in M (g ) • ⊗ • of modular forms with nontrivial type. For convenience, we give Sage code [Sage] 2 , which the reader can modify to perform his or her own computation-The code is not optimized for performance, but for clarity. We consider the representation ρ 3 , which we realize as a matrix representation by

ρ 3 (T ) =    1 0 0 0 0 1 -1 -1 -1    , ρ 3 (S) =    0 1 0 1 0 0 -1 -1 -1    .
The space of Eisenstein series of weight 12 and type ρ 3 has dimension one. A basis element E 12,ρ 3 can be obtained from vector-valued Hecke operators that are discussed in Section 1.4. We let ζ be a third root of unity. The image of T 3 on the level 1 Eisenstein series of weight 12 is representations with representation matrices

ρ ζ (S) = 1 , ρ ζ (T ) = ζ and ρ ζ 2 (S) = 1 , ρ ζ 2 (T ) = ζ 2 .
In Sage, we implement them by means of s z e t a = i d e n t i t y _ m a t r i x (K, 1 ) t z e t a = m a t r i x (K , 1 , [ z e t a ] ) s z e t a 2 = i d e n t i t y _ m a t r i x (K, 1 ) t z e t a 2 = m a t r i x (K , 1 , [ z e t a * * 2 ] ) First, we determine the kernel of the homomorphism that we have determined so far.

bm = reduce ( lambda s , l : s . i n t e r s e c t i o n ( m a t r i x ( [ l ] ) . r i g h t _ k e r n e l ( ) )

,

[ h o m _ t r i v . b a s i s ( ) [ 0 ] ] + [ b [ i x : : 3 ] f o r b i n hom_rho3 . b a s i s ( ) f o r i x i n range ( 3 ) ]
, V e c t o r S p a c e (QQ, 9 ) ) \ . b a s i s _ m a t r i x ( ) . t r a n s p o s e ( ) s r e s t = bm . s o l v e _ r i g h t ( s 3 3 * bm) t r e s t = bm . s o l v e _ r i g h t ( t 3 3 * bm) Second, we observe that S acts trivially on that kernel, so that it suffices to decompose the action of T into eigenspaces. In the present case they are defined over a third order cyclotomic extension of the rationals. In fact, they are isomorphic to ρ ζ and ρ ζ 2 given above. hom_zeta = hom ( s33 , t33 , s z e t a , t z e t a ) hom_zeta2 = hom ( s33 , t33 , s z e t a 2 , t z e t a 2 )

We construct matrices from the homomorphism spaces that we previously determined.

p h i _ z e t a = m a t r i x ( hom_zeta . b a s i s ( ) [ 0 ] ) p h i _ z e t a 2 = m a t r i x ( hom_zeta2 . b a s i s (

) [ 0 ] ) phi_rho3_1 = m a t r i x ( [ hom_rho3 . b a s i s ( ) [ 0 ] [ i x : : 3 ] f o r i x i n range ( 3 ) ] ) phi_rho3_2 = m a t r i x ( [ hom_rho3 . b a s i s ( ) [ 1 ] [ i x : : 3 ] f o r i x i n range ( 3 ) ] )
Assembling the results that we have computed via Sage, we find that with respect to the given bases, we have homomorphisms

φ 1 : ρ 3 ⊗ ρ 3 -→ 1, 1 1 2 1 2 1 2 1 1 2 1 2 1 2 1 , φ ζ : ρ 3 ⊗ ρ 3 -→ ρ ζ , 1 ζ + 1 -ζ ζ + 1 ζ -1 -ζ -1 -ζ -1 , φ ζ 2 : ρ 3 ⊗ ρ 3 -→ ρ ζ 2 , 1 -ζ ζ + 1 -ζ -ζ -1 -1 ζ + 1 -1 ζ , φ ρ 3 ,1 : ρ 3 ⊗ ρ 3 -→ ρ 3 ,    1 0 -1 -1 -1 -2 0 1 -1 -1 0 1 -1 1 0 -2 -1 -1 -1 -2 -1 1 -1 0 0 -1 1    , φ ρ 3 ,2 : ρ 3 ⊗ ρ 3 -→ ρ 3 ,    0 1 -1 -1 0 -3 1 3 0 0 1 3 -1 0 1 -3 -1 0 0 -3 -1 3 0 1 1 -1 0    .
Summarizing, we find that E 12,ρ 3 ⊗ E 12,ρ 3 is supported on 1, ρ ζ , ρ ζ 2 , and ρ 3 . The corresponding subspaces are spanned by elements whose Fourier coefficients are too large to display them all. We confine ourselves to the trivial type: 564856947200 1594323

-1894333004462080000 84584326707 q -1261863434802833408000 28194775569 q 2 + O(q 3 ) ∈ E 12,ρ 3 ⊗ E 12,ρ 3 ∩ M 24 ⊗ 1 .
Beyond this, we illustrate how to compute the remaining Fourier expansions with Sage. The tensor square of E 12,ρ 3 with precision at least n is given by the following function Esq(n).

Esq = lambda n : v e c t o r (R , [ c1 * c2 f o r c1 i n E12rho3 ( n ) f o r c2 i n E12rho3 ( n ) ] )
p r i n t p h i _ t r i v * Esq ( 3 ) p r i n t p h i _ z e t a * Esq ( 3 ) p r i n t p h i _ z e t a 2 * Esq ( 3 ) p r i n t ( phi_rho3_1 * Esq ( 3 ) , phi_rho3_2 * Esq ( 3 ) ) §2.3 From weights to isomorphism classes of irreducible weights. We have defined weights as complex, finite dimensional representations of K (g ) ∞ without any further restriction. If we wished to construct a graded algebra of modular forms whose grading includes all these weights, this would not be possible. All representations of K (g ) ∞ together do not constitute a set, but rather they are objects in a category Rep(K (g ) ∞ ). Given this fact, one might be inclined to pass from weights to isomorphism classes of weights-the skeleton of Rep(K (g ) ∞ ). Alternatively, we can and will focus on a fixed set Rep(K (g ) ∞ ) of representatives of isomorphism class. For every σ ∈ Rep(K (g ) ∞ ) the space of modular forms M (g ) ( σ ) of weight σ is a vector space.

Let us consider to what extend we can define a multiplication. Given f ∈ M (g ) ( σ f ) and g ∈

M (g ) ( σ g ) with σ f , σ g ∈ Rep(K (g ) ∞ )
, we find that

f g = f ⊗ g ∈ M (g ) σ f ⊗ σ g
for their tensor product. Its weight, in general, is not in Rep(K (g ) ∞ ). Denote by σ f σ g the corresponding representative. A priori, f g lies in C ∞ H (g ) → V (σ f σ g ) . We have to obtain from it an element in C ∞ H (g ) → V (σ f σ g ) . The most naive way is to choose an endomorphism φ in Hom σ f σ g , σ f σ g and compose it with f g . The resulting function φ • f g depends on φ. More precisely, it depends on f and g up to automorphisms of σ f σ g .

Except in the case g = 1, there seems to be no natural choice of endomorphisms for all tensor products. For instance, sym l sym l contains sym l +l in a natural way by realizing sym l and sym l as a representation on polynomials of degree l and l in g variables. for example if g = 2, there is no natural inclusion of sym l +l -2 into sym l sym l for l , l ≥ 2. The situation becomes even more difficult for g ≥ 3, because then irreducible representations can occur with multiplicities greater than 1 in tensor products of irreducible representations.

Since there is no natural choice of a single homomorphism from σ f σ g to σ f σ g , we retrieve to taking the span over all of them. The product of f and g in this setting is defined as the space

span φ • f g : φ ∈ Hom σ f σ g , σ f σ g ⊂ σ∈Rep(K (g ) ∞ ) M (g ) ( σ ).
This product yields a hyper-algebra of modular forms whose weights run through all isomorphism classes of representations. Since at this point, we have already lost the algebra structure and arrived at hyper-algebras, there is no additional harm in considering only irreducible weights. This motivates our definition in Subsection 2.1. §2.4 Covariant differential operators on hyper-algebras. As in the case of tensor products the image of lowering and raising operators has, in general, reducible weight. For example, in the case g = 2 and if l ≥ 2, the weight R det k sym l = det k sym 2 sym l allows for a decomposition into irreducible weights det k-2 sym l -2 , det k-1 sym l , and det k sym l +2 . It is therefore natural to define a vector space valued action of covariant differential operators.

Let C L, R be the polynomial algebra in two formal variables L and R. Notation overlaps with the one for lowering and raising operators, but it will be clear form the context to what we refer when writing L f or R f . The action of L and R on [•]

M (g ) • ⊗ • is given by L f = span φ • L σ f f : σ ∈ IrrRep K (g ) ∞ , φ : σ → L σ f and R f = span φ • R σ f f : σ ∈ IrrRep K (g ) ∞ , φ : σ → R σ f .
Viewing the differential operators L and R jointly as a vector-valued Hecke operator at the infinite place acting on almost holomorphic Siegel modular forms, we set for f ∈

[•] M (g ) • ⊗ • T ∞ = C T ∞ , T ∞ f = L f + R f ⊂ [•] M (g ) • ⊗ • . (2.4)
One readily verifies by employing the defining formulas of lowering and raising operators in [Kle+15] that T ∞ acts as a hyper-derivation. §2.5 Hecke actions on hyper-algebras. Based on the vector-valued Hecke operators that we have introduced in Section 1.4, we get an additional hyper-module structure on (almost) holomorphic Siegel modular forms. Let

T f = C T p : p prime (2.5)
be the polynomial ring in infinitely many formal variables T p . As in the case of covariant differential operators, notation coincides with the one for actual Hecke operators, but this should not lead to confusion. The action of T p on [•] M (g ) • ⊗ • is defined by

T p f = span φ • T p f : ρ ∈ IrrRep K (g ) f , φ : T p ρ f → ρ ⊂ [•] M (g ) • ⊗ • . (2.6)
This equips [•] M (g ) • ⊗ • with the structure of a T f -hyper-module. §2.6 The formal Hecke algebra. We combining both formal Hecke algebras into one

T = T ∞ T A f = C T ∞ , T p : p prime . (2.7)
Recall that T ∞ ⊂ T acts on almost holomorphic Siegel modular forms by hyper-derivations, and T f acts by endomorphisms.

Essential surjectivity of tensor products of modular forms

The hyper-algebra structure and the action of the formal Hecke algebra make it rather easy to formulate essential surjectivity for products of modular forms. The blueprint for such results are the ones in [Kle+15; WR14]. The first of them, phrased in the language that we have developed reads as follows. For all k ≥ 8, all l , kl ≥ 4, and all congruence representations ρ, we have

M (1) k ⊗ • = E (1) k ⊗ • + T f E (1) (l ) ⊗ T f E (1) (k -l ) . (3.1)
In the language we have developed, it is straightforward to ask for analogues. §3.1 A conjecture in the case of genus 2. We start with a conjecture, that we will not prove in this paper. There is experimental evidence in [START_REF] Raum | Efficiently generated spaces of classical Siegel modular forms and the Böcherer conjecture[END_REF] that

M (2) k ⊗ 1 = SK (2) k ⊗ 1 + 4≤l ≤k-4 SK (2) l ⊗ 1 • SK (2) k -l ⊗ 1 , (3.2)
where SK denotes the space of (holomorphic) Saito-Kurokawa lifts. In light of this and of Formula (3.1) an extension to all ρ seems possible. Could it be that for sufficiently large k, l , and kl , and for any congruence type ρ of genus 2 Siegel modular forms, we have

M (2) k ⊗ ρ = SK (2) k ⊗ ρ + T f SK (2) l ⊗ 1 ⊗ T f SK (2) k -l ⊗ 1 ? (3.3)
Note that we have restricted to scalar weights in the above. To cover vector valued weights, we have to first study the effect of covariant differential operators on tensor products. This will be executed, in case of g = 1, in the next subsection. §3.2 Hecke operators at all places in genus 1. Making use of not only Hecke operators at the finite places, but of the full formal Hecke algebra T, one can strengthen surjectivity results as the one in (3.1).

Theorem 3.1. Let l , l ≥ 4 be even integers, and ρ a congruence type. Then for every even k ≥ l + l we have

S (1) k ⊗ ρ ⊂ T ∞ T E (1) (l ) ⊗ T E (1) (l ) . (3.4)
Proof. To ease notation, we suppress the superscript (1) for Eisenstein series throughout this proof. Without loss of generality, we can assume that l ≤ l . In complete analogy with [START_REF] Westerholt-Raum | Products of Vector Valued Eisenstein Series[END_REF], we note that the right hand side of (3.4) is a T f -module. It therefore suffices to show the following: Any newform f for the congruence subgroup Γ 0 (N ) vanishes, if for t = (l + lk)/2 ∈ Z ≥0 , the (holomorphic) modular form f D is orthogonal to the almost holomorphic modular form

Ind E l ,1 |D| ⊗ R t E l ,1 |D| ,∞ .
Notation is adopted from [START_REF] Westerholt-Raum | Products of Vector Valued Eisenstein Series[END_REF]: We let 1 |D| be the square of Kronecker character D for a negative fundamental discriminant D. Note hat 1 |D| is a trivial non-primitive Dirichlet character. The modular form f D is the twist of f by D . Further, we define Eisenstein series

E l ,1 |D| (τ) = ∞ n=1 σ l -1,1 |D| (n)q n E l ,1 |D| ,∞,s = γ∈Γ ∞ \SL 2 (Z) e Γ 0 (N ) ⊗ e Γ 0 (N ) y s k,ρ 1 |D| ⊗ρ 1 |D| γ .
The induction of modular forms for Γ 0 (N ) to vector-valued modular forms for SL 2 (Z) is denoted by Ind. The induction of a Dirichlet character χ from Γ 0 (N ) a Dirichlet character χ of modulus N is denoted by ρ χ . We establish the described vanishing condition by relating the Petersson scalar product to special L-values, as in [IK05; KZ84; WR14]. Combining regularization and unfolding as is described in detail in [START_REF] Westerholt-Raum | Products of Vector Valued Eisenstein Series[END_REF], we find that

Ind f D , Ind E l ,1 |D| ⊗ R t E l ,1 |D| ,∞,s = Γ (1) \H (1) π f • E l ,1 |D| (τ) • e Γ 0 (N ) ⊗ e Γ 0 (N ) R t y s dxdy y 2-k , (3.5)
where π is the projection adjoint to the inclusion

1 -→ ρ |D| ⊗ ρ |D| ⊗ ρ N |D| 2 ⊗ ρ N |D| 2 , 1 -→ γ: Γ 0 (|D|)\SL 2 (Z) γ : Γ 0 (|D|N )\SL 2 (Z) e γ ⊗ e γ ⊗ e γ ⊗ e γ .
Evaluating the integral, employing analytic continuation, and then inserting s = 0, we obtain a product of special values of Dirichlet series.

A direct verification shows that lim s→0 R t l y s = (l ) ↑ t y -t , where (a) ↑ n = a(a +1) • • • (a +n -1) is the upper factorial. The above scalar product (3.5) therefore equals

(l ) ↑ t ∞ n=0 ∞ 0 c( f D ; n)e(ni y) σ l -1,1 |D| (n)e(ni y) y -t +s dy y 2-k s=0 = (l ) ↑ t Γ(k -1 -t ) (4π) k-1-t ∞ n=0 σ l (n)c( f D ; n) n k-1-t +s s=0
.

By the extension of Rankin's result [START_REF] Rankin | The scalar product of modular forms[END_REF] in [START_REF] Westerholt-Raum | Products of Vector Valued Eisenstein Series[END_REF], we find that it does converge absolutely, if l > l or allows for a suitable analytic continuation of l = l . It can be expressed in terms of special L-values as follows.

L f D , k -1 -t L f D × 1 |D| , k -l -t L 1 |D| 1 |D| 1, k -1 -2t -l )
.

The denominator equals L 1 |D| , l -1) by the relation k = 2t +l +l . The first factor in the numerator is a special value of an Euler product, since t = (kll )/2 < k 2 -2 and thus k -1t > k 2 + 1. In order to inspect the second factor in the numerator, note that l +t ≤ k 2 , since we have assumed that

l ≤ l . If l + t < k 2 then k -t -l ≥ k 2 + 1, so that L( f D , k -t -l )
is the special value of a convergent Euler product. Otherwise, we infer that the central value L( f D , k

2 ) vanishes for all negative fundamental discriminants D. Using Waldspurger's and Kohnen-Zagiers results [KZ84; Wal81], we can argue as in [START_REF] Westerholt-Raum | Products of Vector Valued Eisenstein Series[END_REF] to finish the proof. Theorem 3.1 does not cover the case of small weights k. The next statement clarifies that k < l + l cannoth appear on the right hand side of (3.4). Proposition 3.2. Suppose that k < l 1 + l 2 for positive even l 1 , l 2 ≥ 4 and positive even k. Then any weight k cusp form f is orthogonal with respect to the Petersson scalar product to R t 1 g 1 • R t 2 g 2 for g 1 and g 2 modular forms of weight l 1 and l 2 , and t 1 ,

t 2 ∈ Z ≥0 such that k = l 1 + 2t 1 + l 2 + 2t 2 . Proof. The almost holomorphic modular form R t 1 g 1 •R t 2 g 2 has depth t 1 + t 2 . That is, it allows for a decomposition R t 1 g 1 • R t 2 g 2 = t 1 +t 2 t =0 R t h t
for holomorphic modular forms h t of weight k -2t . Since d > t 1 + t 2 , each term in this sum is orthogonal to f by Shimura's orthogonality relations [Shi87]-also confer [START_REF] Pitale | Representations of SL 2 (R) and nearly holomorphic modular forms[END_REF].

Adelic automorphic representations

Automorphic representation theory in most modern settings focuses on adelic representations. That is, one investigates the right regular representation of G(A) on L 2 G(Q)\G(A) . It is an important aspect of the theory that one can split any automorphic representation into a restricted tensor product of local components. The local theory is over p-adic fields and the theory over the infinite places.

In the classical theory of modular forms these two aspects of automorphic representation theory are reflected by Hecke operators and covariant differential operators. To support our claim that hyper-algebras of modular forms together with the action of T are mitigating between the classical language and the representation theoretic one, we show how to pass from classical modular forms to the attached local representations and how some of their aspects can be interpreted in terms of hyper-algebras. In particular, we rephrase our result in Section 3 in terms of tensor products of automorphic representations. §4.1 Harish-Chandra modules. It is formally correct to work with the group PGSp g (R) in this subsection. However, as is common, we will instead work with Sp g (R), for which all aspects that we discuss here are the same. Further, we adopt notation from [START_REF] Westerholt-Raum | Harmonic Weak Siegel Maaß Forms I[END_REF] to shorten this exposition.

Given any almost holomorphic Siegel modular form f , we can associate to it an irreducible Harish-Chandra module. Recall that a Harish-Chandra module is an admissible (g, K)-module. A (g, K)-module is a representation of K = K (g ) ∞ that is simultaneously a g-module with g = sp g , and for which these two structures are compatible. A detailed definition can be found in Section 3.3.1 of [START_REF] Wallach | Real reductive groups. I[END_REF]. A (g, K)-module M is called admissible if it is a unitarizable K -representation and if Hom K (σ, M ) is finite dimensional for every finite dimensional K -representation σ.

Starting with f , we produce a function A ∞ ( f ) on G-A ∞ stands for adelization at the infinite place, which here is isomorphic to R:

A ∞ ( f )(g ) = f | σ,ρ g (i I (g ) ).
It takes values in V (σ) ⊗ V (ρ), and when contracting with V (σ) ∨ , we obtain a space of functions that take values in V (ρ) and which is a K -module isomorphic to σ ∨ ⊗ V (ρ). We denote the contraction by A ∞ ( f ) • V (σ) ∨ , and the (g, K)-module generated by it will be denoted by

A ∞ ( f ) = (g, K) A ∞ ( f ) • V (σ) ∨ .
Let k ⊂ g be the Lie algebra of K . It was established in [START_REF] Westerholt-Raum | Harmonic Weak Siegel Maaß Forms I[END_REF] that application of covariant differential operators and the action of the k-complement m ⊆ g commute with passing back and forth between modular forms and K -types in Harish-Chandra modules. Concretely, we established commutativity of the following diagram, featuring the hyper-derivations L and R:

f A ∞ ( f ) R f m + A ∞ ( f ) A ∞ A ∞ R m + f A ∞ ( f ) L f m -A ∞ ( f ) A ∞ A ∞ L m -
The product of vector valued modular forms then contains information about the tensor product of Harish-Chandra modules. And thus it yields a lower bound on smooth functions in the tensor product of G(R)-representations. Given a Harish-Chandra module we can obtain from it a representation of Sp g (R). In our case, we denote this representation by G(R)A ∞ .

Corollary 4.1. Given two almost holomorphic Siegel modular forms f and g , we have the following inclusion of (g, K)-modules:

A ∞ T ∞ f ⊗ T ∞ g ⊆ A ∞ ( f ) ⊗ A ∞ (g ) .
By passing back to representations of Sp g (R), we find that

G(R)A ∞ T ∞ f ⊗ T ∞ g ⊆ G(R)A ∞ ( f ) ⊗ G(R)A ∞ (g ).
Proof. This follows from the commutative diagrams above, except that we have to verify that the hyper-product T ∞ f ⊗ T ∞ g yields an admissible (g, K)-module. This becomes clear when inspecting the action of the center of k. §4.2 The finite places. As opposed to the infinite place it is important to insist on the group G = PGSp g when treating the finite places. Recall that as a maximal compact group over Q p , we choose K (g ) p = PGSp g (Z p ). Also recall that any type in this paper is a congruence type, which means that it gives rise to a representation over A f .

To approximate at p, we need the relation G(Q p ) = G(Q)K As a first remark note that f is constant on ker ρ p ⊆ K (g ) p . The contraction of A p ( f ) with V (ρ p ) ∨ will be denoted by A p ( f ). It is a space of functions on G(Q p ) taking values in V (σ) ⊗V (⊗ p =p ρ p ).

As a K (g ) p -module it has isomorphism type ρ ∨ p ⊗ V (σ) ⊗ V (⊗ p =p ρ p ) . The commutative diagram corresponding to the one in Subsection 4.1 is

f A p ( f ) T p f ∆ p A p ( f ) A p A p T p ∆ p
where ∆ p is the vector space with basis consisting of determinant p matrices in K (g ) p . In analogy with the infinite case, we write G(Q p )A p ( f ) for the representation over Q p that corresponds to f .

  H ,a) H c and H bH = c∈L(H ,b) H c as a disjoint union of left cosets with coset representatives L(H , a), L(H , b) ⊆ G. By definition of the multiplication in Hecke algebras, the multiset {{H (d a • d b ) : d a ∈ L(H , a), d b ∈ L(H , b)}} has a decomposition into the disjoint union of {{H d c : d c ∈ L(H , c)}} where each L(

K

  .< z e t a > = C y c l o t o m i c F i e l d ( 3 )

  a vector-valued almost holomorphic Siegel modular formf of type ρ = ⊗ ρ p . Given g k ∈ G(Q p ) we set A p ( f )(g k) = ρ -1 p (k) f | σ,ρ g (i I (g ) ).

computations make implicit use of the libraries [Flint; Pari] and possibly further ones that are less obvious from the source code of Sage

The author was partially supported by Vetenskapsrådet Grant 2015-04139.

Fourier expansions are computed in terms of q3 = e(τ/3). The components of an Eisenstein series of type ρ 3 can be found by applying a homomorphism from T 3 1 to ρ 3 , which can be computed by the method that we describe below when decomposing ρ 3 ρ 3 .

To compute E 12,ρ 3 ⊗ E 12,ρ 3 , we have to decompose the 9-dimensional representation ρ 3 ⊗ ρ 3 , by exhibiting the trivial representation in (ρ 3 ⊗ ρ 3 ) ∨ ⊗ ρ for various representations ρ. For a systematic decomposition of ρ 3 ⊗ ρ 3 , one could apply the MeatAxe algorithm [HEO05]. We start by defining the representation matrices of the trivial representation 1, ρ 3 , and ρ 3 ⊗ ρ 3 .

t e n s o r _ p r o d u c t ( s 3 ) t 3 3 = t 3 . t e n s o r _ p r o d u c t ( t 3 )

We compute homomorphisms between representations by employing the isomorphism of vector spaces Hom(ρ, ρ ) ∼ = ρ ∨ ⊗ ρ (1). Representation matrices of the dual representation ρ ∨ are given by the transpose inverses of those of ρ. We can immediately compute homomorphisms from ρ 3 ρ 3 to 1 and ρ 3 .

There is one copy of 1 and two copies of ρ 3 in ρ 3 ρ 3 . We are facing the problem of decomposing their complement. In our case, it turns out that it consists of two inequivalent one-dimensional Corollary 4.2. Given Siegel modular forms f and g , we have the following inclusion of G(Q p )-representations:

.3 Global tensor products. We now transfer the statements of Corollary 4.1 and 4.2 to the global setting.

Theorem 4.3. Given almost holomorphic Siegel modular forms f and g , with automorphic representations ( f ) and (g ) associated with them. Suppose that for a newform h we have

Then the associated automorphic representation (h) is contained in the tensor product of those associated with f and g :

Proof. Using strong approximation, we obtain a map from holomorphic Siegel modular forms f to functions

on G(A), which generate an automorphic representation ( f ) = ∞ ( f ) ⊗ ⊗ p p ( f ) associated with f . Note that the Harish-Chandra module attached to a representation of G(R) consists of the subspace of smooth vectors. In particular, it is a subspace of V ( ∞ ( f )). The commutative diagrams in Subsection 4.1 and 4.

As a consequence of Theorem 4.3, we can reinterpret (3.1) and (3.4) in terms of tensor products of global representations. This provides a proof of Theorem II.


domain: info scco

text:
Céline Craye
email: celine.craye@ensta-paritech.fr

David Filliat

Jean-Franc ¸ois Goudou

Environment Exploration for Object-Based Visual Saliency Learning

Searching for objects in an indoor environment can be drastically improved if a task-specific visual saliency is available. We describe a method to incrementally learn such an object-based visual saliency directly on a robot, using an environment exploration mechanism. We first define saliency based on a geometrical criterion and use this definition to segment salient elements given an attentive but costly and restrictive observation of the environment. These elements are used to train a fast classifier that predicts salient objects given large-scale visual features. In order to get a better and faster learning, we use an exploration strategy based on intrinsic motivation to drive our displacement in order to get relevant observations. Our approach has been tested on a robot in indoor environments as well as on publicly available RGB-D images sequences. We demonstrate that the approach outperforms several state-of-the-art methods in the case of indoor object detection and that the exploration strategy can drastically decrease the time required for learning saliency.

I. INTRODUCTION

Object detection or object search in cluttered environments by mobile robots is still a difficult problem. Despite progress in computer vision for detecting objects in unconstrained images, especially using deep-learning which has recently shown impressive results on complex image datasets such as IMAGENET [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], it remains particularly interesting in a robotics context to move the robot to favorable observation conditions in order to improve recognition performances. Such visual exploration of the environment [START_REF] Björkman | Active 3d scene segmentation and detection of unknown objects[END_REF], [START_REF] Frintrop | VOCUS: A visual attention system for object detection and goal-directed search[END_REF], [START_REF] Meger | Curious george: An attentive semantic robot[END_REF] is often associated with a visual attention strategy so as to direct the robot's attention towards areas of interest, while ignoring irrelevant portions of the visual field.

The selection of areas of interest is typically driven by visual saliency maps [START_REF] Björkman | Active 3d scene segmentation and detection of unknown objects[END_REF], [START_REF] Frintrop | VOCUS: A visual attention system for object detection and goal-directed search[END_REF], [START_REF] Meger | Curious george: An attentive semantic robot[END_REF] or, if depth is available, geometrical segmentation [START_REF] Ali | Contextual object category recognition for rgb-d scene labeling[END_REF], [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF]. In the first case, bottom-up saliency maps are based on color images [START_REF] Erdem | Visual saliency estimation by nonlinearly integrating features using region covariances[END_REF], [START_REF] Zhang | Saliency detection: a boolean map approach[END_REF] or RGB-D data [START_REF] Peng | Rgbd salient object detection: A benchmark and algorithms[END_REF], and highlight stimuli that are intrinsically salient in their context. As interesting elements are not always intrinsically salient, some approach suggest to add top-down modulation in order to further enhance elements related to a given task [START_REF] Frintrop | VOCUS: A visual attention system for object detection and goal-directed search[END_REF], [START_REF] Hamker | The emergence of attention by population-based inference and its role in distributed processing and cognitive control of vision[END_REF]. In the second case, indoor object segmentation based on depth data usually rely on finding planar surfaces and objects lying on it. Those methods can accurately detect objects on tables or floor, but are limited by the sensor quality, strong geometrical considerations (size or distance to the objects) and require more computation time than bottom-up saliency maps.

Palaiseau, France celine.craye@thalesgroup.com So far, these visual attention approaches are mostly used as black boxes and are not learned (although sometimes refined) directly during the exploration. Machine learning approaches such as deep-learning are also trained offline, often in a fully supervised setup, and thus may not be adapted to different or dynamic environments. A more adaptive approach would be to learn and adapt to an environment directly on a robot, in an incremental and autonomous way. Learning would then specialize for a specific environment, but would be constantly improved and remain flexible to any change or novelty. Such learning would need an exploration strategy to gather relevant training samples.

Active exploration by robots can be done in many different ways depending on the task and available hardware. On mobile robots, predefined path plans [START_REF] Meger | Curious george: An attentive semantic robot[END_REF], navigation graphs [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF], or frontier-based explorations [START_REF] Jebari | Combined vision and frontier-based exploration strategies for semantic mapping[END_REF] can drive the robot's exploration. When equipped with a camera with zooming capabilities, [START_REF] Björkman | Active 3d scene segmentation and detection of unknown objects[END_REF], [START_REF] Kragic | Object search and localization for an indoor mobile robot[END_REF], [START_REF] Minut | A reinforcement learning model of selective visual attention[END_REF], the perception of the robot can be further improved by moving the camera to relevant portions of the visual field. In the context of task learning, Oudeyer et al. proposed an exploration strategy based on intrinsic motivation such as learning progress [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] and competence progress [START_REF] Baranès | R-iac: Robust intrinsically motivated exploration and active learning. Autonomous Mental Development[END_REF]. In this approach, the robot is not trying to cover the whole environment or to improve its perception for a specific task, but is focusing on areas where learning is the most efficient. This way, the environment is not uniformly covered, but is learned more efficiently based on the current state of knowledge rather than on extrinsic criteria.

In this paper, we propose two contributions. First, we present an algorithm that can learn object-based visual saliency in an incremental and autonomous manner, directly within a robot's environment. Second, we suggest an exploration strategy driven by learning progress to allow the robot to autonomously learn about its environment faster and better. For that, we use an object segmentation algorithm from depth data that provides reliable and accurate, but partial and computationally expensive estimation of the saliency. From its result, we train a classifier that learns the visual RGB aspect of the discovered salient elements. Thus, the classifier is able to estimate the saliency of each pixel based on the RGB component only and reconstruct a saliency map faster and in less restrictive conditions than object segmentation does. It is for example capable of predicting object existence in areas where depth information is not available. We then use an intrinsically motivated exploration strategy based on learning progress to speed up and improve the learning quality. This paper extends previous work [START_REF] Craye | Exploration strategies for incremental learning of object-based visual saliency[END_REF] in terms of segmentation algorithm and exploration strategy. In [START_REF] Craye | Exploration strategies for incremental learning of object-based visual saliency[END_REF], the segmentation was obtained from depth data extracted only in Fig. 1. General architecture of our system along with reference section for each block a small portion of the input image (the fovea). We here use another segmentation technique that is applied on the entire image. Moreover, in previous work, the exploration strategy was used to select the position of the fovea in each input image, without considering the displacement of the robot in its environment. We now go one step further by selecting the next displacement of a mobile robot in its environment instead of the focus point in the current image.

II. PROPOSED APPROACH

Figure 1 presents the general architecture of our system along with the corresponding section for each block. In a learning stage, the system learns the visual (RGB) aspect of salient elements within their context using a depthbased object detector as a supervision signal. We use an adapted version of the Intelligent Adaptive Curiosity (IAC) [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] to drive the exploration of the environment and make saliency learning faster and more efficient. Once exploration and learning is finished, we exploit the model to generate environment specific saliency maps using only the RGB image. The next two sections describe the overall method by describing the incremental saliency learning approach and the exploration strategy based on IAC.

A. Saliency learning

We define salient elements as being objects of the environment that are lying on planar surfaces (typically tables or floor), with a range size between 10 and 150 centimeters. The saliency learning is made possible by the interaction of three modules described below.

1) A feature extraction applied to the RGB image that encodes the color of each pixel and its neighborhood at different scales, averaged using superpixels. The method is more extensively explained in [START_REF] Craye | Exploration strategies for incremental learning of object-based visual saliency[END_REF]. The feature extractor is applied on the whole input frame, and returns a 39 dimensions feature vector for each pixel. The feature extraction is much faster to compute than the geometric segmentation, and is available everywhere. 2) A segmentation algorithm that detects objects lying on planar surfaces is applied on the depth map. We use an adapted version of the method proposed by [START_REF] Caron | Neural network fusion of color, depth and location for object instance recognition on a mobile robot[END_REF]. The depth map is turned into a point cloud for geometrical calculation. First, the main plane of the input point cloud, supposed to be the floor plane is detected and labeled as not salient. Then, big planes that are perpendicular to the ground are filtered and also labeled as not salient, as they are likely to be walls. Remaining points are then grouped by a kd tree clustering and represent object candidates. To avoid false positives as much as possible, we categorize as unknown clusters that are either too small, too larger, or having a contact with the border of the frame. Remaining clusters are considered as salient objects. The point cloud is then converted back to a segmentation mask of the image frame based on the obtained labels (see Fig. 2, second row for illustration). 3) A classifier that is continuously updated based on the saliency labels provided by the segmentation mask and the corresponding RGB features. Each pixel of the input image is associated with a feature vector from the feature extractor and a label from the segmentation. We train our classifier with the feature-label samples in order to predict the saliency of a given pixel. We only train the classifier with data that is classified as salient or not salient. Pixels classified as unknown are not considered. The classifier used in our implementation is a random forest, which is not designed for online training. No available version [START_REF] Balaji Lakshminarayanan | Mondrian forests: Efficient online random forests[END_REF], [START_REF] Saffari | On-line random forests[END_REF] of online random forest was satisfying in terms of speed and performance, so we adapted the offline version to make re-training fast enough. To this end, a limited number of samples from each new frame is used to update the classifier: for each new frame, we randomly pick up 700 pixels and add the corresponding data to a dataset cumulated from the beginning of the sequence. Then, we update the classifier by only re-training a small fraction of the forest at a time: we randomly select 4 trees among the 30 in the forest, and we retrain those tree with 70% of the dataset cumulated from the beginning of the sequence. Lastly we restrict the size of the cumulated dataset to 100000 samples. If the dataset exceeds this size, we randomly remove samples to meet the maximum size requirement. As a result, after each update, the classifier is able to estimate the saliency of an input based on the model trained with the previous observations, and the RGB image only. In the exploitation stage, the saliency map is constructed as follows: RGB features are extracted for each pixel of the input and sent to the classifier. For each pixel, the classifier outputs a score between 0 and 1 that estimates the saliency. The classifier score is then associated to each pixel of the input image to reconstruct the saliency map.

B. IAC-based exploration

Exploration to gather training samples has a critical impact on the learning quality and efficiency. In the scope of autonomous and lifelong learning, an exploration strategy is necessary to guide learning and focus on areas where learning is neither trivial nor impossible. Making sure that learning is still possible avoids decreasing in the learning quality because of irrelevant samples, and speeds up the overall learning rate by focusing on appropriate tasks first.

Our exploration strategy is based on an adapted version of the IAC (Intelligent Adaptive Curiosity) algorithm [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]. IAC is a method to drive a robot's actions toward situations that maximize the learning progress. This algorithm makes the agent focus on cases that are neither too easy nor too hard based on the learning state, so that progress are constantly made and no time is wasted in unlearnable situations. Moreover, progress maximization has been found to be an optimal strategy to learn in a limited amount of time [START_REF] Lopes | The strategic student approach for life-long exploration and learning[END_REF]. IAC has been manly used to learn a mapping between motor commands and sensory feedbacks. To our knowledge, the algorithm has not been applied so far for pure vision problem, or used for robotics navigation.

IAC's key components are:

• a learner that learns to make predictions about the environment. In our case, the learner is the random forest module described in section II • a separation of the exploration space into regions. In our case, the regions are divided based on positions and orientations of the robot. • a meta-learner monitoring the learning rate in each region, and estimating the progress in each of them.

IAC suggests that the next action to be taken by the robot should be randomly picked up in the region having the highest learning progress. We use a similar approach to drive the robot in its environment: the environment (typically the room to explore) is arbitrarily divided into regions. After evaluating the progress in each of them, a position and orientation of the robot is randomly selected in the most progressing region. The robot next moves to this new position, obtains a new RGB-D input there, updates its saliency model and updates the progress estimation.

In our approach, the meta-learner stores for each region a history of the error rate based on the differences between estimated saliency (from the learner estimation) and observed saliency (from the segmentation algorithm). When a new frame is acquired in region i after t observations in the region, the meta-learner of region i is updated by adding the current error rate Err i (t) to the history:

Err i (t) = 1 -F 1 (C(O, E)) = 1 - 2tp 2tp + f p + f n (1)
where F 1 (C(O, E)) is the F 1 score of the confusion matrix C(O, E) based on the observed O and estimated E saliency maps, tp, f p and f n are the true positives, false positives and false negatives, from the confusion matrix. We use the F 1 score as our error metrics for Err i , because not salient pixels are representing more than 80% of the samples, making accuracy inappropriate for error estimation. An estimation of the learning progress in region i, is obtained by a linear regression of the error rate history Err i over the last θ samples:

   Err i (t -θ)
. . .

Err i (t)    = β i (t) ×    t -θ . . . t    +    (t -θ) . . . (t)    (2)
with (t) the residual error. The learning progress LP i in region i is defined as the derivative of the learning curve (or the opposite of the error rate) in i. Therefore, we have

LP i (t) = -β i (t) (3)
In the original version of IAC, the region to be explored next is the one with highest learning progress. This strategy is optimal only if taking a specific action has a negligible cost. In the case where actions are displacements of a robot from one location to another, it is clear that exploring in the same area for a few iteration may be more efficient than constantly making long displacement to the most progressing areas. To make learning progress and action selection better suited to our case, we force exploration to stay in the same region until N frames are obtained. The exploration procedure is then as follows: In a given region, we randomly select N positions (we found 10 to be a good trade-off in our experiments), and get an associated frames for each of them. We update at each new frame the learner and meta-learner. After N frames, we select the next region i to be explored with a probability that is proportional to LP i 75% of the time, and randomly 25% of the time. We then move to this region and get N new frames from there.

III. EXPERIMENTAL RESULTS

To validate the efficiency of our approach, we used two different datasets.

The first one was collected from a Pioneer 3DX robot, with a Kinect RGB-D camera mounted at 1 meter from the ground and tilted slightly downward. We manually controlled the robot in a room containing several objects such as chairs, desks, or trash bins. We recorded a 5 minutes length video sequence in which a large number of views of the room were captured. Then, we moved the objects in the room and recorded another video so as to get a sequence for learning and one for testing. On the testing sequence, we manually labeled the salient elements for about 100 frames in order to evaluate the prediction capability of our system.

The second one is a publicly available dataset called RGB-D scenes dataset [START_REF] Lai | A largescale hierarchical multi-view rgb-d object dataset[END_REF], composed with 8 video sequences of indoor scenes of everyday-life objects on tables. Objects are labeled in bounding boxes, which is not accurate enough for our evaluation. Therefore, we use the segmentation result of 100 frames and clean them up to obtain a ground truth. Those frames are removed from the training dataset.

The experiments were run on Ubuntu 10.12 with an Intel Core i3-3240, CPU at 3.4GHz quadcore processor. To learn saliency, several configurations were tested and compared. Frames of the sequence were either presented in a chronological order, random order, or based on learning progress. Section III-A provides details about the efficiency of the resulting saliency itself, whereas section III-B describes the different strategies and their efficiency to speed up learning.

A. Saliency learning evaluation

Before presenting the incremental learning progression, we first analyze the final performance reached when enough samples are used to train the classifier. In our case, we found that about 1000 frames were enough to obtain the best possible accuracy on both datasets. However, for clarity, the evaluation in this section is done with a saliency model that was learned from the entire training sequence and evaluated on the testing sequence. We first demonstrate the benefit of using a learned saliency approach compared to the original segmentation only. Figure 2 illustrates a few frames from the sequence together with the saliency and the segmentation results.

From these samples, we observe that the segmentation algorithm is such that nothing salient can be detected further than 4 meters away. On the other hand, saliency estimation is applied on the whole image and the generalization capability of the classifier makes it possible to detect salient objects at more than four meters. Second, reflective surfaces are often hard to detect by the Kinect sensor. However, the aspect of salient reflective objects can be partially learned and fully retrieved based on the RGB data only. Third, the segmentation algorithm is very restrictive and is often not able to detect salient elements if they are in contact with a border of the image or badly captured by the Kinect. On the contrary, the saliency algorithm provides an estimate of saliency even if the object is partially cut, occluded, or captured with a poor image quality. Last, the segmentation processes input at 0.5Hz to 1.5Hz, depending on the geometrical complexity of the input. The saliency map estimation is processed at 8Hz, which makes it much more computationally efficient. Once exploration is finished, the segmentation can be disabled and saliency can be estimated at 8Hz for further processing (object localization for example).

To demonstrate the accuracy of our saliency estimation, we select three related saliency algorithms and compute the ROC curves for each method on both datasets. BMS [START_REF] Zhang | Saliency detection: a boolean map approach[END_REF] and GBVS [START_REF] Harel | Graph-based visual saliency[END_REF] are among the most accurate RGB saliency methods according to the MIT saliency benchmark [START_REF] Bylinskii | Mit saliency benchmark[END_REF]. BMS is used with parameters that highlight salient objects rather than salient fixations. In addition, we use the new version of the VOCUS2 algorithm [START_REF] Frintrop | Traditional saliency reloaded: A good old model in new shape[END_REF] along with the configuration file dedicated to the task of object detection in cluttered scenes (top-down saliency). On the robot sequence, we also evaluate the segmentation performance. Regions that are not labeled as salient or not salient are replaced by random noise between 0 and 1. Results are displayed in figure 3.

Based on the ROC evaluation, our method significantly outperforms the evaluated bottom-up and top-down techniques on both datasets, which was expected because it is trained specifically for the environment. We can also observe that most of the evaluated techniques outperforms the depth segmentation. This is because segmentation only returns a saliency result when this information is available.

As a result, a large portion of the input frames is neither salient nor not salient and therefore estimated as noise in the ROC evaluation. For the same reason, when available, the segmentation hardly ever makes mistake about saliency prediction, which explains why the true positive rate is much higher than other methods when the false positive rate is very low.

In figure 4, a comparison between the methods is presented. First, our method provides an estimate of the shape and size of the salient object, which is not the case of techniques such as GBVS that are more stimuli-based. Second, the segmentation, that is based on geometrical consideration, avoids the detection of distractors that are visually salient (windows, trees outside, red power outlet) but irrelevant for our tasks. Last, it enhances elements that are not naturally salient (mobile container, desk) but consistent with our definition.

B. Exploration strategy

We now look at the evolution of the saliency quality during incremental learning. Figure 5 first shows a qualitative example of the evolution of the saliency at a given point of view, while the sequence is used in chronological order for training the classifier. We can observe the generalization capability because even before the seat was observed (in frame 400), the classifier is already able to recognize it as a salient element, because it has already learned a partial model of the background.

To evaluate the exploration strategy proposed in Section II-B, we use the two datasets by simulating displacements of the robot. Instead of selecting a random position/orientation in a region and physically moving to this location, we associate each frame of the dataset with a position/orientation (and, as a result, a region), and we randomly pick frames in the regions selected by IAC. For the two sequences recorded in our laboratory, the positions and orientations of the robot were obtained by a SLAM algorithm. We then arbitrarily defined 16 regions based on the position and orientation of the robot in the room: the positions were segmented in 4 equal regions based on the center of the room map, and the orientations were segmented in 4 equal cluster inside each region (see Figure 8). For RGB-D scenes, we used the sequence table small 2 only. The video sequences are provided without any localization information. However, the trajectory of the acquisition sensor is such that each point of view is seen only once in the sequence. We then created 5 regions by dividing the video into five sub-sequences of equal length.

To demonstrate the benefits of exploring the environment using IAC, we compare the performance of the system when positions of the robot are selected with IAC, randomly, and following the chronological order of the sequence. In practice, selecting a position of the robot is equivalent in our case with selecting a frame from the dataset.

The performance of the system was evaluated using the evolution of the overall error rate of the system. Based on the reference frames on which a ground truth is available, we Input image Segmentation Saliency map Fig. 2. Saliency and segmentation results for a few examples. First row: the RGB image. Second row: the segmentation result, obtained from the depth input only. Grey areas of the segmentation images are pixels labeled as unavailable (dark gray), or undetermined (light gray). Third row: the saliency result, obtained from RGB features only. The classifier outputs for each pixel a fuzzy score between 0 (not salient) and 1 (salient). Note that the classifier is able to generalize so as to retrieve the saliency of data that is unavailable in the depth input. of the BMS [START_REF] Zhang | Saliency detection: a boolean map approach[END_REF] saliency maps. Results are displayed in Figure 6. As expected, the offline model is a lower bound of the error rate, and the system tends to reach this limit after a certain number of observations, whatever the exploration strategy. Our method rapidly outperforms BMS when enough observations are obtained. The chronological exploration is slower to converge than random exploration at the beginning. Last, IAC seems to be the exploration strategy that provides the faster learning rate. Note that the difference between random and IAC-based exploration is less significant on the robot sequence, but the error variance of IAC is much lower than the one for random exploration.

To get a better insight of the selection of the region and the proportion of time spend in each region, we represent in Figure 7 the proportion of time spent in each region at each new observation. The curves were obtained from the table small 2 sequence, each curve representing the percentage of time spent in a given region. As opposed to random selection, where the time spend in each region is almost constant, IAC has a clear evolution of the most visited region. Starting with regions where error rate is decreasing fast (i.e. where progress is high), it then spends much more time on regions where progress is slower.

IV. CONCLUSION AND FUTURE WORK

In this paper, we have presented an approach to incrementally learn visual saliency, using an exploration strategy based on learning progress. Using a reliable, yet restrictive and slow segmentation of salient elements, we construct a model of saliency that is much faster to compute, and getting better as new observations are obtained. This method shows good performance in the case of detecting objects lying on planar surfaces, as a clear geometrical definition can describe these situations. In this case, our results outperform state-of-the-art saliency approaches. To allow the robot to autonomously discover and learn about its environment, we use an adapted version of IAC: we divide the exploration space into regions, and drive our exploration to spend more time on regions where progress is the highest. This type of exploration strategy makes learning faster and better than it would be with random exploration and enables lifelong learning.

In a future work, we would like to investigate further the exploration strategy based on learning progress. Indeed, the time spent by the robot for displacement from one region to another one is not taken into account and an optimal trade-off between displacement time and learning time should be found. We could also apply this framework with other definitions of saliency. Instead of a generic object segmentation, we could for example use objects detectors and specialize our saliency to find those objects within their environments.

Fig. 3 .Fig. 4 .

 34 Fig. 3. ROC curves of several approaches on the RGB-D scenes dataset (a) and on the robot sequence (b).

compare the estimated saliency map for all of these frames with the available ground truth. Using the formula provided by equation 1 for each frame and taking the average error, we obtain the overall error rate. Note that the overall error rate differs from the region error rate used to determine the learning progress in Section II-B. The overall error rate is an extrinsic metrics used to evaluate the performance of the system, whereas the region error rate is intrinsic (based on segmentation rather than ground truth) and is used to get an estimate of the error in each region. At each new observation, we evaluate the current overall error rate and compare its evolution in time for each exploration strategy.

For IAC and random frame selection, we run the experiment 10 times and consider the average and variance over those sequences. As a reference, we also display the overall error rate obtained with a model trained offline with all samples of the entire sequences, as well as the performance


D M Rojas Castro

A Revel

M Ménard

Document Image Analysis by a Mobile Robot for Autonomous Indoor Navigation

Keywords: floor plan analysis, symbol recognition, robot navigation, document information merging. I

This paper proposes the analysis of a paper-based floor plan of a building for reference purposes in the navigation task of a mobile robot. The novelty of this approach lies in the combination of this information extracted from the document image in real time just prior to real world exploration with dynamic visual perception. An artificial neural system integrates both sets of information by having the robot recognize navigation signs from the floor plan and then look for them in the expected sequence in the environment. Thereafter, while determining and executing appropriate movements as a response to the environment, the robot links and learns to associate the undertaken movements to the respective signs. The analysis of the floor plan involves a detailed process permitting the robot to extract the required information in real time. This process is elaborated in this paper, along with experiments with the robot using the proposed architecture in a simple navigation scenario, hence proving the feasibility of the approach.

INTRODUCTION

Digital floor plans of buildings (such as hospitals, schools, residential complexes or factories) are typically not as readily available as digital maps of entire cities, countries etc. Hence, when trying to navigate an unknown building, one has to rely on floor plans available in physical form at the entrance of the building or on paper to achieve one's final destination. Since such a floor plan provides one of the fastest way to access comprehensive information about the inside of the building, it can be used as a priori information for reference purposes in the navigation task of a mobile robot.

This paper proposes a document analysis-based system allowing a robot to emulate the cognition process of a human brain when navigating an unknown building by "reading" a map or a floor plan using its camera and "remembering" a sequence of navigation signs. These memorized signs will serve as reference landmarks in the real environment allowing it to navigate towards its final destination.

Navigation strategies allowing mobile robots to autonomously travel from a starting point to a goal are extremely diverse [START_REF] Siegwart | Introduction to autonomous mobile robots[END_REF]. Control architectures are the essential component of such navigation. They define the capacities of the robot to plan a path trajectory, to undertake autonomous decision-making and to execute the appropriate reaction according to the perceived environment information.

Therefore, the underlying idea of this work is to present the conception of such system, based on an approach at the same time top down (use of an a priori model of the environment represented in a paper-based document) and bottom up (reactive navigation based on the perceived visual information of the real environment), that allows the robot to achieve its goal while overcoming the challenges presented by an environment prone to change. Hence, a document analysisbased control architecture which integrates both à priori and dynamic visual perception has been conceived. This integration is brought about using a neural network system. Our approach addresses the control in a manner that is significantly different from existing robot navigation systems in two ways. Firstly, the a priori global knowledge of the environment is gathered by the robot in real time from a paperbased floor plan of the building and it is used to only corroborate the dynamic visual information perceived during navigation instead of directly controlling the actions of the robot as found in the literature [START_REF] Qureshi | The cognitive controller: a hybrid, deliberative/reactive control architecture for autonomous robots[END_REF]. Secondly, instead of using a complete motion path, this architecture makes use of navigation signs (and their expected sequence in the route) extracted from the said document, the associated directional implications of which may be learnt as a consequence of a stimulus-response model during navigation. As a result of this perceptual combination in real time, the robot is capable of performing autonomous navigation, memorizing and learning sensory-motor associations, decision-making and rapidly responding in dynamic environments.

The reminder of this paper is organized as follows. Section II introduces the document analysis based control architecture and describes the integration of the two types of perceptual information mentioned above using a neural network. This approach is the main contribution of this paper. Section III describes in detail the specific image processing steps allowing the extraction of the navigation signs from a document in real time. Results of experiments implementing the system in a humanoid robot (NAO) are described in section IV, and section V presents conclusions and potential future directions.

II. DOCUMENT ANALYSIS BASED NAVIGATION SYSTEM

A. Context

The global knowledge of the world is represented by a paperbased floor plan that is placed in front of the robot's camera just once, before the navigation activity starts (Fig. 1a). It contains the important information to define a potential navigation trajectory that might eventually be modified depending on the unforeseen situations presented in the environment (e.g. obstruction of the path by a collapsed wall). In this work, this information is represented by navigation signs used as reference points that are expected to be seen by the robot in the real world navigation (black circles with specific white patterns drawn within, associated with left and right directions as shown in Fig 1b). By means of computer vision methods, the robot "reads" the floor plan, generates an optimal plan to reach the goal, recognizes the navigation signs within the path, extracts and "memorizes" the sequence of signs arranged from the closest point to the furthest with respect to the starting point (see section (III)).

The robot then integrates this extracted sequence of signs into the neural system Fig. 2 and combines it with the visual perception of the environment in real-time (Fig 1c). The neural system uses a cognitive mechanism performing recognition of a particular sign and allows learning of its association to the realization of a particular action during navigation. As it encounters the signs, it may or may not know the meaning of each of them in terms of the instruction they represent with respect to way finding. The architecture is designed in such a way that if the sign is unknown, the robot explores its environment as a 'reflex' to find the next sign in the sequence from its memory. Once this next sign is discovered, the robot is able to denote a "meaning" ("turn right" or "turn left") to the previous sign, since this sign was supposed to lead the robot to the newly discovered sign. This meaning is then learnt for future reference. In this way, the robot continues to learn the meaning of new signs after deducing them. In following encounters with the same signs, it just executes the associated action based on its learning.

B. Overview of the system

The overall architecture proposed in this paper integrates the pre-exploration floor plan analysis into an organized neural structure (Fig. 2). It is based on a perception-action mechanism that constantly evolves because of the dynamic interaction between the robot and its environment [START_REF] Gaussier | PerAc: A neural architecture to control artificial animals[END_REF].

The analysis of the floor plan in real time, represented by the box in the top half of Fig. 2, undertakes a thorough process permitting the robot to extract the relevant information for its integration into the system. It follows a process similar to the systems found in the literature [START_REF] Ahmed | Improved automatic analysis of architectural floor plans[END_REF]. It consists of an information segmentation process which identifies and separates different types of information; followed by structural analysis where the information is extracted (walls and navigation signs separately); and finally a semantic analysis where the information is extracted (walls and navigation signs separately); and finally a semantic analysis allowing the extraction of the sign sequence based on the computation of the path and the information of the signs.

The neural system, shown in the lower half of Fig. 2, is composed of three different layers. The first layer of the system is in charge of merging the information coming from the long term memory and the real-time visual perception of the environment to navigate the robot towards its final destination. This layer is the major contribution of this paper and is explained in the next subsection. The other two layers, which are omitted in this paper for the sake of brevity in favor of detailing the more relevant contribution (information merging and document analysis processing), use a reflex mechanism that controls directly the robot's action based on the perceived information of the environment. Hence, the second layer determines the direction to be taken by the robot (left or right) by using the next expected sign from the pre-captured sequence, while the third layer directs an approach towards the sign by keeping it in the center of the robot's vision when the robot is far from it. More information about these two layers can be found in another paper by the same authors [START_REF] Rojas-Castro | A robust neural robot navigation using a combinationof deliberative and reactive control architectures[END_REF]. The whole system works in parallel and a 'competitive mechanism' allows to decide on the best behavior (among the layers) for controlling the robot according to the stimulus received. This is possible because the neural interconnection is done by either excitatory or inhibitory connections allowing or preventing the activation of neurons respectively.

C. Memory and Visual Perception Integration (first layer)

In order to reach its final destination, the robot needs to be able to use the information extracted from the floor plan and compare it to what it sees during real world exploration. Therefore, it has to be able to combine both sources of information and execute some actions accordingly. To that end, a system composed of artificial neural networks is implemented. The network comprises of interconnected artificial nodes known as "neurons" which compute their output values from one or more inputs received, akin to a biological neural network in the nervous system of an 1b 1c 1a 1a 1a organism. These computational models have properties such as associative memorization, learning and parallel multiinformation processing. Their interconnectivity is such that the whole system is capable of self-adapting with regard to its inputs and thus develop news behaviors.

Fig 3 shows the memory and visual perception integration layer as constructed with neural networks. It is composed of a number of interconnected groups of neurons. The first five groups correspond to the sensory perception and comprise of a number of neurons greater than the number of signs that can be recognized by the robot, each sign being represented by a single neuron. Thus, should the environment change with a different set of signs, the system will be able to represent them by using new neurons. The output direction group is composed of two neurons corresponding to the output motor actions (left and right movements respectively) and the reset group is composed of only one neuron. The flow of information through this unit and that represented in Fig. 3

is explained below.

There are two sources of information: the sign sequence extracted from the floor plan (long term memory) and the signs recognized by the robot in the environment during navigation (visual perception). Each one is computed in real time and stored in the "sign sequenced group" and the "sign detection group" respectively. Thereafter, a third neural group "sign merged detector group" combines both sets of perceptual information. Thus, when exploring the environment two scenarios are possible: the next expected sign in the sequence is detected, or not. In the former case, if the sign has already been associated to a particular movement, the robot executes the related movement i.e. turn left or right directly. On the other hand, if the sign has not been learnt previously, the robot uses the position of the next expected sign with respect to the current one to associate a direction instruction to this current sign. Therefore, it is necessary to keep the current sign in memory for a short while the next one is being looked for. This information is memorized in the "short term memory group" and is deleted by the "reset group" once the next sign is found and the movement is performed. The generation of the said association takes place in the "winner-takes-all (WTA) group" which links the information of the recognized sign and the reflex movement performed in the second layer (Direction Determination (DD)). This group also allows the learning of such associations by activating and sending a reinforcement signal. Finally the information of the action associated to the recognized sign is stored in the "output direction group" and sent to the motor output.

III. FLOOR PLAN ANALYSIS

When working in real time situations, reading and analyzing a document from an image acquired from a mobile device such that of a robot is a challenging task. It not only has to be accurate but it also has to be fast so that the robot can react accordingly. Image acquisition of documents under constrained and dynamic conditions may cause not only degradation but also distortions on the captured image itself due to unpredictable orientation, distance, lighting conditions etc. Different scenarios might be possible. For instance in the case of this work, since there is no use of a planar support when acquiring the image, the map might not be completely or at all within the image frame. It might also be too small or suffer from perspective distortion if the image captured is not orthogonal to the plane of the camera (Fig. 4). All these circumstances make it hard to properly identify and retrieve the content of the map. Therefore, a set of processes after image acquisition has been implemented (top half of Fig. 2).

Usually maps and floor plans contain building elements like walls, doors, windows, etc. and textual information such as the name of the room or the area. The focus of this work is to read a modest floor plan in the simplest way in order to prove the concept of robot navigation as explained above. Therefore, the representation of the doors in the design of the floor plans is done by blank gaps and only the walls (external and internal) as well as the navigation signs are here considered (Fig 4 ).

A. Information segmentation

As the outcome of the segmentation process directly affects the performance of further processing, it is important to find a suitable image pre-processing algorithm that allows removal of any noise and help to simplify the representation of the image for better analysis.

A global thresholding has been applied to the images in this work, as it has proven to be the simplest of image segmentation methods while providing a fast and convenient way to perform a good result when working in real time situations [START_REF] Singh | A study of image segmentation algorithms for different types of images[END_REF]. However, non-uniform illumination of the document may cause inaccuracies or complete failure of segmentation. In order to overcome these effects, a black top-hat morphological operator with a 7*7 square mask is first applied on the grayscale image. This process consists of removing noise and small objects from the image that are not relevant to the analysis of the image. Hence, it improves the clarity of the image outcome (see Fig. 5) and consequently, it makes possible the identification of relevant regions (map) and objects of interest (navigation signs and walls) in the image.

The identification and then extraction of the map from the image is made possible using analysis of the connected components. This process together with the application of some filters based on the size of the connected regions, permits rejection of images likely to contain irrelevant components outside the convex hull (Fig. 5b) or with missing information like an incomplete map boundary (i.e. user's fingers holding the map as in Fig. 5c or map outside the camera frame as in Fig. 5d and5e). Hence, only the images containing the complete and visible map are selected (Fig. 5a and5f).

Several methods have been proposed for image perspective correction. Some of them aim to calculate the horizontal and vertical vanishing point based on the horizontal lines of the text or the stroke boundaries [7] [8]. Others, instead, aim to find the corners of the quadrilateral formed by the boundary between the background and the document [9] [START_REF] Rodríguez-Piñeiro | A new method for perspective correction of document images[END_REF]. The latter method is more suitable for the type of document used in this work. Therefore, the method proposed in this work consists of first creating a bounding box around the image (top half of Fig. 6), then retrieving the corners coordinates (X,Y) of the bounding box (rectangle) formed to use them as reference points to find the coordinates (x, y) of the corners of the map contour. By calculating the minimum distance between the coordinates (X, Y) of each corner of the bounding box and each vector point of the map contour, the closest vector point to each corner of the bounding box is found. This gives the corners (x, y) of the map contour. The perspective correction is then performed by using a 3x3 transformation matrix calculated with the coordinates of quadrilateral vertices of the map and those of the destination image (bottom half of Fig. 6).

The navigation signs are represented in this work by circleshape symbols containing specific patterns that allow them to be differentiated from one another. Hence, in order to distinguish them from the floor plan, sign detection is achieved using a feature based approach by using the Hough circles transform [START_REF] Ballard | Generalizing the Hough transform to detect arbitrary shapes[END_REF] (white circles around the signs in bottom half of Fig. 6).

B. Structural Analysis

The information obtained in the previous step is used in order to extract each sign from the floor plan. By considering the parameters referring to the center position and radius of the circles (denoting signs) obtained previously (ycenter, xcenter, r), it is possible to subtract the information within each circle from the floor plan (Fig. 7a) and copy each of the signs into a new image. As a result, a 'walls image' composed of only internal and external walls is produced (Fig. 7b), as well as a set of navigation sign images each consisting of a unique sign.

C. Semantic Analysis

In order to compute the correct sequence of signs, two different process are simultaneously applied to the resulting images from the last step. In the first one, the robot recognizes the signs images. There are two notable methods available for the recognition of signs in maps or paper-based documents using image processing: keypoints based [START_REF] Rusinol | Symbol Spotting in Digital Libraries: Focused Retrieval over Graphic rich Document Collections[END_REF] [13] or template matching based [START_REF] Weber | Symbol spotting for technical documents: Ancient template matching approach[END_REF]. The former aims to extract some points of interest (generally the corners) and describe them using some radiometric features. The latter aims to find small parts of an image that match a template image from existing database. This latter simpler approach was used in this work since a database of the signs images was already available and could easily be recognized by the robot in the environment.

Simultaneously, using the other process, the robot is able to compute a navigation path to reach the final destination in an environment with obstacles (walls on the map). The robot uses a neural network implementation of the resistive grid technique [START_REF] Bugmann | Route finding by neural nets[END_REF]. The image of the walls is used here as a resistive grid where each pixel in the image is represented by a neuron and is connected to its closest neighbors. Each neuron has an activation value which is calculated based on the distance between it and the neuron representing the final destination. This latter having the highest value "1" and those representing the walls the minimum "0". The path is computed by continuously searching among neighboring neurons for the one with the highest activation value. This continues until the final destination is found (Fig. 7c).

Finally the sequence of the navigation signs is calculated by using the position of the recognized signs with respect to the generated path, from the starting point to the final destination. To achieve this, certain regular intervals of distance, 'checkpoints' (in terms of x-y coordinates), are first demarcated along the path. Beginning at the starting point of the extracted path, for each checkpoint, the distance to all visible signs is calculated. The sign located at the shortest distance for each checkpoint is stored in a special array. This is done for all checkpoints until the final destination. As a result, the array comprises of all the navigation signs visible along, and closest to the path at each checkpoint. It is then possible to finally extract the right sequence of signs that the robot will encounter on its way (Fig. 7d).

IV. EXPERIMENTS IN REAL ENVIRONMENT

The complete neural architecture was built, implemented in the humanoid robot NAO and tested in a 2.25m x 4.5m space, configured such that it represented a simplified layout of a floor of a building composed of a room delimited by walls (here tables) and a corridor between them. The intended navigation path was free of dynamic obstacles and only the walls were considered as the static obstacles to be avoided. The NAO robot can detect a sign using its camera (640*480 pixels) if it is within a range of approximately 0.2m to 1.5m away from it. Therefore, the navigation signs were placed in the environment within this range so as to always be readable.

Before exploration, the robot was shown a floor plan of the test environment so that it could process the image and compute the best way to accomplish the task. By following the steps of floor plan analysis explained in section III, the robot was able to compute the path leading towards the final destination and thus extract the sign sequence :{ A, B, A, B} as shown in Fig. 8. When the map was not placed correctly within the frame of the camera, the robot was able to warn the user to move it. Otherwise, when it was correctly positioned, the average computation time to extract the information was of around 5s.

Thereafter, the robot was placed at the entrance of the test environment from which it could distinctly see the first expected sign, A. While navigating, the following behaviors emerged allowing the robot to successfully reach its goal: • When a sign was detected, it was able to compare it to the corresponding sign in the extracted sequence from the memory; • When the comparison resulted in a positive result i.e. the detected sign was indeed the expected sign, and the meaning of the sign was yet unknown, it was able to perform reflex movements by rotating in one place to search for the next sign, figure out the associated direction and then learn it; • When the comparison gave a negative result i.e. the detected sign did not match the expected sign, the robot ignored the detected sign and continued reflex movements to locate the correct sign; • When the expected sign was not in its visual field, it was able to perform a reflex behavior to search for it; • When a sign was faraway, it was able to get closer; • When a sign appeared again, it was able to recall the learning and perform the associated movement.

V. CONCLUSIONS

In this paper, a robot uses a chain of image processing techniques for the analysis of a floor plan in real time. Relevant information is extracted and later compared with the visual information perceived during real world navigation. Such an approach at the same time top-down and bottom-up is merged into a neural system allowing the robot to reach its goal faster while still adapting itself to an environment prone to change. Different behaviors emerging from the interaction between the robot and the environment prove the advantages of a neural approach emulating human behavior.

Experimental results have shown that the use of a floor plan prior to navigation can facilitate and hasten the task of reaching a specific destination inside a building for a mobile robot.

An eventual scenario could be one where the expected signs are no longer available or visible inside the building. In that case, the robot will be able to generate an updated map according to the information assimilated from the environment and still achieve its final destination.

Figure 1 .Figure 2 .

 12 Figure 1. (1a.) Nao robot reading the map. (1b) Generated path by following the navigation signs. (1c) Real time view of the environment

Figure 3 .Figure 4 .

 34 Figure 3. Combination between document and visual information

Figure 5 .

 5 Figure 5. Binary Image by thresholding

Research work supported by European regional development Funds (Contract 35053) and the Poitou Charente Region