Fabio Tanturri
email: tanturri@math.uni-sb.de

DEGENERACY LOCI OF TWISTED DIFFERENTIAL FORMS AND LINEAR LINE COMPLEXES

Keywords: Mathematics Subject Classification. 14C05; 14J10, 14N15 degeneracy loci, Hilbert scheme, skew-symmetric matrices, linear complexes, differential forms

We prove that the Hilbert scheme of degeneracy loci of pairs of global sections of Ω P n-1 (2), the twisted cotangent bundle on P n-1 , is unirational and dominated by the Grassmannian of lines in the projective space of skew-symmetric forms over a vector space of dimension n. We provide a constructive method to find the fibers of the dominant map. In classical terminology, this amounts to giving a method to realize all the pencils of linear line complexes having a prescribed set of centers. In particular, we show that the previous map is birational when n = 4.

Introduction

Degeneracy loci of morphisms of the form φ : O m P n-1 / / Ω P n-1 (2) arise naturally in algebraic geometry and have been extensively studied. In classical terminology, a general degeneracy locus of this form is the set of centers of complexes belonging to a general linear system of dimension m -1 of linear line complexes in P n-1 , i.e. hyperplane sections of the Grassmannian Gr(1, P n-1 ) of lines in P n-1 = P(V ) embedded in P(Λ 2 V ) via the Plücker map (cfr. Sect. 2.2). This nice geometric interpretation, together with the fact that many classical algebraic varieties arise this way, motivated the interest of many classical algebraic geometers as Castelnuovo, Fano, and Palatini. Nonetheless, these loci have been recently studied by several authors, for example Chang and Ottaviani. A more detailed historical account and a few classical examples can be found, for instance, in [START_REF] Bazan | On the construction of some Buchsbaum varieties and the Hilbert scheme of elliptic scrolls in P 5[END_REF][START_REF] Faenzi | Skew-symmetric matrices and Palatini scrolls[END_REF].

In general, in order to parameterize the possible degeneracy loci X φ as φ varies, it is useful to take a modern approach and introduce H as the union of the irreducible components, in the Hilbert scheme of subschemes of P n-1 , containing X φ for general φ's.

Let P(V ) be the projectivization of an n-dimensional vector space V . Relying on a nice interpretation due to Ottaviani ([Ott92, §3.2]), we can identify a morphism of the form above with a skew-symmetric matrix of linear forms in m variables, or with an m-tuple of elements in Λ 2 V ; the natural GL m -action does not modify its degeneracy locus, so we get the rational map

(1) ρ : Gr(m, Λ 2 V ) / / _ _ _ H sending φ to X φ .
The behavior of the map ρ is fully understood for most of the values of (m, n). In [Tan] it is shown that it is birational whenever 4 ≤ m < n -1 or (m, n) = (3, 5); if (m, n) = (3, 6), ρ is dominant but 4 : 1, while for m = 3 and n > 6 it is generically injective but not dominant. Partial results in this line of thought were already provided in [START_REF] Bazan | On the construction of some Buchsbaum varieties and the Hilbert scheme of elliptic scrolls in P 5[END_REF][START_REF] Fania | On the Hilbert scheme of Palatini threefolds[END_REF][START_REF] Faenzi | Skew-symmetric matrices and Palatini scrolls[END_REF].

We are interested here in the case m = 2. Our aim is to give a description of the behavior of the map ρ, as n varies. The case n = 6 has already been tackled in [START_REF] Bazan | On the construction of some Buchsbaum varieties and the Hilbert scheme of elliptic scrolls in P 5[END_REF], where ρ was proved to be dominant and its fibers were described and shown to be two-dimensional. Our main result is the following Theorem. Let n ∈ N such that n ≥ 4, V be a vector space of dimension n and let

ρ : Gr(2, Λ 2 V ) / / _ _ _ H
be the rational morphism introduced in (1), sending the class of a morphism φ :

O 2 P(V )
/ / Ω P(V ) (2) to its degeneracy locus X φ , considered as a point in the Hilbert scheme. Then ρ is dominant. Moreover i. if n is even, the general element of H is the union of n 2 lines spanning the whole P(V ). The general fiber is the Grassmannian Gr(1, σ) of lines lying on a suitable ( n-2

2 )-dimensional projective space σ. In particular, ρ is birational for n = 4; ii. if n is odd, the general element of H is the image in P(V ) of a map P 1 [f1:...:fn] / / P(V ), where f 1 , . . . , f n are forms of degree n-1 2 spanning the whole vector space

H 0 (P 1 , O P 1 ( n-1 2 )). The general fiber of ρ has dimension n 2 -3n 2 .
When n is even, the general X φ is union of n 2 lines. A crucial step in proving the first part of the statement is the reinterpretation of a morphism φ as a pencil φ of linear line complexes. The n 2 lines are the set of centers of the linear line complexes belonging to φ ; using a similar approach to the one adopted in [START_REF] Bazan | On the construction of some Buchsbaum varieties and the Hilbert scheme of elliptic scrolls in P 5[END_REF], we are able to generalize the existing result to all even values of n.

When n is odd, the general X φ is a rational curve, image of the map [f 1 : . . . : f n ] as in the statement. A priori, the forms f 1 , . . . , f n have to be the Pfaffians of order n -1 of a linear n × n skew-symmetric matrix, but we will show that a general n-tuple of forms of degree n-1 2 arises as such. As pointed out later, an explicit construction of the preimages of a given degeneracy locus is possible. For the special case n = 4, this amounts to the following: it is possible to construct the unique pencil of linear line complexes having two given skew lines as set of centers.

These results complete the general picture, describing the behavior of the map ρ for arbitrary values of m. For m > 3, such description had already been given in [Tan]; the above theorem, together with the previous contributions, shows that the unique pairs (m, n) with 2 ≤ m < n -1 for which ρ is not generically injective are (2, n) with n ≥ 5 and (3, 6), while the unique pairs for which ρ is not dominant are (3, n) with n ≥ 7.

We remark that the results contained in this paper hold over non-necessarily algebraically closed fields of arbitrary characteristic.

Notation and preliminaries

2.1. Notation and geometric interpretation. Let k be any field and let n ∈ N such that n ≥ 4. We will denote by U, V two k-vector spaces of dimensions 2, n respectively; by P(U ) and P(V ) we will mean the projective spaces of their 1quotients, i.e. H 0 (P(U ), O P(U ) (1)) ∼ = U . We set {y 0 , y 1 } and {x 0 , . . . , x n-1 } to be the bases of U and V respectively.

Let Ω P(V ) be the rank n -1 vector bundle of differential forms on P(V ) and let φ : U ⊗ O P(V ) → Ω P(V ) (2) be a general morphism. We can define X = X φ to be the degeneracy locus associated to φ, i.e. the scheme cut out by the maximal minors of the matrix locally representing φ.

A different interpretation of X, due to Ottaviani [Ott92, §3.2], allows φ to be interpreted as an

(n × n) skew-symmetric matrix N φ = y 0 N 0 + y 1 N 1 of linear forms in y 0 , y 1 . A point p in V represents a point in X φ if and only if p ∈ ker(b 0 N 0 +b 1 N 1 )
for some (b 0 , b 1 ) = (0, 0). Thus, the geometry of X φ strongly depends on the parity of n: when n is even, it is a scroll over the hypersurface cut out by the Pfaffian of N φ , i.e. a union of n 2 lines in P(V ); when n is odd, it is the blow-up of P 1 along the set of points described by the n Pfaffians of order n -1 of N φ , which is empty for a general φ. For more details about this geometric interpretation, we refer to [START_REF] Tanturri | On the Hilbert scheme of degeneracy loci of twisted differential forms[END_REF]§2].

Linear line complexes. Another interpretation (cfr. [BM01]) of a morphism

φ : O 2 P(V ) / / Ω P(V ) (2)
is the following: let G := Gr(1, P(V )) be the Grassmannian of lines in P(V ), embedded in P(Λ 2 V ) via the Plücker map. The dual space P(Λ 2 V * ) parameterizes hyperplane sections of G, or, in classical terminology, linear line complexes in P(V ).

An element A ∈ Λ 2 V , up to constants, may be regarded as an element of P(Λ 2 V * ), hence giving rise to a linear line complex Γ. A point p ∈ P(V ) is called a center of Γ if all the lines through p belong to Γ; the space Deg(A) := P(ker(A)), called the singular space of Γ, turns out to be the set of centers of Γ.

Let ∈ G and T G the corresponding tangent space. The hyperplane V(A) in P(Λ 2 V ) contains T G, i.e. A belongs to the dual variety Ǧ, if and only if Deg(A) ⊇ .

We distinguish the following two cases:

• if n is even, then a general linear line complex Γ has no center, as a general skew-symmetric matrix of even order has maximal rank. Linear line complexes Γ corresponding to the points of Ǧ have at least a line as center and will be called special ; • if n is odd, then a general linear line complex Γ has a point as center. Γ will be said to be special if it corresponds to a point of Ǧ: in this case, the center of Γ contains (at least) a P 2 .

We can therefore interpret the degeneracy locus of a general morphism φ :

O 2 P(V ) / / Ω P(V ) (2) as Deg(N φ ) := A∈N φ Deg(A),
where the skew-symmetric matrix N φ = y 0 N 0 + y 1 N 1 is regarded as a line in P(Λ 2 V * ). Special complexes on N φ are parameterized by the intersection N φ ∩ Ǧ.

The behavior of ρ: even case

Within this section, we assume n even. We will show that the map ρ

ρ : Gr(2, Λ 2 V ) / / _ _ _ H
defined in (1) is dominant. This leads naturally to asking what is the preimage of a general point X φ in H, i.e. which lines N ⊂ P(Λ 2 V * ) have Deg(N ) = X φ . We will give a geometric description of the fibers, providing a constructive procedure to realize the elements of a general preimage. Recall that a linear line complex A ∈ P(Λ 2 V * ) is special if its center contains a P 1 . We distinguish special complexes of the first type, whose center is exactly a line, from special complexes of the second type, whose center is at least a P 3 .

A general matrix N = N φ with linear forms in k[y 0 , y 1 ] as entries has corank two in n 2 distinct points of P 1 , corresponding to the roots of Pf(N ); N has maximal rank in any other point. This means that the general line of complexes N ⊂ P(Λ 2 V * ) does not contain any special complex of the second type, and that Deg(N ) is the union of n 2 lines { 1 , . . . , n 2 }. We claim that these lines are general, in the sense that their span is the whole P(V ). It is clear that this condition is open, so it is sufficient to exhibit a matrix N satisfying it. We will do more, showing constructively in Proposition 2 that any set of lines { 1 , . . . , n 2 } spanning P(V ) is Deg(N ) for some pencil of complexes N . Let us examine the Gauss map

ζ : Ǧ / / _ _ _ G
which sends a special complex of the first type A to the point in G corresponding to the line Deg(A). Fixing a line ∈ G, the fiber ζ -1 ( ) is a linear space. A complex A ∈ G has as center if and only if the hyperplane V(A) in P(Λ 2 V ) contains T G, so the space of such A's has dimension

dim(ζ -1 ( )) = dim(P(Λ 2 V )) -dim(G) -1 = 1 2 (n -1)(n -4).
We observe that, given a linear space S of dimension n-3 in P(V ), it is uniquely determined a complex H, up to constants, such that the center of H is S. Indeed, all the lines in P(V ) intersecting S have to be contained in Γ = V(H) ∩ G, but this is a linear condition in the Plücker coordinates.

Fixing a set of lines { 1 , . . . , n 2 } spanning P(V ), for any j such that 1 ≤ j ≤ n 2 we will denote by H j ∈ P(Λ 2 V * ) the unique complex having < i > i =j as center. We will denote by F i the ( n 2 -2)-dimensional linear span <H j > j =i ⊂ P(Λ 2 V * ); a point in F i is a complex having at least i as center.

Remark 1. Let 1 , . . . , n 2 be lines spanning P(V ). Up to a projectivity of P(V ), we may assume that

∀ i, i = j / ∈{2i-2,2i-1} V(x * j ),
being V(x * j ) the hyperplane in P(V ) whose points have x j coordinate zero. With this choice, H i is the complex represented by a skew-symmetric matrix with (j, k)-th

entry    α if j + 1 = 2i = k for some i -α if k + 1 = 2i = j for some i 0 otherwise
for some α ∈ k \ {0}. The elements of ζ -1 ( i ) have zero entries in the (2i -1)-th, (2i)-th rows and columns.

Proposition 2. Let { 1 , . . . , n 2 } be a set of lines spanning P(V ). Let σ be the ( n-2

2 )-dimensional linear space <H i > 1≤i≤ n 2 ⊂ P(Λ 2 V * ). Then, for a line N ⊂ P(Λ 2 V * ), the following are equivalent: i. N ⊆ σ and N ∩ F i ∩ F j = ∅ for any 1 ≤ i < j ≤ n 2 ; ii. N does not contain special complexes of the second type and Deg(N ) = { 1 , . . . , n 2 }.

Proof. i. ⇒ ii. Let P i := ζ -1 ( i ). Let N ⊆ σ. For any j, the linear space F j has dimension n 2 -2, so any line N ⊆ σ intersects it; hence, Deg(N ) ⊇ { 1 , . . . , n 2 }. Moreover, with the choice of coordinates of Remark 1, points of σ are represented by skew-symmetric matrices with (j, k)-th entry

   α i if ∃ i such that j + 1 = 2i = k -α i if ∃ i such that k + 1 = 2i = j 0 otherwise for some sequence (α i ) in k.
It is easy to check that a point of σ is a special complex if and only if α j = 0 for some j; is special of the second type if and only if α j = α k = 0 for some j = k, i.e. if and only if it lies on

F j ∩ F k . If N ∩ F j ∩ F k = ∅ for any j < k, then N contains n
2 special complexes of the first type and no special complexes of the second type, hence the conclusion.

ii. ⇒ i. Let N be as in ii.. For any i define R i := N ∩ ζ -1 ( i ), which is non-empty by hypothesis. We have R i = R j for any i = j, as otherwise N would contain a point having < i , j > as center. We claim that N ⊆ σ, for which it is sufficient to show that R 1 and R 2 are both contained in σ.

Let us choose the coordinates as in Remark 1. For any i = j we have

N = <R i , R j >, so (2) R 1 ∪ R 2 ⊂ N = i<j <R i , R j >.
Complexes in R i ⊂ ζ -1 ( i ) have zero i-th and (i + 1)-th rows and columns, hence the entries A k,l of a complex A in <R i , R j > are zero at least when

(k, l) ∈ ({2i -1, 2i} × {2j -1, 2j}) ∪ ({2j -1, 2j} × {2i -1, 2i}) .
From (2), we deduce that any complex

A in R 1 ∪ R 2 has non-zero entries A k,l only if ∃ i such that k + 1 = 2i = l or l + 1 = 2i = k, hence it belongs to σ = <H i > 1≤i≤ n 2 . This is enough to show that N ⊆ σ. If N ∩ F i ∩ F j = ∅ for some i = j, then Deg(N ) would contain < i , j >, hence a contradiction.
Proposition 3. If m = 2, n ≥ 4 and n is even, then ρ is dominant. The general element of H is the union of n 2 lines spanning P(V ). The general fiber has dimension n-4; its general element is a general line N ⊂ P(Λ 2 V * ) lying on σ, as in Proposition 2. In particular, ρ is birational if (m, n) = (2, 4).

Proof. We can define a rational map

ξ : Gr(1, P(V )) n 2 / / _ _ _ H,
defined outside the closed subset corresponding to sets of n 2 lines non-spanning P(V ), sending n 2 lines to the corresponding point in H. It is finite and its image Im(ξ) is irreducible.

The general morphism φ has n 2 lines ( i ) spanning P(V ) as degeneracy locus, so Im(ξ) = Im(ρ). It remains to show that (3) dim(Im(ξ)) = dim H, so that Im(ρ) is the unique irreducible component of H, hence ρ is dominant. On the one hand, dim(Im(ξ)) = dim(Gr(1, P(V ))

n 2 ) = n 2 -2n. On the other hand, let Y be the union of n 2 skew lines. From

0 / / T Y / / T P(V ) Y / / N Y /P(V ) / / 0
and Euler sequence restricted to Y , we get

h 0 (N Y /P(V ) ) = n 2 (2n -1 -χ(T Y )) and h 1 (N Y /P(V ) ) = 0.
Riemann-Roch Theorem yields χ(T Y ) = 3, so equality (3) holds. Finally, fixing a point ∪ i i in H, the general elements of its preimage via ρ is a general line N ⊆ σ, as showed in Proposition 2. In particular, the space of such lines has dimension dim Gr(1, σ) = n -4.

When n = 4, σ is a line and the unique preimage is σ itself.

We remark here that the fibers of ρ can be explicitly constructed. By means of a projectivity we can send any set of n 2 general lines to the lines chosen in Remark 1; then, we just need to apply to any line lying on σ as in Proposition 2 the same projectivity backwards.

The behavior of ρ: odd case

Let n be odd from now on. The general degeneracy locus X φ is easy to describe: similarly to the case m = 3 in [Tan, §6], the elements in Im(ρ) are the images of maps (4) P(U )

[f1:...:fn] / / P(V ),

where f 1 , . . . , f n are forms of degree n-1 2 in k[y 0 , y 1 ], Pfaffians of a general n × n skew-symmetric matrix N with entries in k[y 0 , y 1 ] 1 . These forms are general, in the sense that they generate the whole vector space k[y 0 , y 1 ] n-1 2 . Lemma 4. For a general n × n skew-symmetric matrix N with entries in k[y 0 , y 1 ] 1 , its Pfaffians of order n -1 span the whole k[y 0 , y 1 ] n-1 2 .

Proof. For the (n -1) × (n -1) Pfaffians of a general N , not to span k[y 0 , y 1 ] n-1 2 is a closed condition, so it is sufficient to exhibit, for any odd k, a matrix N k not satisfying it. For this sake, we consider the k × k matrix

N k =             0 y 0 -y 0 0 y 1 -y 1 0 y 0 -y 0 0 y 1 . . . . . . 0 y 1 -y 1 0             .
If we denote by Pf i (N k ) the (k -1) × (k -1) Pfaffian obtained from N k by deleting the i-th row and column, it is easy to check that

Pf 2i+1 (N k ) = y i 0 y k-1 2 -i 1 for any 0 ≤ i ≤ k -1 2 , Pf 2i (N k ) = 0 for any 1 ≤ i ≤ k -1 2 ,
and this concludes the proof.

Remark 5. As a consequence of the previous lemma, every sequence of general forms f 1 , . . . , f n of degree n-1 2 corresponds to the sequence of Pfaffians of a suitable skew-symmetric matrix. Indeed, these forms can be expressed as linear combination of the Pfaffians of N k above, giving rise to β ∈ PGL(V ) such that the diagram

7 7 o o o o o o o o o o o o [Pf1(N k ):...:Pf k (N k )] ∼ / / V 2
commutes. This produces an automorphism of P(U ), hence a change of basis of k[y 0 , y 1 ] 1 . In terms of this new basis, N k has the desired Pfaffians f 1 , . . . , f n . Proposition 6. If m = 2, n ≥ 5 and n is odd, then ρ is dominant. The general element of H is the image in P(V ) of a map P(U ) ) -1 = n-1 2 . We can define a rational map

ξ : A (r+1)n / / _ _ _ H
sending an n-tuple of forms f 1 , . . . , f n to the image of the map (4). It is defined on the n-tuples which span the whole linear space k[y 0 , y 1 ] n-1 2 ; its image Im(ξ) is irreducible and its dimension is easily computable. Indeed, on the one hand there is a natural GL 2 -action on k[y 0 , y 1 ] 1 , acting as a change of basis on U ; this induces an action on k[y 0 , y 1 ] n-1 2 and therefore on A (r+1)n , and one can see that ξ factors through this action. On the other hand, take two points V 1 , V 2 in Im(ξ) such that V 1 = V 2 . By the commutativity of the diagram we get an automorphism of P(U ), i.e. the two maps [f 1 : . . . : f n ] and [g 1 : . . . : g n ] belong to the same class modulo GL 2 . Hence dim(Im(ξ)) = dim(A (r+1)n ) -dim(GL 2 ) = n 2 + n -8 2 .

V 1 P(U )
Since n general forms are obtained as the Pfaffians of a suitable matrix N by Remark 5, Im(ρ) = Im(ξ). If we prove that dim(Im(ξ)) = dim H, then Im(ρ) is the unique irreducible component of H, hence ρ is dominant. This can be done again as in the proof of Proposition 3.

The dimension of the fibers is finally

dim Gr(2, Λ 2 V ) -dim H = n 2 -3n 2 .
As in the even case, also for odd values of n is possible to construct explicitly the fibers of ρ. Up to a projectivity, we can assume that the degeneracy locus is the image of the Pfaffians of the matrix N k from Lemma 4. We can find by linear algebra all the skew-symmetric matrices having those as Pfaffians and apply the same projectivity backwards to get the elements of the desired fiber.

[ 2 ..

 2 f1:...:fn] / / P(V ), where f 1 , . . . , f n are forms of degree n-1 2 spanning the whole space k[y 0 , y 1 ] n-1The general fiber of ρ has dimension n 2 -3n 2 Proof. Let r = dim(k[y 0 , y 1 ] n-1 2

Partially supported by the PRIN 2010/2011


Nicolas Pessemier
email: pessemie@lifl.fr

Lionel Seinturier
email: seinturi@lifl.fr

Thierry Coupaye
email: thierry.coupaye@rd.francetelecom.com

Laurence Duchien
email: duchien@lifl.fr

A Model for Developing Component-Based and Aspect-Oriented Systems

Aspect-Oriented Programming (AOP) and Component-Based Software Engineering (CBSE) offer solutions to improve the separation of concerns and to enhance a program structure. If the integration of AOP into CBSE has already been proposed, none of these solutions focus on the application of CBSE principles to AOP. In this paper we propose a twofold integration of AOP and CBSE. We introduce a general model for components and aspects, named Fractal Aspect Component (FAC). FAC decomposes a software system into regular components and aspect components (ACs), where an AC is a regular component that embodies a crosscutting concern. We reify the aspect domain of an AC and the relationship between an AC and a component, called an aspect binding, as first-class runtime entities. This clarifies the architecture of a system where components and aspects coexist. The system can evolve from the design to the execution by adding or removing components, aspects or bindings.

Introduction

Component-Based Software Engineering (CBSE) proposes to structure a program by separating concerns into clearly defined entities, called components. Reusable components with contractually specified interfaces are defined and composed together [START_REF] Szyperski | Component Software: Beyond Object-Oriented Programming[END_REF]. Subsequently, Architecture Description Languages [START_REF] Medvidovic | A classification and comparison framework for software architecture description languages[END_REF] can be used to specify the component compositions and interactions.

Aspect-Oriented Programming (AOP) [START_REF] Kiczales | Getting started with AspectJ[END_REF] identifies the code tangling and the code scattering in AOP programs. Some concerns mixed within an entity (code tangling), and some concerns scattered across several entities, are said to be crosscutting. These concerns hinders the reusability, the maintainability, and the evolvability of applications. AOP proposes artifacts (aspect, pointcut, advice) to modularize crosscutting concerns.

It has been shown that the issues of code tangling and scattering arise at the level of CBSE as well [START_REF] Duclos | Describing and using non functional aspects in component based applications[END_REF][START_REF] Lieberherr | Programming with Aspectual Components[END_REF]. This is why merging AOP and CBSE makes sense. The integration of AOP into CBSE has already been proposed in [START_REF] Lagaisse | Component-based open middleware supporting aspectoriented software composition[END_REF][START_REF] Mezini | Conquering Aspects with Caesar[END_REF][START_REF] Suve | JAsCo: an aspect-oriented approach tailored for component based software development[END_REF], by providing a support for AOP in a component-based system. However, the application of CBSE principles to AOP is rarely proposed. In particular the implicit link between advice code and the base program where the advice applies is frequently hidden behind pointcut declarations (PcDs). Generally defined with a pattern language, PcDs select a set of joinpoints among those offered by the system. Unfortunately, once woven to the system, the implicit relationships created between a piece of advice code and the advised entities are never explicitly discernible and can surely not be individually manipulated at runtime.

In this paper, we propose a general and symmetrical model for mixing components and aspects. The approach is symmetric by considering aspects as plain components. The approach improves the component approach by giving a support for AOP, and improves the aspect approach by applying CBSE concepts to AOP. Our proposal relies on three main notions: aspect component, aspect domain, aspect binding. Aspects are contractually specified components called aspect components (ACs), and the relationships between ACs and regular components are reified with aspect domains, and aspect bindings. An AC embodies a crosscutting concern and can be reused in different contexts. An aspect domain is the reification of the components picked out by an AC. An aspect binding is a binding between a regular component and an AC. Thus, the model supports two levels of composition: regular components are composed together using regular bindings, and an AC is composed with regular components using aspect bindings.

We experiment this model by extending a reflective and general component model, named Fractal [START_REF] Bruneton | An open component model and its support in Java[END_REF] and its ADL. In our extension, called Fractal Aspect Component (FAC for short), we introduce the notions of aspect component, aspect binding and aspect domain to the component model itself and to the Fractal ADL.

The rest of this paper is organized as follows. Section 2 introduces our general model for component and aspect. Section 3 presents the mapping of our model to the Fractal component model. Section 4 presents related work around the merging of components and aspects and some reference component models. Section 5 concludes and gives some open issues.

A General Model for Components and Aspects

This section describes the three main concepts we introduce to support AOP in a component model: aspect component, aspect binding, and aspect domain. Section 2.1 gives the motivations of our approach. The concepts are presented in the remaining three sections.

Motivations

When merging Aspect-Oriented Programming (AOP) and Component-Based Software Engineering (CBSE) two dimensions have to be considered: the inte-gration of aspect-oriented principles into component-based systems, and the application of component-based principles to Aspect-Oriented Programming. The integration of AOP into CBSE is motivated by the code tangling issue inherent in CBSE [START_REF] Duclos | Describing and using non functional aspects in component based applications[END_REF][START_REF] Lieberherr | Programming with Aspectual Components[END_REF]. On the other hand the application of CBSE concepts to AOP is less investigated. Some approaches focus on the representation of an aspect as a component to contractually specify aspects and to increase their reusability [START_REF] Lagaisse | Component-based open middleware supporting aspectoriented software composition[END_REF][START_REF] Suve | FuseJ web site[END_REF].

In our proposal we realize a twofold integration of CBSE and AOP. We introduce three main concepts: aspect component, aspect domain, and aspect binding. These three notions are closely related to the three main concepts of the component approach: component, composite, and binding.

Aspect Component

An aspect component (AC for short) embodies a crosscutting concern. It is a regular component providing as a service a piece of around advice code. This service represents the behavior which will be woven around a set of regular components. This notion is similar to the notion of Aspectual Component proposed in 1999 by Karl Lieberherr et al [START_REF] Lieberherr | Programming with Aspectual Components[END_REF] to express each aspect separately in a modular structure.

Our approach is symmetric by making no differences between an aspect and a component. Thus, an aspect which is represented as a component, becomes a reusable contractually specified entity. Another consequence of the symmetric approach is that regular and aspect components are composed the same way using the same rules. This facilitates the adaptation to new requirements when the system evolves

Aspect Binding

Two kinds of binding exist within our model: regular and aspect binding. A regular binding expresses that a component is using a service provided by an other component. An aspect binding expresses that a component is aspectized by an aspect component. It is the reification of the individual relationship between an aspect component (AC) and a regular component where the AC applies.

In most existing AOP languages, the relationship between an aspect and the objects containing joinpoints picked out by the aspects is explicit in the source code but is implicit at runtime. Indeed, this relationship is structurally defined by a pointcut in the source code, but is lost when the woven code is executed. By introducing the notion of an aspect binding, we reify at runtime this relationship.

Because our approach is symmetric, all possible interactions between regular components and aspect components require full consideration. In a system using components and aspects, the possible interactions are described below.

-The component to component interaction is the classical client-server interaction. The client component uses a service provided by a server component interface. This kind of interaction exists in every component model using the notion of binding. -The component to AC interaction is our notion of an aspect binding. It expresses the fact that an aspect component is woven on a component. In Figure 1 we can see this type of interaction between C, D, E and the transaction aspect component. -The AC to component interaction, using a regular binding, is used as an AOP best practice. In Figure 1 we can see this kind of interaction between the transaction aspect component and various transaction policy components.

In this example, changing a transaction policy for example is performed through a reconfiguration between the transaction AC and the components providing transaction policies. -The AC to AC interaction can express a collaboration between two aspects using regular bindings, or the fact that the second aspect is woven on the first one. In asymmetric approaches this type of relationship is frequently unconsidered. Few techniques are given to make two aspects collaborate such as the use of context passing. The possibility of weaving aspects on other aspects is also uncommon.

Aspect Domain

An aspect domain is the reification of the components picked out by an AC. The goal of an aspect domain is to keep an overview on all the components affected by an aspect. It offers an abstraction on each AC woven on a set of components. A benefit that can be derived from the aspect domain notion is that the crosscutting interactions of a component-based system are clearly specified and as easily manipulable as regular interactions. Figure 2 illustrates the notion of aspect domain on a generic component-based application (components A to F). The aspect domains are represented as dotted Fig. 2. FAC overview: weaving of three crosscutting concerns rectangles, aspect bindings have been omitted for clarity sake. This application contains several crosscutting concerns: a logging, a persistence, and a transaction concern. These concerns are well known to be scattered and not cleanly modularized into one specific module. Their integration into an application is a hard task. In a full-fledged component, obtaining the same result requires numerous and tricky modifications. Moreover, once integrated to a system, it is difficult to remove one of these concerns in an easy and proper way. Once woven to a set of components the aspect domains of the ACs appear, offering reification on crosscutting relationships over the system.

Mapping onto the Fractal Component Model

This section presents the mapping of the main notions presented in the previous section onto the Fractal component model, which is a general and extensible component model supporting dynamic (regular) bindings. Our extension of Fractal is called FAC for Fractal Aspect Component. Section 3.1 presents the Fractal component model, and Section 3.2 proposes our extension FAC.

Fractal: A general and reflective component model supporting dynamic bindings

Fractal is an ObjectWeb consortium 3 project that proposes an extensible and modular component model [START_REF] Bruneton | An open component model and its support in Java[END_REF]. This Section describes Fractal main features. Note that Fractal is independent of any programming language. Several implementations exist in different languages such as Java, SmallTalk, C, C++, and the languages supported by the .NET platform. Contrary to component models for application servers such as EJB or .NET, Fractal is a general and reflective component model for developing complex software systems, such as operating systems and middleware. Besides the notion of a component, Fractal offers the notion of composite-component (allowing different views and abstractions on a system), shared component (a component nested by several composite components), dynamic binding (between components). Fractal is a reflective component model and offers introspection (system monitoring), and reconfiguration capabilities (modification of the system architecture).

A Fractal component has two parts: a content and a membrane. The content of a composite component is built as a set of sub-components, and the content of a primitive component (black box component) implements its provided services.

A component membrane can offer a level of control and a level of interception. The level of control is a set of interfaces to manage the non-functional properties of a component such as life cycle, bindings', content', name', or attributes' management. This set of control interfaces can be extended with new control interfaces that can be added to a component membrane. The interception mechanism reifies messages sent by and received on component interfaces. These messages can be modified, discarded or delivered to the component.

An interface is an access point to a component comparable to the notion of a port in several component models, like ArchJava [START_REF] Aldrich | ArchJava: Connecting software architecture to implementation[END_REF] or CCM [START_REF] Omg | CORBA Components, v3.0 ([END_REF]. A Fractal component offers external and internal interfaces. External interfaces are accessed from the outside of the component, while internal interfaces are only accessible from the composite's sub-components.

A binding is a communication channel between a client interface and a server interface. A client interface uses operations provided by a server interface. Fractal architectures can be described with Fractal ADL, which is an XML language to describe and to instantiate a Fractal component assembly. Figure 4 presents the example of Figure 3. Lines 2-3, 4, and 9 show the definition of server interfaces (role="server"). Lines 3-7 define the component A and Lines 8-11 the component B. Lines 12-13 are binding declarations of the binding between the server interface r of the composite and the server interface r of component A, and the binding between the client interface s of the component A and the server interface s of the component B.

Although component approaches such as Fractal offer several artifacts for the strong encapsulation of entities, the reification of dependencies, and the building of architecture from high level point of view, these approaches suffer from code <interface name="r" role="server" signature="java.lang.Runnable"/> 05 <interface name="s" role="client" signature="Service"/> 06 <content class="AImpl"/> 07 </component> 08 <component name="B"> 09 <interface name="s" role="server" signature="Service"/> 10 <content class="BImpl"/> 11 </component> 12 <binding client="this.r" server="A.r"/> 13 <binding client="A.s" server="B.s"/> 14 </definition> Fig. 4. Fractal-ADL: XML description of the helloworld application tangling and code scattering. These two issues seriously limit the evolution of a system. Thus, when a crosscutting concern has to be plugged to a Fractal component assembly, the amount of reconfigurations tthat must be performed may become quite heavy. The next section details the mapping of our concepts of aspect component, binding, and domain onto the Fractal component model.

Fractal Aspect Component (FAC)

FAC is our mapping of the general model exposed in Section 2 onto the Fractal component model. It uses existing notions of the Fractal model and introduces new ones.

Figure 5 presents the FAC metamodel. It is based on the Fractal metamodel. The mapping of the three main notions (aspect component, aspect domain, and aspect binding) is straightforward. An aspect component is defined as a regular component; it provides as a service a piece of advice code (see the AspectComponent Interface). An aspect domain is a composite component that contains a set of ACs representing a crosscutting concern, and the components impacted by the ACs. Within the context of an aspect domain, aspect bindings can be defined between components and the AC. The following sub-sections describe the concepts of aspect component interface, and weaving interface. We then detail the FAC pointcut language. Finally, we discuss the two implementations of FAC.

FAC join point model

Two different types of join points are supported by FAC: incoming calls on server interface operations, and outgoing calls on client interface operations. This choice is motivated by the fact that we consider AOSD in a component world.

As components are black boxes, it is rather natural to consider only join points on externally visible elements, i.e., exported and imported interfaces. Taking into account other kinds of join points, such as the ones on implementations, would break component encapsulation. Yet, for cases where this would be necessary, we believe that a best practice is to use a combination of component-based and implementation (e.g. object) based aspect-oriented tools.

The level of interception defined by FAC is very similar, at the component level, to the composition filters approach ([1]), which defines IN and OUT filters on objects to intercept messages.

FAC pointcut language

The FAC pointcut language is used to select join points. A pointcut expression is divided in two parts:

-A keyword that specifies if the incoming calls (keyword SERVER) or outgoing calls (keyword CLIENT) or both of them (no keyword) must be selected, -Three regular expressions separated by semicolons that specify which components, interfaces, and operations must be selected.

Figure 6 gives some examples of PcDs. The regular expressions relies on the java.util.regexp package.

The Aspect Component Interface (ACI)

The Aspect Component Interface (ACI) follows the AOP Alliance API4 , which is an open source initiative to define a common API for AOP frameworks. Figure 7 presents the AspectComponent Java interface and an example of an AC. ACs apply on component methods exposed by client and server interfaces. The parameter of the invoke method is a reification of a Fractal interface invocation. It provides a set of methods to introspect a join point. The argument of the invocation can be modified, the intercepted method can be called (proceed ), and the reference of the intercepted component can be retrieved.

Writing an AC requires implementing the AspectComponent interface. The invoke method describes the behaviour of the aspect. The code written in this method will be executed around the join point, i.e., a method call or execution on a component interface. The proceed call denotes the original method call. The code written before and after proceed() represents the before and after advices of AOSD. If more than one aspect applies on a given join point, the proceed call will trigger the next aspect, till the original method code is reached. If proceed is omitted the original method call will not apply. This can be useful to prevent, for example, the execution of the intercepted method.

Weaving Interfaces

The Weaving Interface (WI) of a component plays a key role in FAC. It manages the weaving of ACs around the interfaces of the component it controls. In the context of Fractal, we chose to represent the WI as a Fractal control interface in the component membrane. The WI uses the interception mechanism, which is provided by the membrane of components to intercept incoming and outgoing calls on its functional interfaces, and then, delegates the calls to the aspect components bound to (with an aspect binding) these operations. The weaving interface in FAC has three main objectives:

-Set/unset aspect bindings to aspect components, void setAspectBinding(Component comp, ItfPointcutExp regExp, AspectComponent ac); void unsetAspectBinding(AspectComponent ac); -Automatically weave an AC around a set of components following a pointcut declaration (this weaving task will automatically create an aspect domain, add the components which match the pointcut declaration into this aspect domain, and bind with aspect bindings the AC and the impacted components), void weave(Component rootComp, AspectComponent ac, ItfPointcutExp pExp, String aspectDomain); void unweave(Component rootComp, Component ac); -Provide a set of operation to order/re-order ACs which apply on an interface operation.

String[] changeACorder(String acName, int newPosition);

In FAC, a component supporting the weaving interface is called an aspectizable component. Otherwise, no aspects can be woven to this component. Since the weaving of an AC using the weaving interface is recursive and traverse the component hierarchy, if the component controlled by the WI is a composite component the weaving is also performed by its sub-components. A weaving operation can be initiated on the system as a whole (top-level composite) or on any sub system (intermediate composite).

All the operations provided by the interface can be invoked either with the Fractal ADL (extended with FAC notions) or directly at runtime.

The following piece of XML code presents the architecture of a Fractal assembly where a directive (tag <weave>) weaves a traceAC component (defined lines 2-4) to each component of the composite C (rootComp="this" line 12), which has an interface operation starting with "s" and returning "void". The aspect domain of this weaving will be automatically created and the composite representing this domain will be named "D" (adomain="D" line 12). 01 <definition name="C"> 02 <component name="traceAC"/> 03 <interface name="ACI" role="server" signature="AspectComponent"/> 04 </component> 05 <component name="A"/> 06 <interface name="itf1" role="client" signature="Itf1"/> 07 </component> 08 <component name="B"/> 09 <interface name="itf1" role="server" signature="Itf1"/> 10 </component> 11 <binding client="A.itf1" server="B.itf1"/> 12 <weaving ac="traceAC" pcd="*;*;s*:void" rootComp="this" adomain="D"/> 13 </definition> Every reconfiguration operations including the ones of our extension: setting/unsetting of aspect binding, weaving of an AC are dynamic operations.

Implementation issues

The mapping of our general model for component and aspect on the Fractal component model has been validated with two different implementations in Java. Our first implementation extends the reference implementation of the Fractal component model in Java called Julia ( [START_REF] Bruneton | An open component model and its support in Java[END_REF]). Julia uses a mixin ( [START_REF] Bracha | Mixin-based inheritance[END_REF]) mechanism to program the level of control of components. The second implementation extends another implementation of the Fractal component model in Java, called AOKell ( [START_REF] Seinturier | AOKell: an Aspect-Oriented Implementation of the Fractal Specifications[END_REF]), which uses AspectJ ( [START_REF] Kiczales | Getting started with AspectJ[END_REF]) aspects to implement control membranes.

Related Work

In this section, we compare FAC with different kinds of approaches. Firstly, we focus on approaches using the notions of component and aspect at a programming language level. Secondly, we investigate approaches using a symmetric representation of components and aspects. Thirdly, we study others component models.

Component and aspect at the programming language level

CaesarJ [START_REF] Mezini | Conquering Aspects with Caesar[END_REF] is a Java based programming language, defined as an extension of the AspectJ language. The components are implemented as collaborations of classes. A collaboration defines a provided part and an expected part. The provided part is implemented with reusable CaesarJ components that are a set of virtual classes. A virtual class in CaesarJ is a kind of inner class, which can be overridden in the subclasses of the enclosing class. When overriding an inner class the new functionalities are directly usable by the parent class. With this mechanism the provided operation of a collaboration interface in CaesarJ can be delayed to virtual sub-classes. On the other hand, the expected part is achieved by CaesarJ bindings, which are aspects woven afterward during a deployment phase. The main advantage of CaesarJ is its ability to stay close to the programming language and to be a superset of the AspectJ language.

JAsCo [START_REF] Suve | JAsCo: an aspect-oriented approach tailored for component based software development[END_REF] is an AOP language originally designed for component-based systems. It introduces two main notions: aspect bean and connector. Aspect beans are reusable Java beans describing the extra behavior to apply to components. Aspect beans uses hooks that are similar to inner classes describing the advice code and the pointcut declaration. Connectors are used to deploy hooks (some kind of access points to join points) within a specific context.

Contrary to FAC, CaesarJ and JAsCo are programmatic approaches. In the case of CaesarJ, aspects are dedicated to the expression of the relationships between components. The problem is that these aspects manage the bindings of all the components of the system. In the case of JAsCo, only crosscutting relationships are expressed thanks to connectors, the management of dependencies between base entities is missing.

Symmetric and unified approaches: Aspects conceived as components

FuseJ [START_REF] Suve | FuseJ web site[END_REF], which is the follow-up project by the JAsCo team, mainly focuses on the nature of an aspect that is represented as a regular bean component. The approach is symmetric: all the concerns are implemented as plain components. DyMAC [START_REF] Lagaisse | Component-based open middleware supporting aspectoriented software composition[END_REF] is a component and aspect-based middleware framework. It uses aspect-oriented composition to connect the application logic to the middleware services. Similarly to FAC, an aspect component in DyMAC encapsulates an advice. However around advices are not supported. Special kinds of connectors are statically described in XML files to write technical services of the middleware layer. Connectors in DyMAC looks like aspect bindings in our approach. Nevertheless, aspect ordering is static in DyMAC whereas FAC provides an API for aspect component ordering.

Other component models

OpenCOM [START_REF] Clarke | An efficient component model for the construction of adaptive middleware[END_REF] is a lightweight reflective component model and as such close to the Fractal component model. The key concepts of the model are interfaces, receptacles and connections. A component has a set of receptacles and interfaces. Interfaces are used to express provided services and receptacles to express required services (comparable to Fractal client and server interfaces). Unlike Fractal, OpenCOM defines a fixed meta-object protocol for components. The meta objects in OpenCOM can be compared to aspect components (ACs) in FAC. However, this meta level is fixed and thus does not support the dynamic adding and removing of meta objects. K-Component [START_REF] Dowling | The k-component architecture meta-model for selfadaptive software[END_REF] is a component model for building context-adaptive applications. Instead of using an Architecture Description Language to statically describe a component architecture, the model reifies the structure of the application and describes adaptation contracts written with an Adaptation Contract Description Language (ACDL) to dynamically reconfigure the application. The representation of the architecture is defined with a typed graph. Thus, the reconfiguration of the architecture is performed through a graph transformation. The K-Components are defined using the OMG-IDL3 language and C++ idioms. The main drawback of this approach is that adaptation is always realized through reconfiguration of the component architecture. An interception mechanism is missing to add an AOP support to the approach.

JBoss AOP [START_REF] Burke | JBoss-AOP[END_REF] is a Java framework for AOP. It can be used in the context of the JBoss application server or standalone. As JAC [START_REF] Pawlak | JAC : An aspect-based distributed dynamic framework[END_REF], JBoss-AOP offers a set of pre-programmed aspects that can be used directly. JBoss AOP aspects can be woven with annotations, classic pointcut declarations or in a dynamic way at system runtime. When applied to the JBoss application server, aspects are woven to components. Similarly to FAC, dedicated XML fragments are used to deploy aspects. However, in FAC, XML files are used to describe a component assembly, with bindings. The notion of binding in EJB component model is missing. Components are coarse-grained components, encapsulated by containers, which do not express relationships between each other.

Conclusion

In this paper we have presented a general model for components and aspects and its mapping onto the Fractal component model called FAC. This model introduces three main notions: aspect component, aspect domain, and aspect binding. A crosscutting concern is embodied by a regular Fractal component called an aspect component. We have shown that an aspect component is an encapsulation of advice code. An aspect domain is the reification of the notion of a pointcut: the components picked out by an aspect component. The implicit relationship between a woven aspect component and the component in which the aspect component applies is a first-class entity called an aspect binding.

The main contribution of our approach is to bring aspect-oriented concepts to the component world, and conversely, to improve aspect-oriented approach with component notions. Thus, our three main notions (aspect component, aspect binding, aspect domain) are mapped onto the Fractal component model using existing notions of component, binding, and composite component.

We also provide a runtime support for crosscutting relationship reflection, which is an open issue in the aspect-oriented community. Moreover, we offer various abstraction views on aspect components woven on components in order to help the evolution of the modular and crosscutting concerns of a component and aspect system.

The long term objective of FAC is to work with aspects at three different levels [START_REF] Pessemier | A three level framework for adapting component-based systems[END_REF]. The first level is the use of AOP at the program level, namely the level of objects that are encapsulated by components. Current AOP approaches fulfill this need. The second level is FAC itself with the notions of aspect component, aspect binding, and aspect domain. Joinpoints at this level are invocations on component interfaces. And finally, we plan to consider a third level, an architectural level, where joinpoints are architectural operations and transformations.

Fig. 3 .

 3 Fig. 3. Fractal-ADL: helloworld application

Fig. 5 .

 5 Fig. 5. FAC metamodel

  Components in FuseJ are equiped with gates. A gate is a kind of interface to specify component services: aspect-oriented and regular. The connectors (extension of JAsCo connectors) specify the types of interaction between gates. FuseJ defines regular and aspect-oriented connectors. Regular connectors are in charge of functional connections between gates, and aspect-oriented connectors are in charge of weaving a component behavior to another component. All the connections defined by a component can be locally consulted. FuseJ does not yet propose a global description of a component architecture with its connections. The model does not support the managing of aspect domain and aspect binding as FAC does.

http://aopalliance.sourceforge.net/

Acknowledgments

This work was partially funded by France Telecom under the external research contract number 46 131 097.


Marco Corneli

Pierre Latouche

Fabrice Rossi

Modelling time evolving interactions in networks through a non stationary extension of stochastic block models

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Modelling time evolving interactions in networks through a non stationary extension of stochastic block models Abstract-The stochastic block model (SBM) [START_REF] Holland | Stochastic blockmodels: first steps[END_REF] describes interactions between nodes of a network following a probabilistic approach. Nodes belong to hidden clusters and the probabilities of interactions only depend on these clusters. Interactions of time varying intensity are not taken into account. By partitioning the whole time horizon, in which interactions are observed, we develop a non stationary extension of the SBM, allowing us to simultaneously cluster the nodes of a network and the fixed time intervals in which interactions take place. The number of clusters as well as memberships to clusters are finally obtained through the maximization of the complete-data integrated likelihood relying on a greedy search approach. Experiments are carried out in order to assess the proposed methodology.

I. A NON STATIONARY STOCHASTIC BLOCK MODEL

We consider a set of N nodes A = {a 1 , . . . , a N }. Directed links between these nodes are observed over the time interval [0, T ] and self loops are not considered. Nodes in A belong to K disjoint clusters:

A = ∪ k≤K A k , A l ∩ A g = ∅, ∀l = g.
We introduce an hidden vector c = {c 1 , . . . , c N } such that:

c i = k iff i ∈ A k , ∀k ≤ K
and assume that c follows a multinomial distribution:

P{c i = k} = ω k with k≤K ω k = 1.
As a consequence, modulo an independence assumption, the joint density for vector c is:

p(c|ω, K) = k≤K ω |A k | k , where |A k | = #{a i : a i ∈ A k } is the number of nodes inside cluster A k .
In order to introduce a temporal structure, consider a sequence of equally spaced, adjacent time steps:

∆ u := t u -t u-1 , u ∈ {1, . . . , U }
over the interval [0, T ] and a partition C 1 , . . . , C D of the same interval 1 . We introduce a random vector y = {y u } u≤U , not observed, such that:

y u = d iff I u :=]t u-1 , t u ] ∈ C d , ∀d ≤ D.
We attach to y a multinomial distribution:

p(y|β, D) = d≤D β |C d | d ,
where

|C d | = #{I u : I u ∈ C d }.
Assuming that labels c and y are independently distributed, their joint distribution is:

p(c, y|Φ, K, D) =   k≤K ω |A k | k     d≤D β |C d | d   , (1)
where Φ = {ω, β}.

We now define N Iu ij as the number of observed connections from i to j, in the time interval I u . The following crucial assumption is made:

p(N Iu ij |c i = k, c j = g, y u = d) = p(N Iu ij |λ kgd ), where: p(N Iu ij |λ kgd ) = (∆ u λ kgd ) N Iu ij N Iu ij ! e -∆uλ kgd . (2)
We note Λ the set of all parameters λ kgd .

Remarks: For i and j fixed and c known, the sequence:

N I1 ij , . . . , N I U ij
is independently but not identically distributed, therefore this model is a non-stationary extension of the SBM. Moreover, this is the sequence of increments of a non-homogeneous Poisson process, counting interactions from cluster c i to cluster c j . Notation: In the following, for simplicity, we will note: and similarly for cj =g and yu=d . The adjacency matrix, noted N ∆ , has three dimensions (N × N × U ):

N ∆ = {N Iu ij } i≤N,
j≤N,u≤U and its observed likelihood can be computed explicitly:

p(N ∆ |Λ, c, y, K, D) = k,g,d ci=k cj =g yu=d p(N Iu ij |λ kgd ) = k,g,d (∆λ kgd ) S kgd P kgd e -∆λ kgd R kgd , (3)
where we noted:

S kgd := ci=k cj =g yu=d N Iu ij , P kgd = ci=k cj =g yu=d N Iu ij ! R kgd := |A k ||A g ||C d | if g = k |A k ||A k-1 ||C d | if g = k
Note that the subscript u was removed from ∆ u to emphasize that time steps are equally spaced for every u.

II. EXACT ICL FOR NON STATIONARY SBM

In order to estimate the label vectors c and y and the number of clusters K and D, we directly maximize an exact version of the Integrated Classification Criterion (ICL) [START_REF] Biernacki | Assessing a mixture model for clustering with the integrated completed likelihood[END_REF] by means of a greedy search [START_REF] Côme | Model selection and clustering in stochastic block models based on the exact integrated complete data likelihood[END_REF]. The quantity we focus on is the complete data log-likelihood, integrated with respect to the model parameters Φ and Λ:

ICL = log p(N ∆ , c, y, Λ, Φ|K, D)dΛdΦ .
Introducing a prior distribution ν(Λ, Φ|K, D) over the pair (Φ, Λ) and making an independence assumption, allow us to write the ICL as follows:

ICL = log ν(N ∆ |c, y, K, D) + log (ν(c, y|K, D)) .
The choice of conjugated prior distributions is crucial to obtain an explicit form of the ICL. We impose a Gamma a priori over Λ:

ν(λ kgd |a, b) = b a Γ(a) λ a-1 kgd e -bλ kgd
and a factorizing Dirichlet a priori distribution to Φ:

ν(Φ|K, D) =Dir K (ω; α, . . . , α) × Dir D (β; γ, . . . , γ),
where the parameters of each distribution have been set constant for simplicity.

III. EXPERIMENTS

The dataset we used was collected during the ACM Hypertext conference held in Turin, June 29th -July 1st, 2009 and represents the dynamical network of face-to-face proximity interactions of 113 conference attendees over about 2.5 days 2 .

2 More informations can be found at: http://www.sociopatterns.org/datasets/hypertext-2009-dynamic-contact-network/.

We focused on the first conference day, namely the twenty four hours going from 8am of June 29th to 7.59am of June 30th. The day was partitioned in small time intervals of 20 seconds in the original data frame and face-to-face proximity interactions (less than 1.5 meters) were monitored by radio badges that conference attendees volunteered to wear. Further details can be found in [START_REF] Isella | What's in a crowd? analysis of face-to-face behavioral networks[END_REF]. We considered 15 minutes time aggregations, thus leading to a partition of the day made of 96 consecutive quarter-hours (U = 96 with previous notation). The greedy search algorithm converged to a final ICL of -53217.4, corresponding to 20 clusters for nodes (people) and 4 time clusters. In Figure 1, the total number of interactions for each quarter-hour can be observed. Different colors correspond to the time clusters found by the model, in particular time intervals corresponding to the highest number of interactions have been placed in cluster C 4 (blue), those corresponding to an intermediate interaction intensity, in C 2 (green) and C 3 (yellow). Cluster C 1 (magenta) contains those quarter-hours where the frequency of interactions is very low or null. It is interesting to note how the model closely recovers times of social gathering:

• 9.00-10.30 -set-up time for posters and demos.

• 13.00-15.00 -lunch break.

• 18.00-19.00 -wine and cheese reception. A complete program of the day can be found at: http://www.ht2009.org/program.php.

  75634 Paris Cedex 13 Email: Marco.Corneli@malix.univ-paris1.fr Pierre Latouche SAMM laboratory Université Paris 1 Pantheon-Sorbonne Paris, France 75634 Paris Cedex 13 Email: pierre.latouche@univ-paris1.fr Fabrice Rossi SAMM laboratory Université Paris 1 Pantheon-Sorbonne Paris, France 75634 Paris Cedex 13 Email: Fabrice.Rossi@univ-paris1.fr

Fig. 1 :

 1 Fig. 1: Aggregated interactions for each time interval

T and U are linked by the following relation: T = ∆uU .


Siddhartha Chandra
email: siddhartha.chandra@inria.frstavros.tsogkas@centralesupelec.friasonas.kokkinos@inria.fr

Stavros Tsogkas

Iasonas Kokkinos

Accurate Human-Limb Segmentation in RGB-D images for Intelligent Mobility Assistance Robots

Mobility impairment is one of the biggest challenges faced by elderly people in today's society. The inability to move about freely poses severe restrictions on their independence and general quality of life. This work is dedicated to developing intelligent robotic platforms that assist users to move without requiring a human attendant. This work was done in the context of an EU project involved in developing an intelligent robot for elderly user assistance. The robot is equipped with a Kinect sensor, and the visioncomponent of the project has the responsibility of locating the user, estimating the user's pose, and recognizing gestures by the user. All these goals can take advantage of a method that accurately segments human-limbs in the colour (RGB) and depth (D) images captured by the Kinect sensor. We exploit recent advances in deep-learning to develop a system that performs accurate semantic segmentation of human limbs using colour and depth images. Our novel technical contributions are the following: 1) we describe a scheme for manual annotation of videos, that eliminates the need to annotate segmentation masks in every single frame; 2) we extend a state of the art deep learning system for semantic segmentation, to exploit diverse RGB and depth data, in a single framework for training and testing; 3) we evaluate different variants of our system and demonstrate promising performance, as well the contribution of diverse data, on our in-house Human-Limb dataset. Our method is very efficient, running at 8 frames per second on a GPU.

Introduction

Mobility impairment is widespread among elderly people today. Recent studies indicate that 20% of people between the age of 70 -85 years, and 50% of people above the age of 85 years report difficulties accomplishing basic activities of daily life [START_REF] Papageorgiou | Advances in intelligent mobility assistance robot integrating multimodal sensory processing[END_REF]. Mobility impairment not only restricts independence severely, it is also detrimental to the self-esteem of the individuals and drastically limits their quality of life. Studies in demographics show that mobility impairment among people over the age of 65 is constantly increasing in industrialized countries [START_REF] Census | The elderly population[END_REF]. The goal of this work is the design and implementation of intelligent mobility assistance robots, or nurse-type robots, that help the user accomplish basic everyday tasks such as standing up, sitting down, and moving about, avoiding stationary physical obstacles. Users would be able to interact with these robots through pre-determined gestures, and be able to draw the robot closer, or farther, and indicate the type assistance they need. The computer vision component of such systems would exploit information, in the form of RGB-D images, from Kinect-sensors to allow the system (a) locate the user, (b) estimate their pose, and (c) understand their gestures. All these tasks involve recognizing landmarks/parts on the human body. While locations of these parts would give the system a fair indication of the user's location, the relative orientations of these parts would enable the system to recognize pose/ gesture. To this end, we develop a deeplearning based solution to human-limb segmentation, which provides accurate pixel-wise class labels to each pixel in the images captured by the camera. Figure 2 shows one prac-tical usecase of our solution. Human-limb segmentations, in conjunction with the depth field, can be used to estimate 3 -D surfaces of body parts of the user. Normals of these surfaces can be used to determine if the user's pose is unstable. In case of instability, the robot can assist the user achieve a stable position by exerting a physical force of the optimal magnitude in the optimal direction.

Our contributions are threefold. We first create the Human-Limb-dataset from videos captured by a Kinect sensor, and devise a scheme for annotating pixelwise classlabels that does not require us to annotate every single frame in the video. Second, we combine the state of the art Deeplab network for semantic segmentation [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF] in RGB images, with the state of the art Alexnet-HHA network [START_REF] Gupta | Learning rich features from rgb-d images for object detection and segmentation[END_REF] on object detection in depth images to train a single network with the objective of pixelwise semantic segmentation in RGB-D images. Finally, we explain how we can re-purpose a deep CNN using Caffe [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF], to use diverse data (RGB + depth fields) in a unified framework for training and testing. Importantly, our method is also very efficient, running at 8 frames per second.

Figure 3 gives a visual summary of our framework. We begin with a brief review of recent advances in semantic segmentation, followed by describing our contributions, and finally report empirical results on the Human-Limb dataset.

Figure 2. Human-limb segmentations, alongside depth, can be exploited to fit surfaces corresponding to parts of the body. Normals to these surfaces can then be estimated, and these normals give an indication of the stability of the user's pose. In the event of the user being in an unstable pose, the robot can use knowledge of these normals to help the user gain a stable pose by exerting the necessary physical force in the optimal direction.

Related Work

Semantic segmentation of body parts, and fine-grained semantic segmentation have attracted increased interest in recent years. Bo and Fowlkes [START_REF] Bo | Shape-based pedestrian parsing[END_REF] sequentially merge superpixels obtained from an oversegmentation, to form larger parts, such as head, arms, clothes etc., and parse pedestrians. Simple constraints, (such as that head appears above upper body and that hair appears above head) enforce a consistent layout of parts in resulting segmentations

In [9] Eslami et al. introduced the Shape Boltzmann Ma-chine (SBM), an hierarchical generative model that can generate realistic samples from the underlying object shape distribution. This model was extended to deal with multiple region labels (parts) in [START_REF] Eslami | A generative model for partsbased object segmentation[END_REF] and coupled with a model for part appearances.

Lu et al. formulate car parsing as a landmark identification problem [START_REF] Lu | Parsing semantic parts of cars using graphical models and segment appearance consistency[END_REF]. In [START_REF] Wang | Semantic part segmentation using compositional model combining shape and appearance[END_REF] the authors draw inspiration from previous work on hierarchical models for objects [START_REF] Jin | Context and hierarchy in a probabilistic image model[END_REF][START_REF] Kokkinos | Inference and learning with hierarchical shape models[END_REF][START_REF] Zhu | Learning a hierarchical deformable template for rapid deformable object parsing[END_REF][START_REF] Zhu | A stochastic grammar of images[END_REF] and propose a mixture of compositional models that represent horses and cows in terms of their boundaries and the boundaries of semantic parts. Their algorithm starts by segmenting large parts first, such as head, neck, torso, and moves on to segment legs, which are deformable and thus much more difficult to segment.

Many works have used depth images to complement RGB information for scene labelling. Ren et al. convert local similarities (kernels) to patch descriptors, and incorporate context information using superpixel MRFs and segmentation trees [START_REF] Ren | Rgb-(d) scene labeling: Features and algorithms[END_REF]. Couprie et al. [START_REF] Couprie | Indoor semantic segmentation using depth information[END_REF] adopt a multiscale convolutional neural network to learn features directly from RGB-D images, whereas Wang et al. attempt to learn visual patterns from RGB and depth in a joint manner via an unsupervised learning framework [START_REF] Le | Ica with reconstruction cost for efficient overcomplete feature learning[END_REF][START_REF] Wang | Multimodal unsupervised feature learning for rgb-d scene labeling[END_REF]. They sample RGB-D patches and feed them as input to a two-layer stacked structure. The output of their method is a collection of superpixels that combine features from different sampled patches. As a final step, they train linear SVM classifiers to assign scene labels to each superpixel.

Gupta et al. generalize the popular contour detection and hierarchical segmentation gPb algorithm [START_REF] Arbelaez | Contour detection and hierarchical image segmentation[END_REF] to make effective use of depth information. In [START_REF] Gupta | Perceptual organization and recognition of indoor scenes from rgb-d images[END_REF], they design features based on local geometric contour information, shape, size and appearance of superpixels, and train classifiers to assign each superpixel a semantic class label. In [START_REF] Gupta | Learning rich features from rgb-d images for object detection and segmentation[END_REF], they follow an object-centric approach. They propose a three-channel encoding of a raw depth field, and modify R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] so as to exploit this new type of information. Detection obtained from this modified R-CNN system are then used to compute additional features for superpixels and improve the semantic segmentation system of [START_REF] Gupta | Perceptual organization and recognition of indoor scenes from rgb-d images[END_REF].

Recently, Banica and Sminchisescu modified CPMC [START_REF] Carreira | Cpmc: Automatic object segmentation using constrained parametric min-cuts[END_REF] to account for both intensity and depth discontinuities [START_REF] Banica | Second-order constrained parametric proposals and sequential search-based structured prediction for semantic segmentation in rgb-d images[END_REF]. A pool of figure-ground segmentations is created and a ranker is trained to distinguish which of those segments correspond to objects. A ranking process selects a valid and compact subset of segments, while local, hand-crafted features, as well as learned CNN features, are used to classify the retained segmentation masks.

Our work combines ideas from [START_REF] Gupta | Perceptual organization and recognition of indoor scenes from rgb-d images[END_REF][START_REF] Gupta | Learning rich features from rgb-d images for object detection and segmentation[END_REF] and recent works on semantic segmentations of objects that employ deep convolutional neural networks [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF][START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]. We augment the Deeplab system to take advantage of depth information, and change the labelling task from object segmentation to hu- man limb parsing. Training and testing is performed using a single network that can seamlessly switch from only using RGB or depth images to the augmented RGB-D inputs.

Dataset

Our data-acquisition setup consisted of a Kinect sensor mounted on top of a passive rollator, as in [START_REF] Papageorgiou | Advances in intelligent mobility assistance robot integrating multimodal sensory processing[END_REF]. The recording process involved capturing RGB-D videos using the open-source Robotics Operating System (ROS) software capturing human subjects in a set of predefined use-cases and scenarios. Example colour and depth images are shown in Figure 4, along with manually annotated ground-truth.

Our Human-Limb dataset consists of six recorded videos, each containing a different human subject performing a sitstand-sit activity several times. These six videos correspond to 2618 RGB-D frames in total. We split the dataset into 1399 training and 1219 testing frames: images of three human subjects are used to construct the training set, while images of the remaining three subjects are used to construct the testing set. We annotate each of these images with 7 labels, namely, (1) head, (2) torso, (3) left upper arm, (4) right upper arm, (5) left lower arm, (6) right lower arm, and (7) background. In Section 3.1, we describe the scheme we employed for efficient annotation, exploiting the visual similarity of frames in a video.

Annotation

Annotating a video/sequence of frames with pixel-wise labels can be a tedious task. One key observation specific to this task is that videos typically contain a lot of visually similar frames. While consecutive frames in a video are usually similar, in our recordings, the human subjects perform the same task (sit-stand-sit), repeatedly, increasing redundancy among frames.

To avoid annotating very similar frames more than once, we use a clustering-based approach to reduce the dataset into a set of representative frames. We annotate each of these representative frames using the open source image editor GIMP (www.gimp.org). Finally, the ground truth labels for each frame in the dataset are transferred from the annotation of the visually most similar frame from the set of representative frames.

The clustering approach we use to determine the representative frames consists of the following steps: we begin by computing HOG [START_REF] Felzenszwalb | Object detection with discriminatively trained partbased models[END_REF] features on all colour image frames in our dataset. Next, we compute euclidean distances in the HOG space between each pair of frames, and lastly, we employ a greedy strategy to group frames into clusters,using a threshold on the distance in the HOG space: any two frames within a distance less than d thresh are merged into the same cluster. The hyper-parameter d thresh is manually picked, ensuring that all the frames in each cluster look very similar. One frame from each cluster is randomly picked to be the cluster representative. Figure 5 shows some example clusters returned by our algorithm. On average, we obtained 28 clusters per video in our dataset. In the end we made a final pass of the full dataset, to visually assess the quality of the transferred annotations.

Human-Limb Segmentation in RGB-D Images

Given the dataset of RGB-D images and pixelwise classlabels, we now describe our deep learning framework for human-limb segmentation. Our approach builds on top of the Deeplab network [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF], which is a state of the art method for semantic segmentation on RGB images. In this work, we propose an extension to the Deeplab architecture, that allows it to exploit RGB-D images. We begin by describing the original network, followed by our extension.

Deeplab Network for Semantic Segmentation in RGB images

In recent years, Convolutional Neural Networks have shown unprecedented performance in object classification Figure 5. Example clusters returned by our clustering method, which puts two images together in the same cluster if the euclidean distance between them in the HOG space is below a threshold. We exploit the visual similarity of images in the same cluster by annotating one image from the cluster and propagating its annotation to other images in the cluster.

and detection [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], and they have already become the standard learning machinery for many other computer vision tasks. Recently, it was shown that CNNs can also be used in a fully convolutional setting for semantic segmentation [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF][START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF].

One of the challenges faces when using convolutional networks for semantic segmentation is the downsampling factor. Due to repeated max-pooling and downsampling, the spatial resolution of an input image is reduced as it is propagated through the network. The popular Alexnet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] has a down-sampling factor of 16. This means if we use a 128 × 128 image as input, we will get 8 × 8 activations (ignoring the last dimension for brevity-we are only concerned with the spatial dimensions in this example). While this is not a problem for tasks like image classification and object detection, it is undesirable for tasks such as image segmentation where we require a class label for each of the 128 × 128 pixels. Output maps typically have to be scaled 16 times in a post processing procedure, thus introducing approximation errors. The Deeplab network proposed by Chen et al. [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF], reduces the down-sampling factor to 8 by introducing holes in the convolution kernel. In this work, we build on the fully convolutional Deeplab CNN and adapt it for the semantic segmentation of human limbs. A schematic representation of the Deeplab network is shown in Figure 6.

The Deeplab network is based on the very deep model for object classification by Simonyan et al. [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], after dropping the fully connected layers. This results in a fully convolutional network, preserving spatial information that is normally scraped away by fully connected layers. More precisely, the Deeplab network consists of 16 convolutional layers, interleaved with 5 pooling layers. It also has 15 normalization layers, and 2 dropout layers. We initialize the weights using the VGG pretrained network and finetune the network for semantic segmentation of human limbs. For optimization, we use standard stochastic gradient descent, minimizing a cross-entropy loss, averaged over all image positions. As in [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF], we also use the post-processing step of applying a fully-connected CRF to obtain sharper masks, rather than simply relying on the CNN coarse responses. Our experiments indicate that using CRF leads to moderate improvements in segmentation accuracy.

Deeplab Convolutional Neural Network

Loss RGB Image Pixelwise Annotations Figure 6. Schematic representation of the Deeplab network, using RGB images as input.

Alexnet-HHA Network for Semantic Segmentation in depth images

The Deeplab network described in the previous subsection uses RGB images. However, we would like to exploit the available depth frames in addition to the RGB images. A straightforward approach to exploit depth information for our task, would be to simply add the raw depth field as an extra channel in the RGB input images during training. This is problematic for two reasons: first, the range of values for the depth channel is very different than the one for RGB inputs, which means that a calibration step could be needed to account for this discrepancy during training. Second, instead of taking advantage of the initialized weights in the pre-trained model and simply finetune the network with a few passes over the dataset, we would be forced to retrain the full network, a procedure that usually takes several days.

Gupta et al. [START_REF] Gupta | Learning rich features from rgb-d images for object detection and segmentation[END_REF] propose an alternative approach: instead of taking the depth fields at face value, they encode each pixel value of the depth fields with a 3×1 vector representing horizontal disparity(H), height above the ground(H) and the angle(A) the pixel's local surface normal makes with the inferred gravity direction. As a result, an M × N depth field is transformed into an M × N × 3, HHA encoding, and values are linearly scaled to map observed values across the training dataset to the 0 to 255 range. Authors in [START_REF] Gupta | Learning rich features from rgb-d images for object detection and segmentation[END_REF] finetune the popular Alexnet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] using HHA images for the task of object detection. This representation encodes properties of geocentric pose a CNN would have difficulty learning directly from a depth image, especially when very limited training data is available. Rows 2,3 in Figure 4 show two example depth images and their corresponding HHA representations respectively.

In this work, we take advantage of the pretrained Alexnet-HHA in a similar way. We start by encoding the depth frames in our dataset to the HHA representation, and use them to finetune a modified version of Alexnet-HHA for the task of semantic segmentation. We optimize performance on the training set using the same cross-entropy loss function as in [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF]. The original Alexnet has 5 convolutional layers, 3 pooling layers, and several normalization and dropout layers. As stated in section 4.1, Alexnet has a downsampling factor of 16. To retain a downsampling factor of 8 while being able to use the pretrained network weights, we discard any convolutional / fully connected layers after the second convolutional layer. A schematic representation of the Alexnet-HHA network is shown in Figure 7.

Human-limb Segmentation in RGB-D images

Having described our extensions of the Deeplab and Alexnet-HHA networks for semantic segmentation in RGB and depth images respectively, we now turn to the task of combining them to train a single network which learns from colour and depth images simultaneously. To this end we combine the two networks by concatenating their penultimate layers, and then using a single loss layer on top. A schematic representation of our combined network is shown in Figure 8.

Learning from RGB, HHA, RGB+HHA images

Proliferation of depth sensors is a recent phenomenon. Consequently, we have a lot of publicly available datasets with colour images, and comparatively fewer depth image datasets. From a technical standpoint, it is desirable to have a single network that can exploit RGB-D images, or resort to using only one type of information when the other is not available. To incorporate this flexibility in our network, we implement a novel adaptiveConcatenation layer in Caffe [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF]. The adaptiveConcatenation layer accepts a flag alongside each image which indicates whether the input image is RGB only, HHA only or RGB+HHA. During the feed-forward phase, the adaptiveConcatenation layer ignores penultimate layer activations of the HHA component if the image is RGB only, and ignores the penultimate layer activations of the RGB component if the layer is HHA only. Similarly, during the back-propagation phase, if the image is RGB only, the loss is back-propagated only to the Deeplab network component. If the image is HHA only, the loss is back-propagated only to the Alexnet-HHA component. Otherwise, the loss is backpropagated through both components of the network. In our experiments, we demonstrate the utility of this layer, by using images of outdoor scenes from RGB only INRIA person database to augment our training set with more background examples.

Experiments

We use our method for the task of semantic segmentation of human limbs. All experiments are conducted on the Human-Limb Dataset, which is described in detail in section 3. We compare four different setups in our empirical evaluation, (1) Deeplab network using RGB images, (2) Alexnet-HHA network using HHA images, (3) Deeplab+Alexnet-HHA network (described in section 4.3) using RGB+HHA images, and (4) Deeplab+Alexnet-HHA network using RGB+HHA images from our dataset alongside 1218 RGB images from the INRIA person database, containing outdoor images for additional background examples. The last method uses the adaptiveConcatenation layer introduced in section 4.4.

While training/testing all networks, we resize input images to 242 × 322 pixels. Since we have 7 labels (6 human body parts + 1 background label), our networks are designed to output 7 scores per pixel, one corresponding to each label. As pointed out in sections 4.1, 4.2, our networks have a downsampling factor of 8. Due to this downsampling of spatial dimensions, the size of the final output score map is 31 × 41 × 7. Thus, during training we also downsample the ground truth label maps to 31 × 41 pixels by nearest neighbour interpolation, to avoid averaging artifacts. At test time, we resize the output scores to the original image size of 242 × 322 pixels via bilinear interpolation. To get the labels, we compute the softmax probabilities of each label from the scores, and pick the label with the highest probability. We also employ dense CRF to refine segmentations provided by our networks, as in [START_REF] Chen | Semantic image segmentation with deep convolutional nets and fully connected crfs[END_REF]. The dense CRF framework serves as a post-processing step on our networks' output scores, before computing the softmax-probabilities for class labels.

We report quantitative results of our evaluation in table 5. We use the accuracy of segmentation as the evaluation metric for our experiments. In table 5, we report results both with and without the dense CRF. Our results indicate that using dense CRF improves results, boosting the networks' capability to capture fine details. Figure 9 shows some qualitative results on the test images.

Our implementation is built on top of the Caffe library, and our testing time for RGB+HHA is about 8 frames-persecond on an NVIDIA-Tesla K40 GPU. 1. Segmentation Accuracies of our system for the four combinations of input types. We also show the effect of postprocessing the scores using dense CRF on the performance of these methods.

Method

Discussion

In this paper we presented a method for combining information from RGB images and depth images to train a system for semantic segmentation of human body parts. Our system is flexible, allowing us to take advantage of the two types of inputs, or fall back on using just one type, when the other is not available. We also introduced a novel set of body part annotations, and used it to evaluate the performance of our approach. We demonstrate promising results and, as a future direction, we plan to explore the application of body-part semantic segmentation for action recognition and pose estimation.

Figure 1 .

 1 Figure 1. Our system takes RGB or RGB-D images as input and produces a parsing of the person's limbs, torso, and head.

Figure 3 .

 3 Figure 3. Our method combines RGB and depth information to train a fully convolutional deep neural network for human limb semantic segmentation.

Figure 4 .

 4 Figure 4. Example images from the dataset. Row 1 shows the RGB images. Row 2 shows the depth images. Row 3 shows the depth images encoded to 3 channel HHA [13], and Row 4 shows the pixelwise ground truth class labels.

Figure 7 .

 7 Figure 7. Schematic representation of our modified Alexnet-HHA network that uses a 3-channel encoding of the depth fields as input.

Figure 8 .

 8 Figure 8. We combine RGB and HHA-encoded depth information in a single deep network.