Carlos Olarte

Camilo Rueda

Frank D Valencia

Frank D Valencia Concurrent

Music Systems.. Gérard Andrew Gerzso Assayag

Concurrent Constraint Calculi: a Declarative Paradigm for Modeling Music Systems

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Concurrent constraint programming (CCP) [START_REF] Saraswat | Concurrent Constraint Programming[END_REF] has emerged as a simple but powerful paradigm for concurrent systems; i.e. systems of multiple agents that interact with each other as for example in a collection of music processes (musicians) performing a particular piece. A fundamental issue in CCP is the specification of concurrent systems by means of constraints. A constraint (e.g. note > 60) represents partial information about certain system variables. During the computation, the current state of the system is specified by a set of constraints (store). Processes can change the state of the system by telling (adding) information to the store and synchronize by asking information to the store.

The ntcc calculus [START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF] is a CCP formalism for modeling temporal reactive systems. The calculus is built upon a small number of basic ideas that capture several aspects of temporal nondeterministic behavior. In ntcc, processes can also be constrained by temporal requirements such as delays, time-outs and pre-emptions. Thus, the calculus integrates two dimensions of computation: a horizontal dimension dealing with partial information (e.g., note > 60) and a vertical one in which temporal requirements come into play (e.g., a process must be executed at any time within the next ten time units).

The above integration is remarkably useful for modeling complex musical processes, in particular for music improvisation. For example, for the vertical dimension one can specify that a given process can nondeterministically choose any note satisfying a given constraint. For the horizontal dimension one can specify that the process can nondeterministically choose the time to play the note subject to a given time upper bound. This nondeterministic view is particularly suitable for processes representing a musician's choices when improvising. Similarly, the horizontal dimension may supply partial information on a rhythmic pattern that leaves room for variation while keeping a basic control, as in one of the examples given below.

We shall also illustrate how implementing a weaker ntcc model of a musical process may greatly simplify the formal verification of its properties. We argue that this modeling strategy provides a "runnable specification" for music problems that eases the task of formally reasoning about them.

Furthermore, ntcc is provided with a linear temporal process logic for verifying specifications of ntcc processes. We shall show how this logic gives a very expressive setting for formally proving the existence of interesting musical properties of a process. We shall also give examples of musical specifications in ntcc and use the linear temporal logic for proving properties of a realistic musical problem.

Preliminaries

CCP calculi are parametrized by a constraint system [START_REF] Saraswat | Concurrent Constraint Programming[END_REF] which specifies the basic constraints agents can tell or ask during execution. A constraint represents a piece of information (or partial information) upon which processes may act. For instance, in a system with variables pitch 1 , pitch 2 taking MIDI values, the constraint pitch 1 > pitch 2 specifies possible values for pitch 1 and pitch 2 (those where pitch 1 is at least a tone higher than pitch 2 ). The constraint system defines also an entailment relation (⊢) specifying inter-dependencies between constraints. Intuitively, c ⊢ d means that the information d can be deduced from the information represented by the constraint c. For example, pitch 1 > 60 ⊢ pitch 1 > 42.

Formally, we can set up the notion of constraint system by using First-Order Logic as it was done in [START_REF] Smolka | A foundation for higher-order concurrent constraint programming[END_REF]. Let us suppose that Σ is a signature (i.e., a set of constant, function and predicate symbols) and that ∆ is a consistent first-order theory over Σ (i.e., a set of sentences over Σ having at least one model). Constraints can be thought of as first-order formulae over Σ. Consequently, the entailment relation is defined as follows: c ⊢ d if the implication c ⇒ d is valid in ∆. Notice that true is the smallest constraint in the sense that for any constraint c, c ⊢ true. Analogously, false is the greatest constraint since from false any constraint can be deduced. This gives us a simple and general formalization of the notion of constraint system as a pair (Σ, ∆).

As an example, take the finite domain constraint system (FD) [START_REF] Van Hentenryck | Design, implementation, and evaluation of the constraint language cc(fd)[END_REF]. In FD, variables are assumed to range over finite domains and, in addition to equality, we may have predicates that restrict the possible values of a variable to some finite set. For instance, assume x to be a FD variable with initial domain {1, 2, 3, ..., 10}. Adding the constraint x > 5 will restrict the domain of x to be the set {6, 7, ..., 10}.

Alternatively, the notion of constraint system can be given in terms of Scott's information system without consistency structure as it was done in [START_REF] Saraswat | Concurrent Constraint Programming[END_REF].

CCP Processes

In the spirit of process calculi, the language of processes in the CCP model is given by a small number of primitive operators or combinators. A typical CCP process language features the following constructs:

• A tell operator adding a constraint to the store, thus making it available for other processes.

• An ask operator querying if a constraint can be deduced from the store.

• Parallel Composition combining processes concurrently.

• A hiding operator (also called restriction or locality) introducing local variables and thus restricting the interface a process can use to interact with others.

Following the notation in [START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF], we present the syntax of CCP in Definition 1.

Definition 1 (CCP Processes) Processes P, Q, . . . in CCP are built from constraints in the underlying constraint system by the following syntax:

P, Q := skip | tell(c) | when c do P | P Q | (local x; c) P
The process skip does nothing. It represents inaction. The process tell(c) adds the constraint c to the store. The process when c do P asks if c can be deduced from the store. If so, it behaves as P . In other case, it remains blocked until the store contains at least as much information as c. This way, ask processes define a synchronization mechanism based on entailment of constraints.

The process P Q denotes the parallel composition of P and Q, i.e., P and Q running in parallel and possibly "communicating" via the common store.

Hiding on a set of variables x is enforced by the process (local x; c) P . It behaves like P , except that all the information on the variables x produced by P can only be seen by P and the information on the global variable in x produced by other processes cannot be seen by P . The local information on x produced by P corresponds to the constraint c representing a local store. We shall write (local x) P as a shorthand for (local x; true) P.

Timed CCP

In CCP, the information in the store grows monotonically, i.e., once a constraint is added it cannot be removed. This condition has been relaxed by considering temporal extensions of CCP such as Timed CCP (tcc) [START_REF] Saraswat | Foundations of timed concurrent constraint programming[END_REF]. In tcc, processes evolve along a series of discrete time intervals (or time units). Each interval contains its own store and information is not automatically transferred from one interval to another. In music, the duration of each time unit is related to the minimal metric value used in the piece.

Definition 2 (Temporal Constructs) The tcc processes result from adding in Definition 1 the following constructs

P, Q := next P | unless c next P | ! P
The constructs above allow the processes in Definition 1 to have effect along the time units. The unit-delay next P executes P in the next time interval. The negative ask unless c next P is also a unit-delay but P is executed in the next time unit iff c is not entailed by the final store at the current time interval. This can be viewed as a (weak) time-out (or weak preemption): It waits one time unit for a piece of information c to be present and if it is not, it triggers activity in the next time interval. The process P must be guarded by a next process to avoid paradoxes such as a program that requires a constraint to be present at an instant only if it is not present at that instant (see [START_REF] Saraswat | Foundations of timed concurrent constraint programming[END_REF]).

Notice that in general Q = unless c next P does not behave the same as Q ′ = when ¬c do next P . This can be explained from the fact that d ⊢ c does not imply d ⊢ ¬c. Take for example the final store d = "x > 0" and let c = "x = 42". We have both, x > 0 ⊢ x = 42 and x > 0 ⊢ x = 42. Then, the process P is executed in

Q but it is precluded from execution in Q ′ .
Finally, the replication ! P means P next P next 2 P..., i.e. unboundedly many copies of P but one at a time.

The tcc model. As we said before, processes in tcc evolve along a series of discrete time intervals. In each time interval, a deterministic CCP processes receives a stimulus (i.e. a constraint) from the environment (an input to the system). It executes with this stimulus as the initial store. When it reaches its resting point, i.e., no further evolution is possible, it responds to the environment with the final store (the output of the system). Furthermore, the resting point determines a residual process, which is then executed in the next time interval. Roughly speaking, the residual process is obtained by "unfolding" the sub-terms within "next" and "unless" expressions (see [START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF]). This view of reactive computation is particularly appropriate for programming reactive systems in the sense of Synchronous Languages [START_REF] Berry | The Esterel synchronous programming language: Design, semantics, implementation[END_REF], i.e., systems that react continuously with the environment at a rate controlled by the environment.

Extensions to Timed CCP

Several extensions of the basic constructs presented above have been studied in the literature in order to provide settings for the programming and specification of systems with the declarative flavor of concurrent constraint programming. For example, temporal extensions to deal with reactive systems have been introduced in [START_REF] Saraswat | Foundations of timed concurrent constraint programming[END_REF][START_REF] Frank | A timed concurrent constraint language[END_REF][START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF], non-deterministic behavior and asynchrony [START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF][START_REF] Frank | A timed concurrent constraint language[END_REF], probabilistic behavior [START_REF] Gupta | Probabilistic concurrent constraint programming[END_REF][START_REF] Bortolussi | Modeling biological systems in stochastic concurrent constraint programming[END_REF][START_REF] Olarte | A stochastic concurrent constraint based framework to model and verify biological systems[END_REF][START_REF] Pérez | Non-determinism and Probabilities in Timed Concurrent Constraint Programming[END_REF], mobility [START_REF] Olarte | Universal concurrent constraint programing: Symbolic semantics and applications to security[END_REF], linearity [START_REF] Fages | Linear concurrent constraint programming: Operational and phase semantics[END_REF], etc. We shall briefly describe some of these extensions (see Figure 1).

Non-determinism and Asynchrony.

The tcc calculus is a model of reactive computation based on the idea of Synchronous Languages [START_REF] Berry | The Esterel synchronous programming language: Design, semantics, implementation[END_REF]. i.e, programs must be determinate and respond immediately to input signals. Therefore, it is not meant for modeling nondeterministic or asynchronous temporal behavior. Indeed, patterns of temporal behavior such as "the system must output c within the next t time units" or "the message must be delivered but there is no bound in the delivery time" cannot be expressed within the model. It also rules out the possibility of choosing one among several alternatives as an output to the environment.

The ntcc calculus [START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF] is obtained by adding guarded-choices for modeling nondeterministic behavior and an unbounded finite-delay operator for asynchronous behavior. Computation in ntcc progresses as in tcc, except for the nondeterminism induced by the new constructs.

Definition 3 (ntcc Processes) The ntcc processes result from adding to the syntax in Definition 2 the following constructs:

i∈I when c i do P i | ⋆ P
The guarded-choice i∈I when c 1 do P i where I is a finite set of indices, represents a process that, in the current time interval, must non-deterministically choose one of the P j ( j ∈ I ) whose corresponding guard (constraint) c j is entailed by the store. The chosen alternative, if any, precludes the others. If no choice is possible then the summation remains blocked until more information is added to the store.

The operator "⋆" corresponds to the unbounded but finite delay operator ǫ for synchronous CCS [START_REF] Milner | A finite delay operator in synchronous ccs[END_REF] and it allows to express asynchronous behavior through the time intervals. Intuitively, process ⋆P represents P + next P + next 2 P + ..., i.e., an arbitrary long but finite delay for the activation of P.

Strong Pre-Emption in tcc

The work in [START_REF] Saraswat | Timed default concurrent constraint programming[END_REF] introduces the notion of strong pre-emption in tcc, i.e. the time-out operations can trigger activity in the current time interval. Strong pre-emption is useful when an action must be triggered immediately on the absence of a constraint c rather than delayed to the next interaction with the environment as in unless c next P . In this case, it is assumed that c has not been produced in the store, and will not be produced throughout system execution at the present time instant. Definition 4 (Default tcc Processes) The Default tcc processes result from adding in the syntax in Definition 2 the following construct: if c else P Intuitively, if c cannot be deduced from the current store now and it will not be produced during the current time interval, the process P is executed. To avoid paradoxes, it is assumed that the process P does not add c to the current store (see [START_REF] Saraswat | Timed default concurrent constraint programming[END_REF]).

Stochastic and Probabilistic Behavior

When modeling a complex system, it is no longer possible to assume that enough information will be provided for all its components. Hence, models have to deal with uncertain behavior or a incomplete description of these components. A natural model is then to consider these components as stochastic ones where a probability distribution is given to the set of outputs the partially known components may perform. Indeed, apart from providing an intuitive way of describing alternative behaviors, probability distributions often ease the integration of statistic and empirical data into models.

In [START_REF] Gupta | Stochastic processes as concurrent constraint programs[END_REF], the authors develop Probabilistic CCP (pcc) and its temporal extension ptcc to model synchronous reactive probabilistic systems. In these languages, (discrete) random variables with a given probability distribution are introduced. A run of the system will choose a value for the random variables with the given probability; these probability values accumulate as more and more choices are made in the course of the run. Alternate choices lead to alternate runs, with their own accumulated probability values.

Definition 5 (pcc and ptcc Processes) The pcc(resp. ptcc) processes result from adding to the syntax in Definition 1 (resp. Definition 2) the process new(r, f ) in P where r is a variable and f is its probability mass function.

Intuitively, the process new(r, f ) in P chooses a value for r according to the function f and then execute P .

The authors in [START_REF] Pérez | Non-determinism and Probabilities in Timed Concurrent Constraint Programming[END_REF] study also the integration of probabilistic information into CCP. The language pntcc is then proposed featuring both non-deterministic and probabilistic behavior. The operational semantics of pntcc ensures the consistent interactions between both the probabilistic and the non-deterministic choices. The semantics is based on a probabilistic automaton [START_REF] Segala | Modeling and verification of randomized distributed real-time systems[END_REF] that separates the internal choices made probabilistically by the processes from those external choices made non-deterministically under the influence of a scheduler. As a result, the observable behavior of a system -what the environment perceives from its execution-formalized by the semantics is purely probabilistic; the influence of non-determinism is regarded as unobservable.

In the CCP spirit, pntcc allows to reason about process specifications in terms of properties including explicit quantitative information: pntcc processes are related to formulae in the probabilistic logic PCTL [START_REF] Hansson | A logic for reasoning about time and reliability[END_REF]. This relation is based on the fact that the observable behavior of a process can be interpreted as the discrete time Markov chain (DTMC) defining satisfaction in PCTL.

Definition 6 (pntcc Processes)

The pntcc processes result from adding to the syntax in Definition 2 the process i∈I when c i do (P i , a i ) where I is a finite set of indices, and for every a i ∈ R (0,1] we have i inI

a i = 1
The intuition of this operator is as follows. Each a i associated with the process P i represents its probability of being selected for execution. Hence, the collection of all a i represents a probability distribution.

Mobile Behavior

In [START_REF] Olarte | Universal concurrent constraint programing: Symbolic semantics and applications to security[END_REF] the authors introduce Universal Timed CCP (utcc), a calculus aiming at modeling mobile reactive systems, i.e., systems that can change their communication structure while interacting continuously with their environment. Here we understand mobility as in the π-calculus [START_REF] Parrow | A calculus of mobile processes, Parts I and II[END_REF], i.e. generation and communication of private channels or links.

The basic move from tcc to utcc is to replace the ask operation when c do P by a temporary parametric ask constructor of the form (abs x; c) P . This process can be viewed as an abstraction of the process P on the variables x under the constraint (or with the guard ) c. Intuitively, (abs x; c) P performs P [ t/ x] in the current time interval for all the terms t s.t c[ t/ x] is entailed by the store. The abstraction construct in utcc has a pleasant duality with the local operator. From a programming language perspective, the variables x in (local x; c) P can be seen as the local variables of P while x in (abs x; c) P as the formal parameters of P .

Definition 7 (utcc Processes)

The utcc processes result from replacing in the syntax in Definition 2 the expression when c do P with (abs x; c) P where variables in x are pairwise distinct.

When the set of variables x in (abs x; c) P is empty, we retrieve the classical ask operator when c do P .

The extra expressiveness of utcc to model mobile behavior has found application, e.g., to model dynamic music systems where the composer can dynamically change the hierarchical structure of the score according to the information derived from the environment [START_REF] Olarte | A Declarative Language for Dynamic Multimedia Interaction Systems Proc[END_REF].

Logic Characterization of CCP

CCP calculi enjoys a declarative nature that distinguishes it from other models of concurrency: CCP programs (or processes) can be seen, at the same time, as both computing agents and logic formulae (see e.g., [START_REF] Saraswat | Concurrent Constraint Programming[END_REF][START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF][START_REF] De Boer | Proving concurrent constraint programs correct[END_REF][START_REF] Olarte | The expressivity of universal timed CCP: Undecidability of monadic FLTL and closure operators for security[END_REF]), i.e., programs can be read and understood as logical specifications. A natural benefit of this alternative view is to provide the modeler or developer with a language suitable for both the specification and implementation of programs. Let us elaborate on this ideas in the context of the ntcc calculus. See [START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF] for a detailed treatment on this topic.

Linear temporal logics (LTL) have been extensively used to specify properties of timed systems [START_REF] Manna | The Temporal Logic of Reactive and Concurrent Systems: Specification[END_REF]. Formulae in this logic are built from the following syntax: Definition 8 (Logic Syntax) The formulae A, B, ... ∈ A are defined by the grammar A, B, . . .

:= c | A ⇒ A | ¬ A | ∃x A | •A | A | ✸A
Here c denotes an arbitrary constraint which we shall refer to as atomic proposition. The intended meaning of the other symbols is the following: ⇒, ¬ and ∃ represent linear-temporal logic implication, negation and existential quantification. These symbols are not to be confused with the symbols ⇒, ¬ and ∃ of the underlying constraint system.

The symbols •, , and ✸ denote the temporal operators next, always and sometime. Intuitively • A, ✸ A and A means that the property A must hold next, eventually and always, respectively.

Let P be a process and A a LTL formulae specifying a given temporal property. One may wonder whether the process P satisfies the specification A, written as P |= A. The intended meaning of this assertion is that, regardless the input, every output of P satisfies the temporal formula A.

Let us give some examples. Since in every infinite sequence output by ⋆ tell(c) on arbitrary inputs there must be an element entailing c,

Representation of time

As described above, in CCP calculi time is usually thought of in terms of state transitions. Conceptually, a computation instantly produces some information and then reaches a quiescent state. The information output is what is observable in that state. Besides information, as a result of the computation some processes might be scheduled for a "future time". This "future time" refers to a particular new state where the information the process computes is observable. In ntcc, a time unit is thus related to a state. Two different states refer to different times, but there is no explicit notion of distance between the two states. In practice one might think the extension of each time unit to represent the smallest temporal separation between observable changes in the system. The analogy in music would be the smallest rhythmical unit in a score, with no "absolute" temporal duration involved. Temporal models in ntcc would then typically "synchronize" so that information changes conforms to multiples of the smallest unit. The ntcc interpreter would then have to give some actual absolute duration to this unit, like a conductor giving tempo.

For the simple score shown below a ntcc model could be where each time unit is seen to correspond to a sixteenth note. Now, it is up to the implementation (i.e. the interpreter) to ensure that each time unit takes exactly whatever actual absolute duration is desired for the sixteenth note. A more realistic model would take advantage of the synchronizing capabilities of the calculus to associate each note to a process deciding when to play by looking at a conductor: The score is then defined by a process Q Q = P 1 P 2 P 3 P 4

P 1 =
In this case the interpretation of the absolute duration of the sixteenth note is left to a signal from the environment (say supplied by a Max-MSP process [START_REF] Puckette | Max at seventeen[END_REF]) or from a ntcc process modeling the conductor.

Let us now consider each of the above musical process with respect to some simple temporal operations. Suppose corresponding ntcc definitions for process R modeling Now, we would like to model In the first representation this is done by process R next 16 P whereas in the second the conductor would have to use two hands, providing signal 1 for R and signal for Q. In general, each process should use its own different signal and the conductor should be aware of that. Moreover, this is only a partial solution since it would not work for two or more instances of, say, process P . When processes are combined as voices going simultaneously, e.g., parallel composition P R works appropriately in both representations. However, a slight metric transformation such as cannot be expressed as some composition of P and R, but it could, in principle, still be modeled as Q R assuming the conductor accelerates the pace of signal while maintaining that of signal 1 .

Timed CCP calculi thus provide a logical but not a metrical view of time. In particular, no support for metric combinations of the temporal occurrence of processes is given and it is up to the user to devise coordination patterns to represent metric structures in some way. Devising schemes to overcome this limitation while keeping the clean semantics of tcc is one of our aims in the future. One can envision combinators such as combinator meaning P ; k Q P followed by Q delayed k units P ≪ k process P stretched proportionally k units P ≫ k process P expanded proportionally k units P ⊎ Q process Q time-intercalated with P and others.

When real-time behavior of models is intended, there is the issue of taking explicitly into account the time needed for a computation. That is, using a more fine grained notion of time in which the "duration" of a state of affairs is determined by the time spent in doing the computation defined for that state. In this view, a state could no longer be defined by whatever information is output in a time unit as a whole (considered thus as instantaneous), but instead by each action over the store. Each time unit would be composed of several such states. In this view, a state corresponds to the information contained in the store. Each change in the store defines a new state. What would then be the time elapsed between two such states? Since the only possible change in a store is the addition of a constraint, a natural proposal would be to consider the "distance" between states s 1 and s 2 to be that constraint differentiating them. Whether this provides a coherent metric in practice is something we plan to dwell in the future.

Musical Applications

In the last decade several formalisms have been proposed to account for musical structures and the operations used to construct and transform them [START_REF] Barbar | Structures hierarchiques pour la composition assisté par ordinateur[END_REF][START_REF] Balaban | Hierarchy, time and inheritance in music modelling[END_REF][START_REF] Pachet | Representing temporal musical objects and reasoning in the muses system[END_REF]. We can regard music performance and composition as a complex task of defining and controlling interaction among concurrent activities. In [START_REF] Rueda | Integrating constraints and concurrent objects in musical applications: A calculus and its visual language[END_REF], PiCO, a concurrent process calculus integrating constraints and objects was proposed. Musical applications are programmed in a visual language having this calculus as its underlying model. Since there is no explicit notion of time in PiCO some musical examples, in particular those involving time and synchronization, are difficult to express. In this section we model two of those examples.

In the examples below we shall use the derived operators ⋆ [m,n] P , [m,n] P and W (c,P ) . The process ⋆ [m,n] P means that P is eventually active exactly once between the next m and m + n time units, while ! [m,n] P means that P is always active between the next m and m + n time units. The construct W (c,P ) waits until c holds and then executes P : We shall use the more readable notation wait c do P . Finally, to specify mutable and persistent data structures we shall use cells. Let us assume that the signature of the underlying constraint system is extended with an unary predicate symbol change. A mutable cell x: (v) can be viewed as a structure x which has a current value v and which can, in the future, be assigned a new value.

x: (z)

def = tell(x = z) unless change(x) next x: (z)
Intuitively, definition x: (z) represents a cell x whose value is z and it will be the same in the next time interval unless it is to be changed next (i.e., change(x)). Let exch g [x, y] be defined as

exch g [x, y] def = v when x = v do ( tell(change(x)) tell(change(y)) next ( x: (g(v)) y: (v) ) )
This process represents an exchange operation in the following sense: if v is x's current value then g(v) and v will be the next values of x and y respectively.

Controlled Improvisation.

This example models a controlled improvisation musical system. Such a system can be described as follows: There is a certain number m of musicians (or voices), each playing blocks of three notes. Each of them is given a particular pattern (i.e., a list) of allowed delays between each note in the block. The musician can freely choose any permutation of his pattern. For example, given a pattern p = [START_REF] Bortolussi | Modeling biological systems in stochastic concurrent constraint programming[END_REF][START_REF] Berry | The Esterel synchronous programming language: Design, semantics, implementation[END_REF][START_REF] De Boer | Proving concurrent constraint programs correct[END_REF] a musician can play his block with spaces of 5 then 4 and then 3 between the notes. Once a musician has finished playing his block of three notes, he must wait for a signal of the conductor telling him that the others musicians have also finished their respective blocks. Only after this he can start playing a new block. The exact time in which he actually starts playing a new block is not specified, but it is constrained to be no later than the sum of the durations of all patterns. For example, for three musicians and patterns p 1 = [3, 2, 2], p 2 = [4, 3, 5] and p 3 = [START_REF] Berry | The Esterel synchronous programming language: Design, semantics, implementation[END_REF][START_REF] Berry | The Esterel synchronous programming language: Design, semantics, implementation[END_REF][START_REF] Bortolussi | Modeling biological systems in stochastic concurrent constraint programming[END_REF] no delay between blocks greater than 29 time units is allowed. The musicians keep playing this way until all of them play a note at the same time. After this, all the musicians must stop playing.

In order to model this example we assume that constant sil ∈ D represents some note value for silence. The process M i , i ≤ m, models the activity of the i-th musician. When ready to start playing (start i = 1), the i-th musician chooses a permutation (j, k, l) of his given pattern p i . Then, M i spawns a process Play i (j,k,l) , thus playing a note at time j (after starting), but not before, then at time j + k but not before, and finally at time j + k + l. Constraint c i [note i ] specifies some value for note i different from sil. After playing his block, the i-th musician signals termination by setting cell flag i to 1. Furthermore, upon receiving the go = 1 signal, the i-th musician eventually starts a new block no later than pdur which is a constant representing the sum of the durations of all patterns.

M i def = ! when (start i = 1) do (j,k,l)∈perm(pi) ( Play i (j,k,l) next j+k+l ( flag i := 1 wait (go = 1) do ⋆ [0,pdur] tell(start i = 1)) ) Play i (j,k,l) def = ! [0, j-1] tell(note i = sil) next j tell(c i [note i ]) ! [j+1, j+k-1] tell(note i = sil) next j+k tell(c i [note i ]) ! [j+k+1, j+k+l-1] tell(note i = sil) next j+k+l tell(c i [note i ])
Here f lag is a mutable cell (see definition above) and f lag := 1 represents the assignment of 1 to f lag.

The Conductor process is always checking (listening) whether all the musicians play a note exactly at the same time i∈ [1,m] (note i = sil). If this happens it sets the cell stop, initially set to 0, to 1. At the same time, it waits for all flags to be set to 1, and then resets the flags and gives the signal go = 1 to all musician to start a new block, unless all of them have output a note at the same time (i.e., stop = 1). (flag i = 1) ∧ (stop = 0) do (tell(go = 1)

Conductor

i∈ [1,m] flag i := 0)

Initially the m flag cells are set to 0, the M i are given the start signal start i = 1 and, as mentioned above, the cell stop is set to 0. The system (i.e., the performance) System is just the parallel execution (performance) of all the M i musicians controlled by the Conductor process.

Init def = i∈[1,m] (flag i : 0 tell(start i = 1)) stop : 0 System def = Init Conductor i∈[1,m] M i
The temporal logic and the proof system of ntcc [START_REF] Nielsen | Temporal concurrent constraint programming: Denotation, logic and applications[END_REF] can then be used to formally specify and prove termination properties for this system. For example, we may wonder whether the assertion System |= ✸stop = 1 holds. This assertion expresses that the musicians eventually stop playing at all regardless their choices. We may also wonder whether there exists certain choices of musicians for which they eventually stops playing note at all. For proving this we can verify whether the assertion System |= stop = 0 does not hold, i.e., there is a run of the system for which at some time unit all the notes are different from sil.

Rhythm Patterns

In this section we shall model synchronization of rhythm patterns in ntcc. Let us first define a "metronome" process.

M [tick , count, δ] def = !( when (count mod δ) = 0 do tick := tick + 1 count := 0 + when (count mod δ) > 0 do count := count + 1)

One could think of M [tick , count, δ] as a process that "ticks" (by increasing tick ) every δ time units. This process could be controlled by the acceleration process: The process R (s,d,e) [tick , note] runs a certain P itch[note] process, which outputs some pitch on note, at every (d)uration-th tick, from the (s)tart-th tick to the (e)nd-th tick . Adding rhythms of two eight notes, two triplets and two quintuplets can then be defined by the system: where Init = tick : 0 δ : 30 count : 0. Since the Rhythm processes depend on variables tick and δ, complex patterns of interactions of global and local speeds, such as metric modulations, can be modeled.

Accel [signal , δ]

Future work

A strong asset of the declarative paradigm we described in this chapter is the fact that a model of a system is itself a simulator. For this advantage to be put into practice efficient interpreters must be provided. Although the authors have devised interpreters for several of the tcc-like calculi described above, more work is needed in interpretation strategies to ensure efficiency, in particular for real-time settings.

As explained above, the declarative feature of CCP calculi allows to formally reason about system properties. It is thus possible to guarantee the existence of fundamental musical properties in a model, without recurring to a possibly very time-consuming

Figure 1 .

 1 Figure 1. Hierarchy of CCP-based languages.

  we have ⋆ tell(c) |= ✸c. Analogously, ! tell(c) |= c since all output of P must entail c. Let P = tell(c) + tell(d). We have P |= (c ∨ d) as every constraint e output by P entails either c or d. Finally, when c do next tell(d) |= c ⇒ •d since the entailment of c in the first time unit implies the entailment of d in the next one.

=

  tell(note 1 = 72) next 3 (tell(note 2 = 75) next (tell(note 3 = 70) next 4 tell(note 4 = 70)))

1 P 2 = 2 P 3 = 3 P 4 =

 122334 when signal = 0 do tell(note 1 = 72) unless signal = 0 next P when signal = 3 do tell(note 2 = 75) unless signal = 3 next P when signal = 4 do tell(note 3 = 70) unless signal = 4 next P when signal = 8 do tell(note 4 = 70) unless signal = 8 next P 4

W

  (c,P ) def = when c do P unless c next W (c,P ) .

def=!

  when signal = 1 ∧ δ > 0 do δ := δ -1 The process Accel [signal , k] can "speed up the ticks of M [tick , count, δ]" by decreasing δ, if some other process, which we shall refer to as Control [signal ], tells signal = 1. We can now define the Rhythm process R (s,d,e) [tick , note] which can be synchronized by M [tick , count, δ] and thus possibly accelerated by Control [signal ]. R (s,d,e) [tick , note] def = ! when s ≥ tick ≥ e ∧ (tickstart) mod d = 0 do Pitch[note]

  tick δ count signal ) ( Init Control [signal ] M [tick , count, δ] Accel (signal , δ) R (0,30,120) [tick , δ] R (0,20,120) [tick , δ] R (0,12,120) [tick , δ] )

testing process. Automatic verification tools should be developed to ease the task of performing such proofs for complex models.

We have argued how the use of constraints greatly simplifies the task of devising synchronization patterns between processes. Since our aim is to take advantage of this fact to propose the CCP paradigm for complex music system modeling, we need to supply CCP calculi with more sophisticated temporal combinators, as described above. The challenge is to find ways in which these new constructs can find a place in the calculus and still maintain its clean semantics and declarative flavor.

Real-time behavior is at the core of many music systems, such as improvisation. Giving the user the right intuitions into the fine-grained temporal behavior of a model to achieve real-time awareness poses both a theoretical and implementation challenge. The theoretical aspects deals with finding the right calculus semantics for expressing concepts such as "true concurrence", "simultaneity" and "temporal distance" . On the implementation side, the challenge is to couple this with an efficient interpreter with effective user temporal-controls.


Diwei Wang
email: d.wang@unistra.fr

Chaima Zouaoui
email: chaima.zouaoui@ept.ucar.tn

Jinhyeok Jang

Hassen Drira
email: hdrira@unistra.fr

Hyewon Seo
email: seo@unistra.fr

Video-based gait analysis for assessing Alzheimer's Disease and Dementia with Lewy Bodies

Keywords: Gait impairment score, Dementia subtypes, Human 3D motion estimation, Geometric deep learning

Dementia with Lewy Bodies (DLB) and Alzheimer's Disease (AD) are two common neurodegenerative diseases among elderly people. Gait analysis plays a significant role in clinical assessments to discriminate these neurological disorders from healthy controls, to grade disease severity, and to further differentiate dementia subtypes. In this paper, we propose a deep-learning based model specifically designed to evaluate gait impairment score for assessing the dementia severity using monocular gait videos. Named MAX-GR, our model estimates the sequence of 3D body skeletons, applies corrections based on spatio-temporal gait features extracted from the input video, and performs classification on the corrected 3D pose sequence to determine the MDS-UPDRS gait scores. Experimental results show that our technique outperforms alternative state-of-the-art methods. The code, demo videos, as well as 3D skeleton dataset is available at https://github.com/lisqzqng/Video-based-g ait-analysis-for-dementia.

Introduction and related work

Through many previous studies, it is now well understood that the quantitative gait impairment analysis is an established method for accessing neurodegenerative diseases such as Dementia with Lewy Bodies (DLB) or Alzheimer (AD) and gauging their severity, even in the prodromal phase [START_REF] Mckeith | Research criteria for the diagnosis of prodromal dementia with lewy bodies[END_REF]. In order to facilitate quantitative gait analysis, previous works have often relied on wearable sensors [START_REF] Li | Sequential human gait classification with distributed radar sensor fusion[END_REF][START_REF] Teufl | Towards an inertial sensor-based wearable feedback system for patients after total hip arthroplasty: Validity and applicability for gait classification with gait kinematics-based features[END_REF]6,17] or electronic walkways [START_REF] Mc Ardle | Do alzheimer's and lewy body disease have discrete pathological signatures of gait?[END_REF][START_REF] Merory | Quantitative gait analysis in patients with dementia with lewy bodies and alzheimer's disease[END_REF]. According to Merory et al.'s study [START_REF] Merory | Quantitative gait analysis in patients with dementia with lewy bodies and alzheimer's disease[END_REF], individuals with AD and DLB show comparable spatiotemporal gait characteristics that differ significantly from those of the normal population. Conversely, Mc Ardle et al [START_REF] Mc Ardle | Do alzheimer's and lewy body disease have discrete pathological signatures of gait?[END_REF] demonstrate that the two subtype groups exhibit distinct pathological gait signatures. Another study [START_REF] Mc Ardle | The impact of environment on gait assessment: considerations from real-world gait analysis in dementia subtypes[END_REF] has shown that the environment where walking takes place has an influence on the characteristics of gait impairment in different types of dementia. However, these studies do not aim to automatically estimate severity scores based on the measurement data.

Numerous efforts have been made to classify or estimate the severity of a patient's condition and even distinguish between different dementia subtypes, by using gait data. Muller [START_REF] Muller | Corrélation des troubles de la marche au profil neuropsychologique chez les patients atteints de maladie d'alzheimer et maladie à corps de lewy[END_REF] employed a decision tree [START_REF] Winterfeldt | Decision Analysis and Behavioral Research[END_REF] to analyse gait motions of individuals with AD and DLB, and showed that walking speed and the asymmetry in left-to-right step lengths were the two primary factors for distinguishing between dementia subtypes and estimating the disease severity. However, the reliance on wearable sensors equipped with tri-axial accelerometers or electronic walkways in such studies can be cumbersome in terms of wearability, calibration, and may not always be readily accessible.

The progress in deep learning has opened up new possibilities for vision-based severity assessment methods. Albuquerque et al [START_REF] Albuquerque | A spatiotemporal deep learning approach for automatic pathological gait classification[END_REF] have developed a spatiotemporal deep learning technique by producing a gait representation that combines image features extracted through Convolutional Neural Networks (CNNs) with a temporal encoding based on Long Short-Term Memory (LSTM) networks. Lu et al [START_REF] Lu | Vision-based estimation of mds-updrs gait scores for assessing parkinson's disease motor severity[END_REF] extract 3D body pose from videos, track them through time, and classify the sequence of 3D poses based on the MDS-UPDRS gait scores by using a temporal convolutional neural network. Similarly, Sabo et al [START_REF] Sabo | Estimating parkinsonism severity in natural gait videos of older adults with dementia[END_REF] have shown that ST-GCN models operating on 3D joint trajectories outperform alternative models. Motivated by the achievements of previous studies, we adopt a similar approach of extracting 3D pose sequences from gait videos with an aim to enhance pathological gait analysis. However, our work distinguishes itself in that we introduce a new dedicated model for 3D motion estimation from monocular gait videos. Additionally, we employ a geometric deep learning module specially crafted for 3D skeleton-based action recognition. Consequently, our method achieves superior performance compared to numerous state-of-the-art techniques.

Method

Our patient data

The videos of patients undergoing the MDS-UPDRS gait examination at a neurology clinic have been used in our study. The patient walks along a GAITRite (https://www.gaitrite.com/) electronic walkway with dimensions of 0.6m × 8m, from one end to the other end. Three views have been interchangeably chosen for the RGB camera, without calibration: a side view from the mid-way of the walkway, a front view as the patient walks towards the camera, and a back view as the patient walks away from the camera. In the two latter cases, the distance between the camera and the patient varied from 1 meter to 9-10 meters. The recorded images had a resolution of 480 × 640 pixels, and the frame rate was set at 30Hz. A total of 92 sequences have been recorded from 44 subjects, including 41 patients with AD and DLB. In addition, each video has been annotated with the personal data including the height and age of the person, dementia type and the severity, and the gait parameters measured by the GAITRite system, such as walk speed, step lengths of each leg, times of contact, etc. 3D joint positions were not available.

MAX-GRNet for 3D Pose estimation

The first component of our work is the 3D human motion estimation from the 2D RGB video. Like many others, we base our 3D pose estimator on the SMPL [START_REF] Loper | Smpl: A skinned multi-person linear model[END_REF] model, thus the estimated poses are represented in the form of a sequence of SMPL pose parameters [START_REF] Loper | Smpl: A skinned multi-person linear model[END_REF]. Our proposed 3D gait reconstructor, named MAX-GRNet, is illustrated in Fig. 1.

Fig. 1. The architecture of MAX-GRNet, the proposed 3D gait motion reconstructor. P ′ s denote the parameters whereas F ′ s denote the features. LFC stands for locallyconnected linear layers. In LFC, pose parameter of each body joint is regressed by an individual linear layer [START_REF] Kocabas | Pare: Part attention regressor for 3d human body estimation[END_REF].

In our patient video, challenging scenarios can arise due to truncations or reduced visual clarity. To mitigate the impact of truncation and enhance the overall accuracy, we employ the Part Attention mechanism [START_REF] Kocabas | Pare: Part attention regressor for 3d human body estimation[END_REF], encouraging the model to focus on more credible visual features. As illustrated in Fig. 1, the visual features extracted by the HRNet-W32 [START_REF] Kocabas | Pare: Part attention regressor for 3d human body estimation[END_REF] at each frame are fed into the Body Part Attention module, to generate joint-specific features. The subject-camera distance, frequently observed to be fairly long in our video (up to more than 10 meters when the patient and the camera is at either ends of the walkway), can also significantly decrease reconstruction accuracy, leading to unrealistic poses with distorted walking patterns, perturbed foot swings, and illogical step lengths. To tackle this issue, we introduce our Pose Correction Network (PoseCorr-Net), which incorporates several gait characteristics as additional controls and employs the spatio-temporal encoding along with attention mechanism. To address different aspects of the gait motion, we combine average parameters and per-frame parameters as the gait characteristics. The average parameters ensure consistency in the reconstructed walking patterns, while the per-frame parameters accommodate the variations across different gait phases. To regress these parameters, we introduce a GRU-based module named as GaitFeat-Net. The gait parameters P G are subsequently mapped into a higher-dimensional feature space to enable the fusion with the pose feature F ′ P . To further enhance the integration of estimated P G with F ′ P , we introduce a Transformer-based encoder [START_REF] Vaswani | Attention is all you need[END_REF] which we refer to as MAX Encoder, to capture the intra-dependencies across different time steps and among the body joints, respectively, before their final merging.

GaitFeat-Net:

We feed the pose feature F ′ p and camera feature F cam into an one-layer GRU to estimate a number of gait parameters. The effectiveness of gait parameter estimation using a GRU-based network from per-joint 3D position has been previously demonstrated in QuaterNet [START_REF] Pavllo | Quaternet: A quaternion-based recurrent model for human motion[END_REF], in the context of generating plausible locomotion given previous poses and locomotive parameters as controls. Differently from Quaternet, our approach uses per-joint feature as the input, instead of per-joint 3D positions. This allows us to capture more detailed information and potentially improve the accuracy of gait parameter estimation. Additionally, we utilize F cam obtained from camera parameters P cam ([s, t], t ∈ R 2 defined by a weak-perspective camera model) as additional input. As depicted in Fig. 1, we construct a patch of 224 × 224 pixels from each of the initial video frames based on the bounding box, before feeding the patch sequence into the reconstructor. By incorporating the parameters of the bounding box P box , F cam can effectively capture the position of the patient within the original video frame, and the dynamic information regarding the motion in the video. Based on the available locomotive parameters in our patient data (Sec. 2.1), we have selected gait parameters P G = [V, D, Φ], where V = ∥v∥ is the speed amplitude averaged over the sequence, D = [l lef t , l right ] is the average left/right step lengths, and Φ = [cos(ϕ lef t ), sin(ϕ lef t ), cos(ϕ right ), sin(ϕ right )] encodes the phase of the left/right gait cycle.

MAXEncoder: By developing this encoder based on a multi-head self-attention mechanism (MSA) [START_REF] Vaswani | Attention is all you need[END_REF], we aim to overcome the limitations of recurrent models that struggle to capture long-range relationships in the sequence. Fig. 2 depicts its detailed architecture. We devise an attention-based block that builds upon the MSA variants MSA-T and MSA-S of Spatial-Temporal Encoder (STE) [START_REF] Wan | Encoder-decoder with multilevel attention for 3d human shape and pose estimation[END_REF]. To incorporate the gait parameters P G obtained from GaitFeat-Net, and accommodate the joint-wise nature of the pose feature F ′ P , we construct different inputs F T gf and F S gf for our temporal (TAE) and spatial attention (SAE) blocks. Unlike MSA-S, which focuses on modeling the intra-dependencies within each feature map, the proposed SAE explicitly captures the dependencies among each body joint while leveraging the corresponding gait feature. To merge the TAE and SAE blocks, we adopt the parallel connection method as described in [START_REF] Wan | Encoder-decoder with multilevel attention for 3d human shape and pose estimation[END_REF]. Additionally, we replace the Feed-Foward Network (FFN) in the standard Transformer and the Vision Transforer (ViT) [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] with a Joint-Wise Feed Forward network (JWFF). JWFF employs separate linear layers for each joint, contrary to the global FC layers in FFN. Ablation studies in Sec. 3.3 clearly demonstrate the efficacy of such JWFF approach. The total loss of our proposed MAX-GRNet is:

L = w 1 • L 2D + w 2 • L 3D + w 3 • L SM P L + w 4 • L GaitF eat , (1)
where w i is the weight assigned to each loss, more details on these weight values can be found in Sec. 3.2. L SM P L , L 3D and L 2D denote respectively the loss term associated with the Euclidean distances calculated on SMPL parameters, 3D joint positions and projected 2D joint positions. L GaitF eat is designed to supervise the gait parameters regressed by the GaitFeat-Net.

Geometric deep learning for severity assessment

In the second part of our study, we employ the KShapeNet [5], a geometric deep learning model on Kendall's shape space specially developed for skeleton-based human action recognition. It has demonstrated favorable performances on the two large scale skeleton datasets NTU-RGB+D [26] and NTU-RGB+D120 [START_REF] Liu | Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding[END_REF] datasets. Initially, skeleton sequences are modeled as trajectories on Kendall's shape space by filtering out the scale and rigid transformations. Next, the sequences are mapped to a linear tangent space and the resulting structured data are fed into a deep learning model, KShapeNet. Notably, it includes a unique layer that learns the optimal rigid and nonrigid transformation to be applied to the 3D skeletons, thereby enhancing the precision of action recognition. This layer is followed by a Conv Block and an LSTM layer that captures the temporal dynamics of the sequences. A subsequent fully connected block generates the corresponding action class as the output. In this work we utilized only the optimization over the rigid transformation layer.

3 Experiments

Datasets

We utilized Human3.6M motion capture data [START_REF] Catalin | Latent structured models for human pose estimation[END_REF]7] consisting of synchronized 3D joint positions, 2D joint positions, and RGB video frames recorded at 50Hz. We selected 6 subjects to create a set of 79 sub-sequences of walking motions to train and validate the reconstructors. The GPJATK gait dataset [START_REF] Kwolek | Calibrated and synchronized multi-view video and motion capture dataset for evaluation of gait recognition[END_REF] was used for the evaluation. It contains Vicon mocap data with video recordings of 4 calibrated RGB cameras. We estimated 3D poses from their videos, which have been time-aligned with the 3D poses computed from their marker data, allowing us to obtain consistent 2D-3D (video and the 3D pose sequence) pairs. To train and test the classifiers, we combine our patient data with a subset of the Toronto Older Adults Gait Archive [START_REF] Mehdizadeh | The toronto older adults gait archive: video and 3d inertial motion capture data of older adults' walking[END_REF], where we randomly selected 22 walking sequences from 5 subjects.

Implementation details

3D Pose Estimator: We begin by training the GaitFeat-Net module separately on the gait sequences from Human3.6M, to enhance the stability of the overall MAX-GRNet training. Only L GaitF eat in Eq. 1 has been used in this pre-training stage, with gait parameters computed by analyzing the 3D joint positions. We apply L1 loss for V , D and Φ, with distinct weight assignments: w V =250, w D =50, w Φ =100. During the integral training of MAX-GRNet, we use distinct weights for the losses in Eq. 1, specifically w 2D =100, w 3D =100, w θ =60, w β =0.01, where θ and β denote the pose and shape parameters in SMPL. L GaitF eat is applied only for the initial 20 epochs with weights different from the previous training: w V =100, w D =20, w Φ =50, and the subsequent training continues without any supervision on the GaitFeat-Net. The metadata from the electronic walkway only provides the average locomotive parameters in the patient data such as the duration of the left/right gait cycle. Thus, we apply a fast Fourier transform to obtain the spectrogram from the reconstructed gait, to effectively supervise the estimated Φ. Given the average frequency f , the phase loss on the patient data is formulated as

L Φ = w Φ ′ • ( 1 β • A f + β • n i=0 A fi |f i -f |)
, where w Φ ′ =0.05 and β=0.2 are weighting coefficients, and A f denotes the amplitude of f in the spectrogram.

The estimation of SMPL parameters on the Human3.6M dataset is performed by using MoSh [START_REF] Loper | Mosh: motion and shape capture from sparse markers[END_REF] on the available 3D marker data, and each pose parameter is subsequently represented as a 6D vector [START_REF] Zhou | On the continuity of rotation representations in neural networks[END_REF]. The models were trained on 1 Nvidia RTX 3090 GPU using a batch size of 4 and an Adam optimizer with a learning rate of 5 × 10 -5 . Optimal hyperparameters were chosen through grid search.

Motor Severity Assessment: As outlined in Sec. 3.1, we train the classifiers using the estimated skeleton sequences from both healthy and diseased older adults. To obtain the required number of frames for analysis, which has been set to 100 frames, we utilized a sliding window approach. It involved traversing the original video, extracting subsequences consisting of 100 frames each, and maintaining an overlap of 50 frames between adjacent subsequences. The last frames fewer than 50 frames were not used. We refrained from time-warping the sequences in order to preserve the velocity of the gait motion. Each subsequence obtained by this way has been assigned the same label as the original video. This sliding window protocol was applied after dividing the dataset into training and testing sets to prevent the occurrence of subsequences from the same video being present in both sets. Note that this increases the total number of sequences used for training and testing, thereby augmenting the limited amount of our patient data.

Results

Evaluation criteria: The reconstructor model has been validated and evaluated by measuring the 3D per-joint position error respectively on Human3.6M and GPJATK datasets. The classification accuracy has been used to evaluate the overall performance and to compare with other SOTA methods. Different variations of severity assessment tasks has been tested: normal/patient, 3-class diagnosis with normal/Alzheimer/DLB, and 3-class gait scoring (normal-0, moderate-1, and severe-2). All evaluations have used a 10-fold cross-validation scheme. Due to the limited size of the data, we opted to perform a train-test split. Ablation study: We assessed six model configurations es well as VIBE model [9] based on the mean per-joint position error (MPJPE) and the classification accuracy. The design of MAX Encoder has been tested with and without jointwise forward feedback (JWFF). To validate the design of the GaitFeat-Net, we performed three distinct configurations. First, we conducted tests without it, followed by tests using its estimation of only the average parameters V and D (referred to as Avg.), and finally, with the complete estimation P G . The results are shown in Table 1. In general, utilizing gait parameters by GaitFeat-Net and subsequently regularizing the 3D pose estimation improves the reconstruction on GPJATK dataset, which predominantly contains regular walking of young and normal people, but reduces the overall classification accuracy. This somewhat aligns with the observations by [START_REF] Mc Ardle | Do alzheimer's and lewy body disease have discrete pathological signatures of gait?[END_REF], who pointed out that gait asymmetry and variability are significant factors for differentiating between disease subtypes and the patient group from the normal one. JWFF tends to improve the classification accuracy both in gait scores and dementia subtypes, especially when gait parameters are used with it, indicating its efficacy of extracting informative features from the estimated locomotive parameters. Comparison with the state-of-the-art: We compare our method with two closely related studies, which focus on vision-based gait analysis of parkinsonism severity in dementia [START_REF] Lu | Vision-based estimation of mds-updrs gait scores for assessing parkinson's disease motor severity[END_REF][START_REF] Sabo | Estimating parkinsonism severity in natural gait videos of older adults with dementia[END_REF]. Additionally, we include two state-of-the-art models proposed in the vision-based action recognition community [START_REF] Duan | Revisiting skeleton-based action recognition[END_REF][START_REF] Jang | Etri-activity3d: A largescale rgb-d dataset for robots to recognize daily activities of the elderly[END_REF] for further comparison. To evaluate the classifier based on a ST-GCN [START_REF] Sabo | Estimating parkinsonism severity in natural gait videos of older adults with dementia[END_REF], we evaluate their classifier using solely skeletons as input, excluding the spatio-temporal gait features originally utilized in their work. This is due to the inability to compute these features from the skeleton data reconstructed with MAX-GRNet, as it does not provide the root. Results shown in Table 2 demonstrate that our method achieves favorable performance compared to others, and remains competitive with the CNN-based action recognition approach, which demonstrates superior performance in differentiating dimentia subtypes.

Limitations: Our reconstructor struggles to estimate 3D poses in videos exhibiting severe cases where the patient's gait pattern is highly irregular. The classifier incurs an additional time cost and requires separate processing for the projection of skeleton data onto the tangent space, which hinders its seamless integration with the reconstructor in an end-to-end manner.

Conclusion

We have presented a new model aimed at evaluating gait impairment score for assessing the dementia severity using monocular gait videos. Our model features a gait motion reconstructor, which is specifically designed for 3D motion estimation from gait videos based on a gait parameter estimator and a multihead attention Transformer. Additionally, we employ a geometric deep neural network tailored for the specific task of 3D skeleton-based classification. Our method improves the performance over state-of-the-art techniques in both 3D pose estimation and classification, thus demonstrating significant advancements in the field. In the future, we plan to improve the precision of both 3D pose estimation and classification by effectively leveraging image evidences and gait parameters, respectively. Prospect of application. In addition to its clear applications in clinical environments, where the computed gait scores from gait videos can be presented to clinicians, our research could also be used in the realm of care robots assisting elderly individuals in residential settings. Specifically, they could identify early signs or variations in the pathological condition through video analysis.

Fig. 2 .

 2 Fig. 2. Architecture of the MAX Encoder.

Table 1 .

 1 Comparison of performance with different model configurations, measured on mean per-joint position error (MPJPE) (mm), and classification accuracy (%). Numbers in boldface indicate the top-1 performance, with the top-2 denoted as underlines.

		MPJPE MPJPE Normal Normal	Gait Score
		(valid) (test) /Patient /AD/DLB (0,1,2)
	VIBE [9]	70.40	131.52 94.60	71.37	63.69
	Baseline	70.61	103.78 90.11	68.94	62.57
	+ MAX Encoder /wo JWFF	70.35	101.83 88.05	59.28	60.20
	+ MAX Encoder /with JWFF	68.90	106.40 86.76	58.03	65.24
	+ Avg.+ MAX Encoder /wo JWFF 71.13	102.43 85.54	57.74	58.43
	+ Avg.+ MAX Encoder /with JWFF 70.38	102.27 87.36	61.58	59.33
	+ PoseCorr-Net /wo JWFF	66.47 101.78 87.38	66.03	60.37
	+ PoseCorr-Net /with JWFF (Ours) 67.17	103.77 96.22 75.39	65.41

Table 2 .

 2 Comparison of classification accuracy with state-of-the-art methods (% accuracy). Numbers in boldface indicate the top-1 accuracy, while the top-2 is denoted with underlines.

		Normal Normal	Gait Score
		/Patient /AD/DLB (0,1,2)
	VIBE[9] + OF-DDNet[16]	89.60	72.78	60.43
	VIBE[9] + KShapeNet	94.60	71.37	63.69
	MAX-GRNet + OF-DDNet[16]	89.60	69.92	64.68
	MAX-GRNet + ST-GCN [25]	94.24	66.67	72.31
	MAX-GRNet + FSA-CNN[8]	93.02	72.19	66.74
	MAX-GRNet + PoseC3D[4]	92.95	66.59	62.27
	Ours (MAX-GRNet + KShapeNet) 96.22 75.39	65.41

Acknowledgements We would like to thank Dr. Candice Muller and Prof. Dr. Frédéric Blanc at the Robsertsau hospital for sharing patient data and theier valuable expertise. This work has been partially supported by the French national project ArtIC (Artificial Intelligence for Care, ANR-20-THIA-0006) and the binational project "Synthetic Data Generation and Sim-to-Real Adaptive Learning for Real-World Human Daily Activity Recognition of Human-Care Robots (21YS2900)" granted by the ETRI, South Korea. Chaima Zouaoui was supported by the ICube laboratory API project ShaGAI.