meaning, and it's this sensory-based perspective I will be suggesting when using the term. We will see how this thinkering approach has been of particular importance in the empowerment of visually impaired people as described in Chapter 3.

INTRODUCTION

This work accounts on the research I conducted in the last 8 years. In these years I made the deliberate choice to direct my investigations towards research actions with a potential for societal impact. I'm totally aware that this choice orients more towards applied than basic research, and it requires a deep reflection on the theoretical elements that have been structuring my research, giving rise to more personal reflections. In this manuscript I will account for both, the principles guiding my research until now and the practical implementations, trying also to have a reflection on the new elements that have the potential to guide my research in the future.

At the center of all my researches is a Human Centered Design approach to technical and technological solutions. As a matter of fact, in a large number of the projects described in this work this Human Centered Design approach takes the form of Human Computer Interaction and Participatory Design for prototyped artifacts. In recent years, however, a more global socio-technical approach on how to empower users in a real context has been an additional concern of my researches.

In this global context of empowering users through Participatory Design and Human Computer Interaction (which is not a new approach, being empowerment at the origin of the concept of Participatory Design [03]), one of my contributions is the analysis of the role of sensory elements as a part of technical or technology-based artifacts in the learning context. All the researches described in this work relate the "translation1 " of different sensory needs into a system with empowering purposes in a learning context. The answers to these sensory needs are instantiated through different (not always computer based, but always technologically enhanced) artifacts, and inscribed into different interaction paradigms 2 .

The interest of putting the accent on a sensory-based approach is linked with the fact that, while multi-sensory/multi-modal learning is used in offline teaching (for example in the Montessori approach [02]) systematic and reflective thoughts on the topic in technologyenhanced learning are still at their infancy. One particular reason could be that we are evolving inside the third HCI wave (see [17]), where rigid guidelines, formal methods, and systematic testing are no more at the center, nor is "the user" but the "human" creating a more exploratory approach [18] based on cultural, physiological, psychological, (and so on) elements.

Inside this landscape, the works described in this manuscript will show in which way the design of multi-sensory technologies (i.e., not only the sensory translation choices, but also the methods used to collect and interpret them) impacts the results for empowerment. At stake for the analysis are thus not only the objects analyzed and created for humans' interaction, but also the methods used during the researches. It is necessary here to interpret the term method as "the search for a way" [01], a means towards a goal, as a way of moving in between the partial knowledge produced inside our researches, which brings us to admit that this knowledge could not be other than partial and relative, as linked to its elaboration context. This does not means that the illustrated findings are purely relative (I do believe in the existence of an external reality) but that the knowledge generated by them is relatively constructed [12] through the means used for conducting the research, and the people who conducted it. Thus, even if the projects presented in this work use mostly an empiricoinductive approach inducing and generalizing from collected data, during my researches I have tried to avoid the trap of thinking that the model created to describe the particular context and project is a theoretical thesis describing the reality of the world. This reflection joins tensions within the HCI and interaction design research communities regarding the fundamental differences between science and design, and the implications of this difference (e.g., [17] that is the tension between "finding the truth" and "pursuing the non-existent" [19]) for Human Computer Interaction.

In order to pursue the non-existent, two methods are of particular importance for my research: the participatory demarche (in the sense of moving from A to somewhere using a participatory approach), and the thinkering approach. As I will enter into details into the participatory design for empowering users later in this work, I would like to spend here some space to describe why the thinkering approach is particularly relevant for my work. The word "Thinkering" seems to be originated by Michael Ondaatje in his novel The English Patient3 . This expression is nowadays used to describe when we use our hands to engage with something [23] -whether it's putting together a toy, doing the washing up, or even playing with a fidget spinner. When we physically engage with things, especially with our hands, we generate a great variety of sensory images. These sensory images help us understand our immediate reality and help us build personal and shared meaning. Of this personal and shared meaning is part the narrative perspective: how we construct the story of our experiences and opinions has a great impact on the way we think, we feel, and how we stand in front of the world [e.g. , 21]. Thus, even if the term thinkering has been lately used in relation with technology (see e.g., Arduino [22]) it has a more sensory-based narrative As for the first domain, one additional constraint in my researches is the learning context, as the thinkering, sensory, and so on elements have to be relevant for learning through technology usage in different domains. While more in depth experimentations should be conducted in order to assess the real contribution of the proposed affordances and multisensory aspects for learning purposes (I will return on this aspect in the final chapter, addressing the limitations of this work), still initial results shows that a multi-sensory reflection on the design of new technology based artifacts in the learning context seems to be relevant.

In the rest of this section I will give a more general description of the meaning and usefulness of a Technology-Based approach to learning using sensory aspects, leaving a review of current approaches for Technology-Based Learning and their relationship with affordances to each specific chapter, as the proposed solutions are strongly context linked.

As explained by Koller et al. [14] Technology-based learning (TBL) constitutes learning via electronic technology, including the Internet, satellite broadcasts, audio and video conferencing, chatrooms, webcasts, and so on. TBL, thus, today encompasses terms such as online learning and web-based learning adding to the scenario connected objects, augmented visual displays, and the like. This larger view of TBL, that is whatever usage of a technology for learning purposes, is the one I will be adopting in this work.

Attempts to use computer technologies to enhance learning began with the efforts of pioneers such as Atkinson and Suppes at the end of the '60ies. Starting from that moment, the romanticized view of technology in which its mere presence in schools would have enhanced student learning and achievement has been contrasted by the view that money spent on technology, and time spent by students using technology, are money and time wasted [15]. It's agreed upon now, that technology has great potential to enhance student achievement and teacher learning, but only if it is used appropriately. Things are becoming more and more complicated now that learning has moved away from classes, and a pervasive approach (where Tangible User Interfaces [24] play a relevant role) to learning is widely adopted.

As a result, researches in the field still have to assess what is hidden behind the term "appropriateness" for learning in different context.

To partially answer to this "appropriateness" question I used (in particular in the last three years) what Wobbrock and al. call "Ability Based Design" [16]. The idea behind ability-based design consists on focusing on ability -more than dis-ability -throughout the design process in an effort to create systems that leverage the full range of human potential. I think that the concept of ability-based design, because of its focus on empowering what people can do and not on what they are not able to do, can be extended beyond the disability aspect to address all users, creating a very empowering approach.

DESIGN METHODS FOR TEL: USER CENTERED DESIGN, PARTICIPATORY DESIGN, AND HUMAN CENTERED DESIGN During my work I attempted at having a comprehensive approach through all the phases of the project (from analysis to development to test). This means that in creating the learning artifact, I tried to take into account the user's needs (physical and emotional needs, learning needs, and so on), the context (in the larger sense of the space in which the system will be used but also the actors and their systems of beliefs), and the technological panorama at the particular moment of my researches. In order to avoid subconscious biases and assumptions about users' abilities I used at its maximum a Participatory Design approach, in an act of cocreation with the final users. Even if sometimes (as in the case of the post-stroke rehabilitation project I will be discussing in Chapter 1) the co-creation uses a User Centered Design approach, there is an in depth analysis on the reasons why a Participatory Design approach would not be useful in this particular context. As a matter of fact I strongly believe the Participatory Design approach as a huger potential to create an empowering effect.

Participatory Design (PD) is about involving users in the creation of systems in order to make sure that the end object actually serves their needs. From the very beginning PD researchers have been explicit about their concern with the politics of system design [03], and I think that even today is important to be concerned beforehand (i.e., before designing the system) and thus anticipate the possible impacts the system will engender (i.e., if the system will empower or dis-empower the final user in a particular context). A Participatory Design approach could help in detect these possible impacts, anticipating them in the design. I also prefer to use the term Human Centered Design to the term User Centered Design, as both are carriers of different beliefs and actions (which impacts also the methods used during research).

To summarize, the work accounted in this manuscript describes a sensory-based approach for learning in different application fields, work that is framed within the TEL and ability based domains, using a (whole) Human design approach and in particular Participatory Design.

SOME SPECIFIC ELEMENTS STRUCTURING THE WORK

Two aspects that structure my work are the concept of affordances and the multimodal/multisensory elements. The notion of affordance has a special appeal to designers as it offers a way of relating not only objects properties but also values and meanings to designed objects [06]. Affordances have been widely used (and misused) in the HCI field but I still believe they have the potential of structuring the understating of an artifact if used in a functional way.

Affordances

The debate on the role and the meaning of affordances is still high on terms today (as some people talk about affordances as an object in and of itself, while others talk about affordances as properties, or manifestations, of objects). The concept of affordance in the original Gibson's sense is a propriety of the object per sé and not a perceived property, which is apparently in contrast with the phenomenological approach I claim to use elsewhere in the manuscript (see Chapter 3). However Gibson argues [04,05] against a disembodied account of perception expressed in terms of the cognitive analysis of visual stimuli. He argues that visual perception is performed by animals and humans who are situated within and moving around the world, and that this embodiment is fundamental to their visual experience. For the phenomenological approach (and in particular for Husserl as interpreted by Merleau-Ponty [25 p.10-11]) the real is a solid fabric, which does not wait our judgments to annex the phenomena. The world is not an object of which I can possess the laws of constitution, it is the natural environment and the field of all my thoughts and all my explicit perceptions. Thus, the world is an environment where the objects with their characteristics have a meaning only because the man can know himself only inside the world.

In addition, for Gibson, the key element of perception is neither the visual feature nor the object in the world but the affordance, which he defines as a property of the world that affords action to appropriately equipped individuals. The affordance, however, is a property of the world that exists and emerges only in the object-subject relationship and that would not exist if one of the elements of this relationship went missing. An affordance is thus a relational system, and not simply a property of the object.

Perception for Gibson exists only because the man is moving around the world, as for Heidegger perception exists because of the man Being-in-the-world. For Heidegger the world is here, now and everywhere around us (there is thus an external reality). However, we are totally immersed in it, and it is not possible to not being in it. Thus, the distinction between object and subject disappears. As humans (as users and as researchers), we cannot avoid being in the world, moving around it and experiencing it, as a whole.

An applied (and less philosophical) usage of affordances which helped in pragmatically structuring my research for design purposes is Hartson's one. In reaction to Norman's [06] essay on misuse of the term affordance in Human-computer interaction literature, Hartson [07] defines four complementary types of affordance in the context of interaction design and evaluation: cognitive affordance, physical affordance, sensory affordance, and functional affordance. In Hartson's view, the terms cognitive affordance (Norman's perceived affordance) and physical affordance (Norman's real affordance) refer to parallel and equally important usability concepts for interaction design, to which sensory affordance plays a supporting role.

More in detail on Hartson's affordances:

-A cognitive affordance is a design feature that helps, aids, supports, facilitates, or enables thinking and/or knowing about something. -A physical affordance is a design feature that helps, aids, supports, facilitates, or enables physically doing something. -A sensory affordance is a design feature that helps, aids, supports, facilitates, or enables the user in sensing (e.g., seeing, hearing, feeling) something. Sensory affordance includes design features or devices associated with visual, auditory, haptic/tactile, or other sensations. -A functional affordance is an affordance helps or aids the user in doing something (that is functional affordances helps in design for purposeful action)

If we do not lose sight of the fact that these elements are only descriptive and not intrinsic components of reality, Hartson's positioning is pertinent for my researches for the importance he gives to functional affordances, even more than for the explicit introduction of the sensory aspect [20]. I think the usage of the functional affordances is particularly relevant in today's mixed environments where the notion of physical, cognitive, and sensory are highly mingling, like in augmented reality.

In addition to Hartson's four affordances, I use an additional element: the emotional affordance. This concept derived from psychology has already been used in robot/human interaction [08] and in human-machine interactive planning and negotiation [09] but I'm convinced that it could be used in a more extensive way for learning systems.

Emotional affordances are all the mechanisms that have emotional content as a way to transmit and/or collect emotional meaning about any context; it can include bodily expressions, social norms, values-laden objects or extended space, among others. In some way, emotional affordances influence the functional results of an interaction.

I do believe that the above-described affordances are particularly relevant as tools in a whole Human design approach, if intended as an ecosystem. In this ecosystem being in the world with other elements (humans or not) is a non-neutral interaction.

In addition, the emotional affordance allows to go beyond the positivistic approach centered on cognitive and physiological elements as separated controlled entities, giving space for the addition of instability (and potentially the risk of non-reproducibility) in an otherwise positivistic approach to systems design.

While in some way the described affordances can be seen as hierarchical (sensory elements influencing physical and cognitive elements, emotional elements potentially changing the cognitive elements, and so on), and even if these affordances can act as a checklist for the designer (i.e., do not forget to address a particular aspect), they should be seen more like an interconnected web of links. One possible danger of separating affordances is to go back to the old Cartesian/Platonic separation between the mind and the body which is at the basis of the analytical philosophy very often used in the computer science context [10], and from which I believe we should be finally detached.

To explain what I mean with a metaphor, we could see affordances not in a tree model perspective, but in a 3D modeling collapsed or an expanded perspective.

While the second kind of visualization brings a hierarchical perspective with it, with things hidden inside (as could be the "real" properties of an object), the first one do not expose more elements, simply helps the brain to focus on different aspects, without creating a hierarchical structure, in a vision where the links between the parts can or cannot be underlined. During the description of the experiences in the different chapters, I will put the necessary weight on the affordances and their interrelationship (that is, I will talk about one affordance and its relationship with the other affordances), the interaction types they imply in the described system, and the way the global system impacts the context.

Multi Sensory vs Multi Modal

I would like to spend a final paragraph on the usage of the terms multi-sensory and multimodality in this manuscript. The term multi-sensory will be used it in this work to indicate the fact that humans -as animals -perceive the world using multiple sensory modes, including vibration, vision, audition. The multi-modal term will be used to indicate on one hand the means used to translate the sensory needs into the technological and technical systems, and on the other the activities linked with the sensory needs. This is coherent to what happens in the learning domain, where multimodal learning indicates the integration of activities into lessons that utilize various modalities of student learning (i.e. artistic expressions, kinesthetic, visual and audio aids) through high and low technology resources.

To summarize, the sensory-based approach to learning described in this work will be structured in particular through the lenses of the affordance concept, with a particular attention to the multi-sensory and multi-modal aspects.

MANUSCRIPT PLAN

The global approach described in the previous part of this work is a final synthesis of these latter 8 research years. As a matter of fact, each of the researches I conducted contributed or in specifying the research questions, or in supplementing the conceptual framework. The rest of this manuscript, thus, takes a thematic perspective accounting for the evolution of my researches towards the global vision explained in the Introduction. This means that the chapters are centered on a particular issue (for example a body based approach to disabilities) and inside this particular topic I will be detailing the historical evolution of the subjects drafted in the introduction. Figure 1 introduces in a graphical way the different research questions/domains, the time they were brought to the front in my research and in which particular occasion. In next Chapter I will introduce a first set of reflections involving sensory aspects for physical training. Most of the work that will be described is single-user (allowing thus for "easily" isolating the consequences of the usage of certain solutions). What emerges is a first reflection on the usage of different senses and different modalities of interaction for body based systems. In addition, the chapter draws a first observation on the implication of the usage of Participatory Design and User Centered Design in the medical field. Chapter 2 will extend this approach to multi-users evolving systems, complexifying thus the interaction between affordances. The accounted application domains are crisis management and emergency work, and the learning aspects will be anew put at the center. Chapter 3 will describe two attempts to perceptual substitution. The sensory affordances will be the central aspects analyzed, and I will draw consideration on the epistemological stances taken in these latest years in my researches.

The final chapter will be dedicated to a global comparative analysis and the future works part will be dedicated to research opening in the long-term period. In this part I will also draw some initial, general consideration about the appropriateness of the used methods, in the standing of a reflective approach.

CHAPTER 1 -SENSORY AND PHYSICAL

AFFORDANCES IN BODY MOVEMENT

In this Chapter I will introduce a first set of reflections involving sensory aspects in physical training. Most of the work that will be described is single-user (allowing thus for "easily" isolating the consequences of the usage of certain solutions). What will emerge in this part is a first reflection on the usage of different senses and different modalities of interaction for body-based systems, in particular in the case of body rehabilitation. Current approaches to accessible computing share a common goal of making technology accessible to users with disabilities. In particular through the current work we will see a shift from dis-ability to ability-based design as a paradigm for design in my researches. Just as user-centered design shifted the focus of interactive system design from systems to users, ability-based design [31] attempts to shift the focus of accessible design from disability to ability. Finally, following the consideration that the used methods influence the final result of a translated system, I will draw a set of observation on the implication of the usage of Participatory Design and User Centered Design in the medical field.

MIXED MODALITY SYSTEMS FOR POST STROKE

REHABILITATION

The starting point of my reflection on the effects of sensory-based elements on the translation of human needs was the work conducted during the creation of the Rehabcraft system. The Rehabcraft system, was created during the MoJOS (Moteur de Jeux Orientés Santé; Health-Oriented Games Engine) project. As a post-doc I had the possibility to work on it during my stay at LIRMM 5 , in the SMILE 6 team. My researches at the time revolved in particular around two topics: adaptation/personalization 7 and patient's interaction with the system (in an HCI adaptive system perspective). The MoJOS project aimed at creating a middleware dedicated to the design and generation of serious games in the field of health and to provide medical proof that serious games are useful. The middleware had to combine service-oriented computing modules that are interoperable, reusable, customizable and health-oriented and featuring a self-adaptive gameplay (using intelligent artificial agents) according to the learner-player-patient capabilities. As application domain, the project decided to adopt post-stroke rehabilitation.

Stroke is the third cause of mortality and the first one of handicap in adults. It can cause significant disabilities, such as paralysis, language difficulties, and cognitive impairment. These disabilities strongly affect the quality of daily life for survivors both from the behavioral and physical point of view. Concerning this latter point, several researchers on the field have shown that important variables in relearning motor skills and in changing the underlying neural architecture after a stroke are the quantity, duration, and intensity of training sessions (for more information see e.g. [01]). As the effects of a stroke are quite large (ranging from cognitive, to physical, to emotional), the focus was put on upper limb rehabilitation. It's also worth to note that in this case we are talking about re-learning motor skills, where motor skills are surely mapped in the brain -even if after childhood they are used in an automatic way.

In the rest of this part, I will not account for the results of the whole project, but on a particular prototype that initiated my reflection on how multi-modality affects the usage and acceptance of systems for people with disabilities. It's worth also to note that at this point of the reflection I was still in a disability-based paradigm, paradigm that evolved with the following work.

IDENTITY CARD: REHABCRAFT (2010-2013) -Will a game based system increase patients' motivation? -Will a personalized experience (taking into account patient's impairments) increase patient's motivation?

-Are these interactions modalities intuitive and easy to use for the patient?

-What is the added value of therapists tracking?

Working team: 1 Junior Researcher (me), 1 Senior researcher (Abdelkader Gouaich),1 phd student (Nadia Hocine), 1 engineer , 1 master students Spent time: 2 years Related publications: 2 journals, 6 conferences

Learning objectives

Re-learning motor skills after a stroke is a complicated issue as stroke is a highly heterogeneous syndrome. The specific abilities that will be affected by stroke depend on the location, type and size of the lesion. Each patient is thus characterized by a specific combination of deficits [02]. Therefore, stroke rehabilitation programs are strongly personalized and not generic. They are adapted to a particular patient, to regain as much function as possible. The most important gain of function takes place within the first six months after stroke. In most countries, due to high costs, only 6/8 weeks of rehabilitation are executed under the direct supervision of an expert (i.e. in the hospital or rehabilitation centre). In addition, therapy focusing on mobility (recovery of leg function) is the primary concern within these first weeks, so that recovery of arm function is conducted mainly at home [03]. Important in relearning motor skills and in changing the underlying neural architecture are the quantity, duration and intensity of training sessions (see for example [04][05]). To experience significant recovery, stroke patients must then perform a substantial number of daily exercises. In stroke accidents rehabilitation involves thus intensive and continuous training to regain as much function as possible, depending on several factors including the severity of brain lesions and the degree of cerebral plasticity. Unfortunately, typical rehabilitation sessions with therapists include a relatively small number of motions [06] due to the low duration of personal rehabilitation sessions. Home training can increase the amount of executed movements. However, while this is often prescribed, only 31% of the patients actually perform the recommended exercises [07]. As a side effect patients can develop depression or aggressiveness due to the trauma of reduced capabilities. Depression and aggressiveness also imply that stroke survivors may find it difficult to focus on a therapy program, creating thus a vicious circle. This scenario raises the problem of selecting a home-based rehabilitation technology that both can help and motivate patients to perform their therapeutic exercises to ameliorate their recovery after stroke. The challenge for post stroke rehabilitation is then to create exercises able to decrease the monotony of hundreds of repeated motions.

Following the hypothesis that the theories of motor learning can be applied on motor relearning, most rehabilitation techniques are founded on principles of motor learning and skill acquisition established for the healthy nervous system. Thus, patients with upper extremity paralysis typically regain motion starting from their shoulder. Over time, they may gradually regain motion in the elbow, wrist, and, finally, the hand. This progressive approach has thus to be taken into account when creating a system for upper limb rehabilitation.

COMPUTER BASED INTERACTION PARADIGMS FOR POST-STROKE REHABILITATION

In this section I will briefly sketch the state of the art of interaction paradigms for post-stroke rehabilitation when the MoJOS project was conducted.

It is also worth to note that whatever system conceived in the above described scenario has to address at least two stakeholders, the therapist and the patient, who are in an asymmetric relationship, the therapist being the expert guiding the patient towards recovery. I will give some consideration on the impact of this asymmetric relationship at the end of this chapter, when discussing implications for User Centered and Participatory Design.

At the time of the project the most used existing approaches for post stroke rehabilitation through computer based systems were: robot based system, virtual reality based systems, and mixed reality based systems.

The robot based approach for rehabilitation has been used since the late 90s to assist patients (see for example [08][09]). In this kind of system the patient is in front of the robot and her shoulder is strapped to a chair. The patient's impaired arm is strapped to a wrist carrier and attached to a manipulandum. A video screen is above the training table to create a virtual context for movements and provide visual feedbacks to the patients. This approach is very invasive, and, since the patient is attached to the robot, the therapist cannot intervene directly to assist or provide guidance. On the other hand, as the robotic arm is pushing the number movements the patient will do, it assures a high number of repetition.

In the Virtual Reality approach(see for example [10][11]), the patient is normally placed in front or a chroma key green scene and their image is merged, in realtime, with a virtual environment and then projected on a monitor. A camera is used for tracking participant's movements and converting them to interaction events within the virtual environment. In this configuration, the therapist can intervene during the therapeutic sessions. However, the patient sees her image on a monitor, which implies an additional cognitive effort to situate herself in the virtual world. Low Cost devices such as the Sony EyeToy have also been used as an economic alternative for VR rehabilitation [12]. EyeToy is an off-the-shelf low cost gaming device that allows interaction with virtual objects displayed on a standard TV. In this configuration, the therapist can intervene during sessions to assist and provide guidance to patients. The patient is free from any device which makes EyeToy non invasive. However, the patient has to make an additional cognitive effort to situate herself in the virtual environment to coordination her arms and interaction with virtual objects, a not so simple task for people affected by a stroke.

A third paradigm introduced the usage of tabletop interaction. Al Mahmud et al. [13] have developed a social game using a tabletop augmented reality platform, namely Visual Interaction Platform [14]. This platform allows interaction with physical objects that are tracked using infrared-based system. A visual feedback is provided by projecting the display on a table. It is worth noting that this system has been initially used for socializing elder people. Nevertheless, the VIP could be adapted for post stroke rehabilitation. Since the patient is free from any device the therapist can intervene during sessions and consequently the system does not implies changing usual ways of work. The patient interacts with her hand with objects on the table were the display is projected. This makes interaction very natural with both eyes and hands are targeting the same place. Since no device is attached to the patient, the VIP is considered as not invasive. Besides, VIP is a quite large system so a dedicated place is required.

Confronting the three interaction paradigms, we decided to opt for the third one, as it was a carrier of better opportunities, more adapted to the patient/therapist collaboration, less invasive, and source of less cognitive effort for the patient. However, we decided to not to opt for a fixed tabletop based interaction paradigm as the interactive surface for upper limb rehabilitation is dependent from the healing targets of the patients (e.g., small or large movements in 2D or 3D space. In addition, we decided to exploit low cost elements to test the adequateness of the approach to create something that could be used in a home context.

THE ACTUAL PROTOTYPE

In this section I describe the mixed reality system (hereafter MRS) we conceived. It's out of the scope of this work to retrace the global components of the system, however game components that will not be described in this chapter can be found in [X] in the author's publications list at the end of this chapter.

The aim of a mixed reality environment is to merge real and virtual worlds to create a new context of interaction where both physical and digital objects co-exist and interact consistently in real time. The mixed reality system we conceived and developed for this study is composed of three main subsystems: (i) a gaming subsystem responsible for managing the game application and the adaptive elements; (ii) a motion capture subsystem responsible for tracking patient's movement; and (iii) a display subsystem responsible for displaying the virtual game environment in a physical environment.

The gaming subsystem follows a client-server pattern: the mechanics of the game are implemented at the server level. The game server translates this input into game events that are used to compute the next state of the game according to the game mechanics. Once the new state of the game has been computed, update events are sent to the game client(s) to update their local copies of game objects, which in turn update the graphical environment.

The game client fulfills thus the following functions: (i) receiving patient's movement events from the motion capture system; (ii) forwarding these events to the game server; (iii) receiving events from the game server to update game object state; and finally (iv) generating the game's output in terms of 3D scenes and sounds. The MR system described above is able to support different types of therapeutic games, conceived ad hoc to train different types of movements. A module of the game server allows developers to integrate different types of adaptation for every game, offering a personalized gaming experience for each patient.

The motion capture subsystem is responsible for tracking the patient's hand and to communicate these movements to the game client. In the first version of the system (Figure 1), the motion capture system was composed of three components:

1) An IR emitter: this is an active sensor built using a simple infrared (IR) emitter that is attached on the patient's hand using a Velcro strap. This basic device is very cheap and convenient to use even with patients that suffer from spasticity.

2) An IR camera: To track the IR emitter a Nintendo Wiimote was used as an IR camera. The Nintendo Wiimote is an affordable device that has been used in many open source projects. Consequently, a plethora of open source libraries offered the use of Wiimote as pointing device.

3) A processing software: the role of this component is to translate Wiimote events into mouse events so that the IR emitter and IR camera are considered as a standard pointing device from the operating system's point of view. The display subsystem: Within a mixed reality context it is necessary to display game objects directly in the physical world. Within the context of this experiment, a pico projector was attached to a support (the base of a floor lamp with joints, to be able to direct the projection and the capture at whatever angle) to display the game onto a standard table (to let the patient to be seated in case of fatigue). Pico projectors are affordable (less than 200 euros), lightweight (less that 300g), and easy to use by simply plugging a VGA cable into a computer. However, they are limited in terms of brightness (the one that was used in this experiment was less than 30 lumens). This constraint has to be taken into account when building the game's graphical environment by selecting colours and contrasts to compensate for this limitation.

The overall system is presented in Figure 1. The patient holds on his hand (or attach on it) the IR emitter device and sits in front of the table (a) to play the exercise (see also The main idea behind the system was allowing for different devices to be used depending on the patients' impairments and cognitive capabilities. Figure 3 shows an alternative configuration were smallest movements are constraint using a Wiiboard, and differentiated focus (with a large observation space) was enacted by a large-scale projector (Figure 4).

CONSIDERATIONS ON AFFORDANCES

In this part of the chapter I will address the analysis of the affordances, in particular functional and physical affordance, that are the translation of user's needs and enable action in this system.

The usability evaluation of the systems ([VI], [VII],

[VIII] in the author's publications list) did highlight some interesting aspect from the sensory affordance perspective: music in the game wasn't working as a clue or as an ambience creator, but as a distractor in the difficult cognitive and physical task of the patient. However, as I do not have an in depth analysis of these elements (at the beginning of my work I was focusing more on the whole body experience), I will not treat them in depth.

On what concern functional affordances, the evaluation demonstrated that the multi-device approach was potentially successful for physical rehabilitation. A medical validation, showing the actual interest of this kind of system for physical rehabilitation, has been conducted for an evolved version of the system and will be illustrated in next part of the chapter. As for the relationship physical/cognitive affordances we could observe elements already known for the interaction with other kind of augmented systems in post-stroke rehabilitation. While the used games were easily understood and accepted as funny exercises by less affected patients, they created cognitive overload in very affected patients. Thus, the usage of multi-sensory channels with very affected patient is to avoid, and reinforces the idea of a modular and configurable system.

As final consideration, I would like to discuss more in depth the role played by the pair therapist/patient as an indissoluble couple for the system usage and evaluation.

In the first pilot study we held, 4 patients affected by stroke tested a game with a graphical tablet, a mouse and the mixed reality system described above under the therapist observation and help. While we will not enter in detail here, we can summarize results as follow. First of all patients were not able to differentiate the difficulty of using the system from the difficulty of executing the movement, strongly impaired by the stroke. As patients have also coordination problems, having to press a lever can generate spasticity or synkinesia (i.e. uncontrolled movements). In addition the mixed reality system allows for coordination and proprioception (i.e. the sense of the relative position of neighboring parts of the body). In fact, the patient must try to move her/his limb in coordination with other parts of his/her body to reach a target. The cognitive and physical effort of the patient was thus focused on controlling the hand to reach a goal (thus focusing on physical and sensory affordances to achieve functional goals). On the other hand, the clinical eye of the therapist, who assisted the sessions helping the patient for example in avoiding compensation movements, was able to observe for which patient the exercise was more difficult to understand or execute, or when the patient had to put all efforts to move his hand (and thus was concentrating on his hand and not on the screen and the like). To do so, the therapist was taking into account emotional and physical clues.

This case of social interaction during a therapy session raise in my opinion an interesting question on the role of "distributed and differentiated" affordances. In this case we are confronted with in an asymmetric configuration, as the therapist will not focus his attention to the same affordances of the patient, their functional goals being different. Thus, the same objects/subjects most probably do not offer the same affordances to the two users. What this implies for the design and development of a computer-augmented system was a question which remained unsolved by the end of the project, but that is in my opinion worth to explore in the future. -Will a personalized experience (taking into account different impairments) increase motivation? -Does the system and the game allow for a differentiated yet seamless interaction? -Are these interactions modalities intuitive and easy to use?

Working team: 1 Researcher (me), 1 phd student (Geoffre Melia), the developing team from Naturalpad Spent time: 1 year Related publications: 1 journal, 2 conferences

THE INTERACTION PARADIGM AND A TECHNOLOGICAL SOLUTION

From the previous experience it was rather evident that in an asymmetric relationship (such the one between the patient and the therapist) how the different affordances are perceived by each actor and how the actors act together to reach a common functional goal has to be analyzed, designed and implemented in detail. In the particular research I will be describing in this part, we complicated things further bringing another actor to the scene: a healthy player (for example a family member) who could participate in the patient's rehabilitation.

The actual prototype

Hammer and Planks9 , the result of this first reflection with Naturalpad, is a vertical shooter 10 . It consists of a 2D environment scrolling from top to bottom in which the player controls a ship that can move from left to right and top to bottom, and use its cannon to shoot enemies and avoid obstacles. The goal of each game is to defeat all enemies without being destroyed by bullets, reefs, or other obstacles. In this way the player will pass through a series of levels and use what he found to improve its ship. Figure 5 shows a screenshot from the actual game.

The phases of the game are generally intense, short but requiring a lot of concentration.

Enemies and obstacles arrive in large numbers and it is an important part of the game to respond quickly and staying attentive. Also the idea behind the boat improvement is linked to the patient's rehabilitation: as a player he rebuilt his boat in the game, as a patient he do the same with his own body. In this way we tried to address emotional aspects.

From the personalization point of view the doctor can configure game parameters such as difficulty, game speed or areas where enemies should appear through a dedicated web interface (Figure 6 shows this aspect). At the end of a game, the interface allows the therapist to view game statistics such as the number of objectives attained, the missed ones, the time spent in different sectors of the screen as well as the patient's center of mass trajectory during the session. This information will allow the doctor to evaluate the patient progression and to adjust the difficulty for the next game.

A MULTIMODAL IMPLEMENTATION

Hammer and Plank has been developed to be used on several platforms: computer, smartphone, and tablet. On the computer version, it is possible to interact with the game using a large set of controllers: gamepad, keyboard, mouse and some NUI (Natural User Interfaces): Kinect or Wiiboard.

The version developed for the mobile device uses the internal sensors -and in particular the accelerometer -to capture the movements of the player. With this set of interactive devices we were able to propose a large palette of game supports for all the family and for some disabled people.

With this technological approach, we tried to avoid every kind of discrimination towards non-healthy players and at the same time engaging the healthy ones. The main idea here was to avoid developing a diminished version of a normal game to be played by non-healthy players, creating devices that will help whatever person to play inside their ability space. We tested our approach during the Montpellier In Game (MIG), an annual event dedicated to video games, thus in a non formal, non controlled, setting where we performed participant observations.

Our main question during the event was "Will this game really be played by everybody?" (that we can translate into a more specific research question: does the system and the game allow for a differentiated yet seamless interaction?). We thus conducted observations during the event and analyzed the data collected. During the two days of the event we had more than 700 games sessions played by healthy and unhealthy people (knowing that all the players played more than one session). The unhealthy people came because they had heard about the game on the radio and wanted to try it. There were different pathologies ranging from manipulation problems, to strokes, to a quadriplegic person. It's worth to note that sometimes the motivational impact of the fact of being able to play again is strongly underestimated. The fun aspect for someone who has access to everything (from the physical and psychological point of view) is not the same as someone who has access to "something".

In the rest of this section I will briefly report on some of the players who took part to the event whose interactions with the system I find particularly interesting.

One of the players at the event was a child (10-12 years old) who arrived in a wheelchair pushed by his father. The child was suffering from a stroke and was installed on the bench to play with his brother (which generates a small challenge between the two). They played together (or better one versus the other) a couple of game sessions using a gamepad.

A second player who captured my attention was a girl with fine manipulation problems (she wasn't thus able to play with the buttons of the gamepad). She was provided with a tablet (a Nexus7) using the accelerometer to drive the boat. She was very happy to be finally able to play again and the tablet demonstrated to be a good interaction technique for this type of pathology.

Finally, a quadriplegic boy who was barely able to move his hands and thumbs (especially the left one) arrived on the first day. We provided him with a joypad augmenting the sensitiveness of the stick control, and he passed 4hrs playing on the stand. His motivation for playing so long was linked mainly to the fact that he wanted to make the best score but also to the fact that Hammer and Planks was one of the few games he was able to play because of his physical limitations. The learderboard acted thus as activator for competition not only against the others but also against oneself, pushing the player to pass his own limits.

Apart the unhealthy players, the game appealed to healthy players as well.

To quickly explain why with some stats, the person who played the most was a healthy player who played 55 game sessions, followed by the hemiplegic player who played 40 game session. The second observation is on non-distributed/yet distributed affordances in this kind of configuration. I did not discuss in depth in this section the visual and auditory affordances of the game as they are not specific to the developed kind of game, but generic to all vertical shooters (e.g., the usage of visual and auditory feedbacks, the ambient music, the visual configuration, and so on). Still is clear that particular visual and auditory affordance where the same for all the players, and that the players had the same final goal. On the other hand, the previous accounts of experimented devices led to a consideration on the fact that each person created these functional and cognitive affordances interacting with different physical (and sometimes sensory) objects. No actual study was held to evaluate if the different affordances generated by the different devices created a different representation of the system in the players. Still this is a very interesting research question to address.

As a matter of fact, at this point of my researches it was possible to answer positively to the non-therapeutic research questions (Does the system and the game allow for a differentiated yet seamless interaction? Are these interactions modalities intuitive and easy to use?). The MIG experience and other experiments in controlled settings helped to assess more in general the importance of an ability based sensory approach to create engagement. However the other questions (Will a game based system increase patients' motivation? Will a personalized experience (taking into account patient's impairments) increase patient's motivation? What is the added value of therapists tracking?) needed a validation inside a clinical protocol to be answered. The account of the clinical validation aspects can be found in III and IX in the author's list of publications at the end of this chapter.

MONOSENSORY AFFORDANCES FOR BODY TRAINING

More interesting (for the aims of this work) than the clinical validation of the Medimov system is the account of another experiment on the usage of sensory channels in healthcare.

While the previous experiences allowed for a stand towards ability-based design and a multimodal approach, and even if there was an initial understanding of the importance of the sensory aspects for this kind of systems, no systematic research was held on the topic. On the other hand, interesting elements as the cognitive overloading created by music in physical rehabilitation started to give a hint of the sensory element importance. For this reason the following work held on body rehabilitation and training focused on sensory aspects. The work I will briefly describe in this section involved, in fact, the evaluation of sensory elements for rhythm assessment and body training.

Rhythmic skills are natural and widespread in the general population. The majority can track the beat of music and move along with it. These abilities are meaningful from a cognitive standpoint given their tight links with prominent motor and cognitive functions such as language and memory. When rhythmic skills are challenged by brain damage or neurodevelopmental disorders, remediation strategies based on rhythm can be considered. For example, rhythmic training can be used to improve motor performance (e.g., gait) as well as cognitive and language skills. -What is the role of auditory stimuli for motor training? -Which kind of auditory stimuli is more adapted? -What is the role of a game based approach in this particular domain?

Working team: 2 Researchers (Simone Dalla Bella +me),1 phd student (Valentin Bégel), the developing team from Naturalpad Spent time: 1 year

Related publications: 1 journal

Humans display a natural tendency to move, spontaneously or deliberately, to the beat of rhythmic auditory stimuli, such as music [15], [16]. This activity is widespread and is typically participatory. It manifests, for example, in dance, synchronized sport, and in group activities (e.g., waving together at a rock concert). Synchronization to a musical beat is sustained by a complex neuronal network, including perceptual regions, motor regions, as well as sensorimotor integration areas 11 . Disruption of these neuronal networks due to brain damage or neurodevelopmental disorder affects auditory-motor synchronization to a musical beat [17], [18], as well as other functions such as speech. Notably, tracking the beat does not require mandatorily an explicit motor response. Still, mere listening to an auditory rhythm, for example conveyed via music, activates movement-related areas of the brain.

Thus training with rhythmic stimuli may be beneficial to (re)activate the motor system in the damaged and in the healthy brain. In addition, as rhythmic skills are linked to other cognitive abilities such as working memory and reading skills [19], [20], rhythmic training may foster improvements of more general cognitive abilities, which play a critical role in language learning and literacy [21], [22], [23]. Ultimately, these beneficial effects of rhythmic training are likely to have positive consequences for health and well-being, such as promoting an active lifestyle, by reducing motor and cognitive decline in patient populations or reducing the need for healthcare services.

During the last 5 years, studies have focused on the cognitive and neuronal underpinnings of the benefits linked to health-targeted serious games [24]. On top of the physical and physiological benefits associated with serious games (e.g., via dedicated physiotherapeutic exercises), the effects of this type of training extend to cognition. Cognitive functions such as language and memory can also be enhanced by serious games, an effect which is likely to be accompanied by plastic changes of the brain. For example, structural brain changes associated with learning have been observed due to the use of videogames [25]. These promising results indicate that implementing training protocols via serious games may be particularly valuable for enhancing brain functions as well as for therapy and rehabilitation.

The interaction paradigms

In already existing games involving rhythm conveyed by auditory stimuli we can find four different interaction paradigms.

Games that Involve Full Body Movements Recorded via an External Interface (e.g., Kinect, Wii)

Here I refer mostly to dance games (e.g., Just Dance 12 ). These games have interesting applications in physiotherapy for patients with spinal cord injury, traumatic brain injury and stroke [26]. They focus more on physical exercise and activity than on rhythm per se. Indeed, the ability of these games to record and score the rhythmic precision of the player is rather poor. Because these games focus on discrete movements/actions instead of repeated movements (i.e., rhythmic) they cannot be used for delivering specific training of rhythmic skills. For example, Just Dance consists in reproducing movements that are illustrated through images displayed on the screen. The player's score depends on the precision of the movements as compared to a model action sequence. The player has to execute the movements in a given temporal window. Yet, the task is not purely rhythmic and synchronization to the musical beat is not recorded. In spite of the fact that these games do not measure rhythmic skills per se, they provide a motivating setting to perform dance while monitoring the player's movements. Adding a rhythm component to some of these games, as in the case of dance, may be a valuable strategy to translate them into a training program.

Games that Involve Rhythmic Finger Tapping on a Tablet

An example of these games is Beat Sneak Bandit 13 . Here, the player has to tap precisely to the beat in order to make the character progress, avoid the enemies and so forth. This kind of feature is used in serious games dedicated to learning, such as Rhythm Cat 14 , designed to learn music rhythm notation. For the purposes of training rhythmic skills, one major drawback of these games is that the timing precision of the software is very poor. The time window in which a response is considered as good is very wide (i.e., up to several hundreds of milliseconds) and the temporal variability of the recording is high. In addition, no feedback on the rhythmic performance of the player is provided.

Computer or Console Games that Involve Finger Tapping on Keys These games can be played on a keyboard, using a joystick, or on special devices. One of the most famous is Guitar Hero 15 . In this game the player plays on a guitar replica with five keys, and has to push the keys in correspondence of images presented on a screen. Rhythm precision of the responses is recorded and used to compute a performance score. The response must appear in a specific temporal window to be considered as good. The same concept is used in many PC games, but keyboards key (e.g., arrows) are used instead of guitar replica. As in the case of tablet games, the main weakness of these games is their low temporal precision in recording rhythmic performance (around 100ms in Guitar Hero).

Nevertheless, these games are interesting as they represent a good starting point to develop serious-game applications aimed at training rhythmic skills.

Console Games Involving Singing

In these games, the player is asked to sing in synchrony with the music. This is not a rhythmic task per se, but the performance involves a rhythmic component. As in classical karaoke, lyrics are presented on the screen. In this case, a feedback (score) is provided to the player while she/he sings and a final global score is given at the end of the performance, including temporal precision (the response must appear in a given temporal window to be considered as good) but also pitch precision. Here, the potential benefit for health rests upon the fact that singing is a good way to restore speech abilities (e.g., fluency) in aphasia following stroke (for example, see [27]).

All the games presented above involve movement synchronized to auditory or visual cues.

In spite of the fact that some of the aforementioned games present good ground for training rhythmic skills, their main drawback is that the temporal precision of the recording and of the output measures is not sufficient to isolate rhythmic features of the performance (e.g., variability of the motor performance, precision of the synchronization with the beat, etc.). Moreover, in none of these games the rhythmic complexity (i.e. the saliency of the pulse or beat) of musical stimuli has been modified. Varying rhythmic complexity would allow introducing rhythm-based difficulty levels in the game. Difficulty is manipulated only through the amount of responses required during the game (e.g., number of visual tags which the player has to react to), which is not a rhythmical feature.

To summarize, none of the aforementioned games were considered as satisfying for implementing a rhythmic training protocol. For this reason, we devised a new game including auditory stimuli (i.e., musical excerpts and rhythmic patterns), selected based on their rhythmic complexity, and inspired by the most recent research in the neurosciences of rhythm.

THE ACTUAL PROTOTYPE

A serious video game for tablet, Rhythm Workers, was designed to implement a training program for improving rhythmic skills. Tablets were chosen as low-cost and mobile device that can be easily available for patients or children without additional data acquisition devices. Two versions of the game were devised. In the perception version, the player has to detect if a sequence of percussion sounds (a metronome) is aligned to the beat of the stimulus or not. The response recorded is a "yes" or "no" answer. The perception version of the game is adapted from the Beat Alignment Test [28,29]. In the tapping version, the goal is to tap along the beat of the stimulus as accurately as possible. The time of the taps is recorded and the synchronization between the taps and the beat is calculated in real time. Circular statistics are used to evaluate the quality of the performance. Having two different versions of the game is particularly useful, as a purely perceptual training may be sufficient for improving rhythmic skills.

The main aim of the game is to construct buildings. The construction of a building is associated to one of the stimuli presented above (metronome, rhythmic sequence, music), and corresponds to one level of the game. Ninety-nine levels were designed. These levels are divided into nine worlds (eleven levels per world). The aesthetic quality of the building depends on the player's performance (Figure 8). When the player taps accurately to the beat (tapping version) or detects correctly whether the percussion sounds are aligned to the beat (perception version) the building will be better structured (e.g. more symmetrical), richer, and more aesthetically pleasant than when the player's performance is worse. In the perception version of the game, each stimulus lasts eighty beats. Five sequences of tones with a triangle timbre are superimposed on each stimulus at five different moments in the sequence. The first sequence of tones lasts five beats and is an example of an aligned sequence (i.e. the tones are totally aligned to the beat). The following four sequences last ten beats, so that the player has the time to identify if the sequence is aligned or not to the beat.

Aligned and misaligned sequences of triangle tones are separated by four beats where the auditory stimulus is presented without superimposed triangle tones. Ninety-nine stimuli are used in the entire game. The player has to judge as fast as possible if each of the triangle tone sequences is aligned with the stimulus beat. In total, 396 triangle tone sequences are to be judged in the game.

Tapping version

As in the Perception version, a stimulus at a given level lasts 80 beats. After four beats, a regular triangle sequence (metronome) aligned to the beat is displayed during ten beats. This sequence aims at showing the time at which the beat is located in the stimulus, and to which the player is supposed to tap. After this sequence, the player has to synchronize with the stimulus by tapping with her/his finger on the screen. A section of the building is built every 15 beats, based on the performance of the player. Circular statistics are used to calculate the quality of the tapping performance. A mean resultant vector is calculated on the basis of the 15 last taps before a section of the building appears. Synchronization consistency (vector length) and accuracy (angle of the vector) are used to calculate the score.

CONSIDERATIONS ON AFFORDANCES

As we can see, the game was structured from the beginning around auditory stimuli that become sensory affordances to guide towards the motor task. The visual element wasn't part of the game dynamics but acted as motivational aspect. In the rest of this section I will summarize some results and draw some general considerations.

The game scores were higher for participants training with the Perception version of the game, as compared to those who trained with the Tapping version. This suggests that the game is more challenging in the tapping version. Also, the mean time of completion of the game is higher in the Tapping than in the Perception version. This discrepancy between the two versions can be explained by the type of response recorded and by the brain mechanisms linked with rhythm processing. In the Tapping version, the player has to produce a continuous performance across the level, with no possibility of achieving a good score by chance. In the Perception version, the player has a binary choice and is then less likely to perform badly.

The Tapping version implies a sensorimotor integration that engages both a perceptual processing and motor response whereas in the Perception version only a perceptual processing of the temporal information is required. Because of that, we assume that tapping is a more demanding task.

The results of this project are in my opinion very interesting for two reasons.

1) In this project we "isolated" auditory stimuli as sensory affordances, and observed their impact on the physical, cognitive, and functional elements. In the same time, we did not create a simple exercise, but a game that appealed to the fun factor. The experiment raises thus the question of the possibility of isolating sensory stimuli, but still inserting them into an emotional response (in this case the fun and challenge factor) 2) The results of the projects (but this question is raised also by the other projects I will be describing in next chapters) raise the question of how many sensory stimuli the brain is able to handle while solving complex tasks, and how this number impacts learning and training purposes. In the presented experiment, visual, motor and auditory channels worked together to create a rich environment. It goes the same for a learning environment in a normal setting. The usefulness of this kind of auditory approach for medical purposes has been demonstrated by latest medical works of Simone Dalla Bella 16 and Valentin Bégel.

SOME GENERAL CONSIDERATIONS

Table 1 summarizes the different affordances used in the above-mentioned researches. As we can see, some general elements (like the gameplay) were used in most cases to address functional and cognitive affordances, while the devices were used mostly for physical affordances. The emotional elements were most of the time created by the social usage of the system or by the gain in ability (being successful). As for the sensory aspects, sometimes they had a supporting role for physical interaction; sometimes they were at the center of the same gameplay as in the case of the Rhythm Workers game.

Functional affordances

Physical affordances

Sensory affordance

Cognitive affordances

Emotional affordances

Rehabcraft

SOME ANSWERS TO THE INITIAL RESEARCH QUESTIONS

The initial research questions described in the chapter can be regrouped in 3 categories. 1) A first group of questions is linked to patients' motivation (usage of playful interaction, role of the therapist, etc.,) 2) A second group of questions linked to the personalization of the experience. Here, I intend personalization both as personalized content and personalized interactions. This question is in some way linked with question 1, but deserves a specific answer. 3) A final group of questions is linked with the role of sensory stimuli for therapy.

In my opinion, the first group of questions is the one that found the most interesting answers. As seen, each patient and each handicap are different in rehabilitation. The therapist is thus adapting/tailoring her/his actions to the very specific deficit of a patient (sometimes with the usage of tools, sometimes not), while playing a motivational role. Several medical devices developed at the time of the first researches described in this chapter (in particular the robotic ones), tried automating the role of the therapist through the medical device. This kind of approach, while efficient for the number of movements asked to the patient, demonstrated to be unsuccessful on the long run, because of the lack of motivation of the patient. One possible way to supply a motivational aspect was supposed by researchers to be the serious game approach. Several researches, included the ones I conducted, shows that, indeed, a game based approach can act as a motivator. However, the acceptance of experimental therapies (like the ones based on games) passes through the therapist. The observations conducted by Geoffrey Mélia, my PhD student at the time, show well that the faith the patient has in the therapist is the real motivational factor for therapy, regardless of the device used for the therapy. If the therapist believes in the artifact, so does the patient.

The first rule when designing this kind of devices is thus to keep in mind that therapy configuration involves the therapist, the patient, and an artifact, and that whatever advanced artifact will not substitute the role of the therapist. Even better, the artifact (as well as the therapist) is supposed to disappear once the therapy is finished, and any technologyenhanced object is supposed to empower the therapist and the patient in order to obtain their aim: rehabilitation.

Still, there is a need to understand how this kind of empowering object should be designed.

The key answer from the conducted experiments is in my opinion modular difficulty adaptability through affordances scaffolding. As we already know (e.g., 32), difficulty adaptability is the key to successful improvement in any learning aspect (e.g., the zone of proximal development by Vygotsky). How this should be translated in a technologyenhanced artifact has, however, most of the time taken the form of content adaptation. On the contrary, I think that keeping in mind the different kinds of affordances can help in creating a modular adaptive device. Thus, an empowering therapeutic device should allow for cognitive adaptability (scaffolding help in knowing about something), physical adaptability (scaffolding physically doing something), sensory adaptability (scaffolding sensing), functional adaptability (scaffolding doing something) and finally emotional adaptability (scaffolding emotional meaning). As in any scaffolding the elements (as well the global devices), are supposed to disappear once they are no longer needed. The example of MediMoov is a kind of good example of this rule (although the platform has room for improvement). The platform has a space for cognitive adaptation (the set of rules that are presented to the player can be more of less difficult) that is influencing functional elements (the rules influence the difficulty of the task). Finally, the required tasks can be scaffolded by different devices (physical adaptability). Emotional adaptability in this case is played by the therapist who helps the patient become more and more independent through success (i.e., the emotional feedbacks given to the patient are the same during the therapy).

As for the third group of questions, the ones linked with multisensory interaction, a the moment of the projects presented in this chapter there was a tension between the opportunity to use multiple senses (for example in the video auditory game) and the nonopportunity to use them (e.g., in the therapeutic games in which music was not working as a clue or as an ambience creator, but as a distractor in the difficult cognitive and physical task of the patient). A first initial answer could conduct towards the adaptability of sensory clues, in particular for rehabilitation. However, we will see in next chapter that a more holistic approach is needed.

SOME GENERAL CONSIDERATIONS ON DESIGN AND EVALUATION TECHNIQUES AND METHODS FOR GAMES FOR HEALTH

Designing and evaluating in the healthcare field generates very specific constraints. In all the projects described, we tried to initiate a Participatory Design approach, even if for all of them we ended up with a User Centered Approach (i.e., we had to move from co-design session to users' validating sessions).

One reason for the inability to put into practice Participatory Design in this context could be linked with the role played by the different Cultures of Participation of the different stakeholders.

The health field links many actors with many complementary roles. At the centre is the patient, the person whose health will be considered. Around her, people whose numbers and richness of roles vary gravitate. Doctors, specialists, auxiliary nurses and other therapists, spouse and family are all people who interact with the patient, and can influence in one way or another (i.e., more or less directly) her health. The scenario becomes increasingly complex when we add to the scene also the professionals who will develop the games for health. While this inclusion allows for taking into account not only the health aspects but also the playful mechanisms linked with video games, doctors and game developers rarely share a common Culture of Participation (i.e., a similar way of defining the role of participation of external stakeholders in their particular field). If we consider Participatory Design in terms of who should sit around the table during the design phase, we are tempted to look only at the primary circle represented in Figure 9: the patient, the healthcare staff, and the game designers. However what happens is more similar to the secondary circle. Sitting around the table are not only the primary stakeholders, but also the beliefs they have, their cultural background, their habits in participation, and we could go on adding the sociopolitical and technical structure (such as the hospital, the game studio where the game will be developed and so on) surrounding each of these actors.

Medical institutions and services -through their health personnel -constitute complex systems. They include highly specialized knowledge and skills aimed to treat the higher number of people possible. They form what can be called an expert group, using their special skills on a second group, the laypeople (patients and relative), which is inherently in a subordinate position: this second group is requesting care and is usually devoid of health competencies. The relationship between the health personnel and patients and families is thus an asymmetrical one. While this situation tends to evolve, still is not possible to say that the different stakeholders can share the same Culture of Participation 17 .

In this scenario, the faith the patient has in her doctor could influence the acceptance of the therapy. During the cited project on the usage of mixed reality and games for post stroke rehabilitation, the author of this paper heard more than once (from patients not used to playing video games) statements such as: "if my doctor says it is useful, then I will do it". The acceptance/resistance to the usage of the proposed tools was thus subordinated to the confidence relationship between the patient and the doctor. In this asymmetric relationship it is essential to facilitate and improve communication between stakeholders in order to create tools to help achieve the shared goal of care. This communication is particularly important during the design and evaluation of IT tools to create devices that embody the needs of both groups. Donnellon et al., [30, p.44] explain that when groups of participants do not have «shared meanings» or shared interpretations, they may engage in coordinated action by engaging in a set of communication practices that enable them to create "equifinal meanings." Equifinal meanings, they explain, are "interpretations that are dissimilar but that have similar behavioral implications." Achieving equifinal meanings does not require that participants achieve equal or overlapping understandings but rather that they develop a set of complementary understandings that lead to coordinated actions. One possible track to explore could be to initiate mutual understanding not in a rational (through meetings sharing in a formal way languages and practices) but in an emotional way. I will return on this aspect during the discussion of the last set of projects.

CRISIS MANAGEMENT

The work described in previous chapter was single-user (allowing thus for isolating e.g., the sensory aspects and thus the consequences of the usage of certain solutions). What has emerged is a first reflection on the usage of different senses and different modalities of interaction for body based systems. This chapter will extend multi-modal and sensory interaction to multi-users evolving systems, complexifying thus the interaction between affordances.

The artifacts that will be described were designed and developed inside a European project called Mirror 18 , whose aim was to create applications to improve reflection on work practices. The TeseoLab 19 in which I was working at NTNU (Norwegian University of Science and Technology) decided to work in collaboration with Italian Civil Protection to develop learning practices involving reflection. Protecting the population during large events, emergencies, and disasters is a highly complex effort that requires coordination of different competencies, often under time pressure and in challenging environmental conditions. Training of workers in this field is thus very challenging and has to take into account the need to learn not only specific skills, e.g. how to operate specific tools and how to behave during different types of events, but also appropriate communication styles, stress management, coping strategies and so on (soft skills [01]).

It's worth to add that researchers have identified "situational impairments" caused by changing situations, contexts, and environments, using the language of disability and accessibility. Sears and Young for example [15] said: "Both the environment in which an individual is working and the current context . . . can contribute to the existence of impairments, disabilities, and handicaps."

In order to address the challenges of creating a learning environment able to train in an evolving situation (as the crisis management one) as well as the situational impairment, we explored the usage of tangible and mobile collaborative serious games.

TANGIBLE INTERACTION AND MULTI-SENSORY CHANNELS IN CENTRALIZED INTERACTION

In this section I will present Don´t Panic, a game aiming at training soft skills for the management of situations where diffusion of panic might put population at risk. Don't Panic is a cooperative board game inspired by games such as Pandemic 20 and Monopoly 21 . Each player starts the game as member of a panic control team that must work together to calm down people, preventing the biggest panic event humanity has ever seen.

During the game session different potential panicking events will take place in the city represented in a board. Each player assumes a unique role within the team (firefighter, doctor, and the like), with special abilities that improve the team's chances if applied wisely collaboratively (in an orchestrated way). Each player gets a limited number of actions to spend on her turn so that the players have to think wisely how to use the actions they can do. The players have a limited time to calm down the situation, before the panic will spread and they will lose the game.

Learning objectives of the game. Don't Panic wants to teach communication styles useful to manage crisis events but also foster actual team building (that's why the game is a collaborative one). The game was conceived to push local vs. global reasoning, problem dissection, and making plans dividing the board game into zones and adding unpredictable events during the game, which can create contrasting reasoning and priorities. In addition, information linked to a zone is visible only when a player inside the zone, in order to make the game similar to an actual situation. As personalization and tracking of the learning sessions of the game are important aspects, we decided to develop a computer-based version of the board game 22 . Playing board games is an engaging social experience characterized by two levels of interaction: between the players themselves (e.g. discussing strategies), and mediated by physical artifacts representing information and actions (roll a dice, draw a card). Such rich experience is facilitated by the social and physical affordances of the principal elements in common to any board game: board and game pieces. While sitting around a board affords for face-to-face and gestural communication and cooperation, game pieces allow for tangible interaction and physical feedbacks. In a more formal way, boards games pieces embody control and representation. For example: pawns serve as a visual representation of players, shared items (e.g. houses in Monopoly) as a representation of a resource count. The action of rolling a dice or drawing a card act as a control for a (random) variable, allowing the game to evolve from a representational state to another. Each game piece can serve one role (as in Monopoly), or both (as in Chess). As matter of fact pieces in board games are used to convey both static and dynamic information: for example players' identity and role don't change throughout the game and are often represented by a set of distinguishing pawns (or tokens), while resources or score associated to each player vary and they are usually represented by a number of shared artifacts (e.g. houses and hotels in Monopoly). Moreover the spatial configuration of game pieces on the board provides the players with a shared awareness of the state of the game. A board game is thus a very complex system with complex iconic, symbolic, evolving elements that need adequate translation into a computer based system.

INTERACTIONS PARADIGMS IN IMPLEMENTING BOARD GAMES

From an historical point of view, we can identify two main paradigms used to implement board games. In the first paradigm we witness a porting from physical board games to completely virtualized computer games. In this paradigm, keyboard and point-and-click interaction replace physical actions around the board. It is worth to note that once implemented on a PC, even if the game offers a multiplayer mode, the result is two or more players sharing keyboard and mouse in a turn-by-turn, co-located but screen-mediated interaction. This approach doesn't facilitate face-to-face interaction (it's rather a shoulderto-shoulder interaction) [01] and simultaneous actions (2.a in Figure 1). With the years, the evolution of computers fully virtualized digital board game gained higher-definition graphic and sound (and the usage of Artificial Intelligence). Still, the translation to the digital domain lacked the social and physical affordances of analog board game: the lack of tangible interactions impacted the game experience [02], and the computer or the console was still acting as a mediator [03] impacting the social experience.

At the beginning of this century, tabletop computers (large horizontal touchscreens) have been considered an ideal platform for digital board game development [04], able to mix some of the advantages of low-tech board games with the benefits of video games [05] (2.b in Figure 1). Indeed, sitting around a tabletop computer allows users to be closer to the digital information and at the same time it enhances collaboration and communication among the users [06], which are fundamentals of traditional board games. Whilst the direct manipulation of virtual objects supported by vision-based systems and touch-screens makes these games more similar to "analog" board games and allow for a face to face social experience, the resulting global experience is still different from the three-dimension sensory feedbacks experienced by playing with dices, pawns and cards. For example, we still miss physical objects allowing for peripheral interaction during the game and permit passive players (in turn-based games) to manipulate game pieces as long as they don't break the rules [07]. In interactive-surface implementations of board games, technology is usually employed to virtualize pieces' representations by means of computer graphic and sound. The player's physical interactions with game pieces are often substituted with traditional GUI metaphors. For example the actions of rolling a dice or drawing a card are implemented in touchscreen gestures like pushing a button or pinching a virtual dice. Still, it's not possible to do simultaneous actions, manipulating other players' objects and so on.

The paradigm change arrived when, to address these limitations, game designers started combining the touch-based interaction of tabletop computers with interactions through physical objects placed on top of the screen surface as means for controlling virtual game elements. In this way conventional objects can become game pieces (i.e. pawns, cards) by attaching active or passive tags recognizable by a computer vision system (3 in Figure 1). The LCD screen becomes an interactive board capable of graphical and auditory stimuli and reacting to touch-inputs and manipulation of objects tagged with fiducial markers. Although this approach has become mainstream, it confines interactivity to a touchscreen area, it is often significantly less portable than a traditional gamebox, and it poses a tradeoff between span of the touchscreen surface (and thus the interactive space) and costs.

An in depth analysis of these interactions paradigms can be found in X of the author's publications list.

Thus, the main challenge in our work was trying to keep the sensory affordances of a physical board game and in meantime benefit of technological advantages (for example: tracking, configuration, and so on), without losing any affordance (nor cognitive, nor sensory).

To avoid the limitations of the second paradigm we decided to go for an alternative approach. Rather than implementing game dynamics using interactive surfaces, we relied on physical manipulation of technology-augmented pieces on conventional surfaces; the interactive area is thus not confined by a touchscreen size. In our approach thus (which involves different -more or less intelligent -objects communicating between them), the game pieces are the means to bring interactivity and not the board per se, which on the contrary works as a constraint to action. Distributing interactivity across multiple components, we achieve a wider space of possibility and a higher degree of flexibility in shaping the game experience. Also, in this way the interactive area of the board is not limited by size, which also determines the portability of the game (and cost). Designing technology-augmented game pieces enable thus players' simultaneous interaction not just on the board but also over and around it. Game pieces might thus still preserve their traditional aspect, having a tangible representation that complements an intangible or ephemeral representation provided by technology. As in Beaudouin-Lafon [16] instrumental interaction, the objects/game pieces form the set of potential objects of interest for the user of a given application.

To formalize these elements we leveraged Ullmer et al. Token+Constrain framework, as board games have been inspirational for the token+constraint approach [09] [10] (for more information on the actual approach used for Don't Panic implementation see I and II on the author's publications list at the end of this chapter).

The T+C paradigm focuses on the use of tokens and constraints as means to trigger digital operations, physical artifacts also characterized by their physical appearance.

From an architectural point of view, a digital model (e.g. a computer game engine) of physical objects (e.g. game pieces) is controlled by players' manipulation of augmented game pieces. Each artifact has a tangible representation (i.e. shape, color) that identifies the piece and defines its affordances; in addition it might have an intangible but sensory bases representation (graphic, auditory), controlled by the digital model, that is updated anytime the manipulation of an piece with control powers pushes a change in the model. The interaction with pieces is based on a double loop [12]. A first interaction loop consists in the passive haptic and visual feedback the player perceives when manipulating tokens on the board. A second loop adds interactivity by means of graphical and auditory feedbacks conveyed via the tokens' intangible representation. This loop requires technology for sensing tokens' manipulations as well as providing visual/audio feedbacks (Figure 2). The actual implementation of this approach on the Don't Panic game can be seen in Figure 3 Figure 3. The final Don't Panic prototype A video explanation of the system can be found here: https://vimeo.com/95992690 The card deck (Figure 4-center) -is an active token capable, in turn, to print information card tokens. The card deck is crafted in the form of a wooden box that encloses a thermal printer, a CCD barcode scanner and a Raspberry Pi (which also runs the game engine). The card deck allows for printing and recognizing cards during a game. Each card has a textual description of how it affects the game and a barcode that links the card to its digital representation. The top surface of the card deck can identify by proximity cards that have been previously produced and trigger actions in the game. Cards don't affect game dynamics immediately after they are produced; they can be kept or exchanged by players, until when they are activated by proximity with the card deck, anytime during a game. Each printed card displays text, graphic and a distinct barcode that is used to link the physical card to its intangible representation stored in the game engine. The wooden box also contains speakers to create global sounds. 5-right) -are active tokens that embody the players' presence on a node, each player interact with a personal pawn during a game. Pawns can be dragged from node to node, as long as a path directly connects the two. Pawns, and intelligent tokens (described later) have been implemented using the 1 st generation of the Sifteo cubes 23 . Each cube is capable of sensing accelerations, proximity with other cubes on any of its four sides, and to display graphics on the top surface. In order to make the cubes recognize discrete locations on a board (required to use the cubes as pawns reacting to nodes constraints) we exploited in an unconventional way the data from the 3-axis accelerometer embedded in each cube. We designed sockets for the cubes each of them featuring a combination of unique horizontal tilt angles over two axes; the aggregated value of tilts angles is used as a fingerprint for the socket. We 3D-printed and embedded the sockets into the board as nodes constraints (Figure 4-left) and we coded the relation between sockets fingerprints and nodes' location. In this way when a player associates a pawn to a socket, the cube senses the surface tilt over 2 axes using the accelerometer, the aggregated value is matched against socket fingerprints and thus the position of the pawn on the board is updated in in the game engine.

In this way we embedded the technological constraints into the gameplay, so that players get the suggestive impression of being positioned on a building while exploring the map.

Each pawn provides static and dynamic information via a LCD display. The static one shows icons that link a pawn to a specific player by her role in the game. The dynamic representation visualizes the number of people present in sectors adjacent to each of the four pawn's sides and their panic level (symbolized with colors). This information is contextually updated according with a pawn's location on the board, since different nodes face different sectors. Besides their representational functions pawn also have a control role: in order to activate nested actions with other tokens on single sectors the player has to reach a node set on the sector border.

3D printed Barriers (Figure 5-right) -are passive tokens that resemble iconic artifacts. A QRcode on the bottom of these 3D printed artifacts tells the game engine which role they should play in the game.

CONSIDERATIONS ON AFFORDANCES IN THE DON'T PANIC BOARD GAME

In this part, I will discuss Don't Panic through the lenses of the affordances classification introduced in the Introduction. As explained in the introduction, I will not strictly separate the different kind of affordances, but I will discuss them as loosely coupled items, underlining interrelationships between them.

Cognitive affordance and Functional affordances. In our case, cognitive affordances are linked to learning and game element. Field observations and literature review were conducted to collect the information to insert in the game in order to have meaningful actions. First of all the game rules are studied to push the player to put into practice the "best practices" linked to soft skills for crisis management. Secondly the content of the game reflects real life information and events linked to crisis management.

Before starting with the actual implementation of the technology-enhanced game we conducted a first pilot study (using a physical low technology version of the game-Figure 6) with crisis and emergency workers to validate game dynamics and elements. The idea was not only to validate the rules and the content of the game, so that they will be compliant with real life practices, but also the need for different objects in the game. This Don't Panic version was thus tested with 10 Civil Protection experts belonging to different organization. In between our participants we had maxi emergencies coordinators, dog handlers, and medical emergency experts. Details on the game design and this evaluation can be found in XI in the author's publications list.

To summarize findings for the purpose of cognitive and functional affordances analysis, this co-located, low-tech board game version has shown that content and game dynamics engaged aspects linked to soft skills: a high level of conversational exchanges, collaboration between players, a push towards quick communication styles. The whole gameplay pushes towards a reflection on past events. In fact most of the participants used previous experiences in real settings in order to better manage the current game and help less experienced colleagues.

The token-based approach prototype was tested with 16 players (10 who tested the previous version + 6 new players). For what concerns the social dynamics, our evaluation highlights that: a) the introduction of technology didn't alter the traditional social affordances of board games. b) Even if the interactive tokens were richer in terms of actions and feedbacks than traditional game pieces, this choice didn't disrupt the flow of actions in the game.

Physical affordances. Physical affordances can be seen as a bridge from one hand to cognitive affordances, and to the other to sensory affordances. In the first case, physical affordances work as stimuli for cognitive reasoning. In the second they aggregate the sensory elements. As for the validation of these aspects for the first low-tech prototype, manipulating a barrier or a pawn, passing a card, discussing, and so on, help in recollecting information, deciding strategies, ask for other players help. In the first low-tech pilot study, some physical elements of the game were considered too difficult to manipulate (for example the low-tech version of panic displays, rotating round circles with numbers -see Figure x, right), and quickly led to information overloading and difficulty in strategy management. Other objects where useful to trigger different levels of reasoning (the physical cards for example, which led to coordination and prioritization) because of the possibility to exchange, classify and throw them.

As for the token-based prototype, game dynamics were successful implemented through sequences of interaction events. The learning curve was very low for most of the players due to the interactive tokens acting as memory helpers for the game dynamics. From the sensory point of view, by adopting a card printer, we were able to mix the powers from the digital domain, to sort and select a huge number of choices, yet preserving the physicality of tangible interactions with cards, their flexibility of manipulation and extended visibility.

For what concerns the Pawn LCD displays, although providing a quick awareness of the game status, this design has been perceived as overloading and confusing. We used a discrete color-scale instead of digits to symbolize ranges of values (panic levels). Being the information only updated when a threshold is reached, most of players experienced this design choice as a frustrating lack of feedbacks from the system. Though this is a general HCI problem, it takes a different connotation when using a TUI approach, which poses stricter limitations, compared to GUIs to the design space.

Emotional affordances

Results of both studies show that the game was able to foster also strategy/creativity and excitement/stress (elements that appeal to the emotional sphere). The blend of elements taken from the digital and analog worlds resulted in added interactivity and fun for the players. For example, we observed that game cards printed "on the fly" brought elements of excitement and surprise due to the players hanging on while a card gets (slowly) printed. The physicality of cards also allows for playful interaction not conventionally available in traditional board gaming: card can be annotated, kept by the players for future reference or tossed to relieve stress during a game. Audio elements were particularly useful not only to convey rich information, but also to create excitement/stress. Apart from local sounds linked to Pawn actions, general soundtracks were used to create the sense of an evolving situation, for example playing distressing music when the situation was going out of control (mimicking a technique usually used in movies, strategy which helped in creating an augmenting sense of necessity for action.)

DESIGN AND EVALUATION TECHNIQUES AND METHODS:

During the project multiple methods where applied. Apart from classical field observations and literature review, what I find most interesting for this particular case is the usage of multiple prototyping as a way to communicate with stakeholders. The usage of prototypes as communication objects in Human Computer Interaction is not new. However I find a particular anecdote that happened during one of the studies particularly revealing of the actual role played by the prototypes. While conducting the first pilot study with the low technological prototype, we left around the room some colored gummy candy for the players to eat. As we can see from the photos, the first prototype wasn't particularly rich in colors. One player started to use the candies to add visual clues to the game. This action, and the paper affordances of the game, allowed other players to manipulate and adapt the game to their needs using what they could find around the room, leading to an impromptu co-design session during the game round (and some interesting considerations on visual clues).

TANGIBLE INTERACTION AND MULTI-SENSORY CHANNELS IN DISTRIBUTED INTERACTION

After the first pilot study of the Don't Panic board game, I started to be convinced of the importance of actual physical interactions for learning purposes, in particular in a technology augmented context. I was in particular convinced of the importance of multisensory elements (mixing audio, visual and tactile clues) and bodily interaction (moving around, exchange objects, and the like).

The mobile game I will be describing in this section, MoDo (for MObile DOn't Panic) has been developed with distributed action and coordination in mind (as a complement of the board co-located version);

MoDo is structured to be played in teams in a physical environment through the usage of mobile devices and technology-augmented objects (see Fig. 8). The teams have a limited amount of time to complete their missions using the resources (such as augmented hammers, chains, and the like) they are provided with. The game starts with a particular situation in the zone -for example a certain amount of wounded people, panicked people, collapsed walls, and so on -and the teams have to explore the territory in order to save people. However the players are able to see what the situation in a zone is, only by being in proximity of that zone (e.g., they will see the number of panicked people in a room only if they are near that room -see Fig. 8-right). Each team has to "collect" and evacuate the maximum number of people in a limited amount of time. To do so the team has to bring the "collected" people back to a safety point outside the building/zone, without putting themselves in danger (n°1 rule in whatever safety procedure). The game is conceived so that only few key points are fixed: (i) the resources usable by the teams are limited; (ii) there is a limited time to complete the missions. All the rest of the game is linked to emergence dynamics (e.g., when and in which way the players use the resources, if they communicate/coordinate or not, and so on.) as the game was conceived to test already known practices. For this reason an external observer (a trainer) will observe the actual phase of the game. In addition, all the movements and events are tracked through the mobile phone so that they can be used in a debriefing phase.

Learning objectives of the Game. MoDo has multiple aims linked to soft skills and best practices. The missions inside the game are conceived to push local vs. global reasoning, problem dissection and making plans as dividing the game arena into zones and adding unpredictable events during the game that can create contrasting reasoning and priorities. The content of the game reflects real life information and events linked to crisis management. All the actions that the players are able to use within the game were discussed with experts. Also the choice of which kind of tool to augment was decided following experts' interviews (see the methods and techniques subsection). As the game is played in a real environment In their paper "Towards a Theory of Mobile Learning" [11] Sharples and colleagues distinguish what is special about mobile learning compared to other types of learning activity. An obvious difference is that mobile learning starts from the assumption that learners are continually on the move. We learn across space as we take ideas and learning resources gained in one location and apply or develop them in another. We learn across time, by revisiting knowledge that was gained earlier in a different context, and more broadly, through ideas and strategies gained in early years providing a framework for a lifetime of learning. We move from topic to topic, managing a range of personal learning projects, rather than following a single curriculum. We also move in and out of engagement with technology, for example as we enter and leave cellphone coverage. As we can see in this approach it is the learner that is mobile, rather than the technology.

While in current mobile learning it's clear that the learner is mobile, the accessed content is still rather classical: the learner gets information through the mobile device, that is also the means through which the user interacts with the content. On the other hand in games such as Can you see me now 24 and Uncle Roy all around you 25 , the context is the content. The people playing the game and the place the game is played in are strongly part of the game dynamics (i.e., the actions the player can do), and actions do not necessarily happen through interaction with the mobile phone. As we were looking for a low cost way to implement situated learning for crisis management we decided then to design a mobile game, augmented with smart objects.

THE ACTUAL PROTOTYPE

The MoDo system consists of a website which is used by an expert for the initial configuration and the post-game debriefing, plus a mobile application. On the web page it is possible in particular to create a dynamic map (i.e., which zones of a pre-mapped building will be used, creating thus an abstract representation of an actual place), decide which kind of object will be possible to use during the game, and if or not a briefing message will be given before the beginning of the game, stating for example how many people are trapped inside the building. This kind of configuration allows to modify the learning objective from one session to another. For example, if the main objective is territory exploration the map would be larger, if the main objective is learning how to manage multiple difficult situation, the number of people can be higher, and so on. As the mobile phone acts as a tracking device (see Figure 9), once the game is finished the website will present different types of statistics (see Figure 10). In order to validate the game concept before the implementation we conducted an expert evaluation with 4 crisis and emergency experts. Two of them were civil protection leaders, the other two came from the industry as sellers of emergency software (one of them in particular is a volunteer firefighter with a long experience). We used scenarios and low fidelity mockups to discuss about the game. This expert evaluation allowed us to validate some game design aspect (such as the usage of the augmented objects) and to modify some others (like the kind of movement patterns the players have to follow in order to save a trapped persons).

The experts considered the whole game as a possible successful training means and the scenario was easily accepted as realistic. In particular comments went into the direction of hints about how to stick to real life procedures. For example we were explicitly asked to add in the gameplay the necessity to bring people in a common room for a triage operation. Also the importance to find a way to keep track if the augmented object were used in the right sequence was underlined. We can thus say that this evaluation validated storyboards for implementation, and underlined the need to stick to practices.

The actual system was tested in three iterations, with a total of 15 people. Shadowing techniques and post-game debriefing were used in addition to tracking to discuss the results of the sessions. Each iteration allowed improving the system usability and stability. However, the results of these three iterations were not able to validate the prototype affordances and interactions as means to initiate learning. In the rest of the section I will discuss some of the reasons that in my opinion led to his lack of validation (for a full account on this experiment see VIII of the author's publications list).

Cognitive, Functional, and Emotional affordances impacting learning elements.

In this particular case, the game wasn't very successful to create the physical and sensory elements needed for functional and emotional affordances, and for engaging in learning purposes.

For example, the players discussed loosely about their tactics at the beginning of the game, as they were more interested in the technological gadgets. One of the players said the right moment the game started: 'I will just go over there and take a look'. In all the sessions, players started spreading out to different sectors without coordination, and after a short period of time the status of the game was on red alert. While learning by failure can be considered a kind of learning, the system structure did not allow for a full exploitation of the learning potential. No quick learning was observed, no emotional reaction that could lead to better remember events, and so on. On the emotional level, time pressure during the game was used to simulate stressful situations, and provided a certain level of engagement. The time constraint led the players to run (which could let think at a flow state and physical engagement). However no other elements of the game play were cited as source of engagement.

As a matter of fact this system was rich from the cognitive and sensory point of view (rules, augmented objects, exploring the environment), but without any emotional dynamic that could lead to engagement. Rich of the experience of this not so successful experiment, and convinced despite all the mobile interactive paradigm was appropriate for learning how to behave in a territory purpose, I decided to design another mobile game which will be the detailed in next section.

-Analyze the role of physical exploration of space for learning Working team: 1 Junior Researcher (me), 1 master student Spent time: half a year Related publications: 1 conference

THE ACTUAL PROTOTYPE

As flooding is a territory-based problem, being able to support this aspect becomes really important in order to have a good sensitization approach. Flooded is then a location-based mobile game to be played in the player's local territory. The game world is the real world represented through Google maps (and not a simplified mapped space like in the MoDo game). In the map the players can see their own position, game objects, and the position of other players. The game is consists of three different phases. The first phase represents the time just before a flooding, the central phase depicts when the flood is hitting, while the final phase represents the time after the flood has hit. The players will have to help predicting and managing the flood before it hits, can see what happens when a flood occurs, and then they will have to act in the flooded area. The game has also a messaging functionality for facilitating communication between the players during the current game session.

The quests more in detail:

The game features four different quests used to teach different aspects of flooding preparedness, and based on real world practices. Quest 1: Flood Protection. In areas with a flooding history it is vital to learn how to protect the area to minimize the possible damage. In the Flood Protection quest the players must collaborate to place sandbags at key locations to protect the area (this means that they will have to physically walk through the area in order to place the sandbags). The main learning objective of this quest is to teach the player not only that placing sandbags is a suitable strategy for flood protection, but also that a good sandbag placement can help to improve survival possibilities. In order to obtain this learning goal, the intensity of the flood happening in the second phase is dependent from the player's performance in this quest. Players will then be able to see the consequences of good or poor flood protection, depending on how they performed in this part. The number of sandbags delivered to players is constructed so that they will not be able to cover the territory alone, they will have thus to coordinate on the actions to undertake. This quest has a time limit of eight minutes to add real time stress. When the time runs out the players are presented with the group's flood protection score calculated taking into account how much of the area the players were able to protect. In order to calculate this protection score the sandbag distance from the river and the proximity with other sandbags are taken into account. Figure 11 shows some screenshot of the game. Quest 2: Monitor the territory. Measuring the water level of a river is important to be able to predict if the river is about to overflow. In this quest, the players are asked to measure the depth of the water at different points along the river. If they encounter a critical rise in water level they must start an evacuation of the area. The main learning objective of this quest is to teach players that measuring the water level can be used for predicting floods. They will also be sensitized about the fact that an early flood warning is important to be able to evacuate in time. Finally, the players must move along the river to measure the water, which promotes exploration of the territory. From a practical point of view, the quest consists in five checkpoints along the riverbank where the players must measure and submit the water level. The quest has a time limit of 7 minutes. As for the first quest, to be able to measure the water level at all the check points before the time runs out the players must split up and cover different parts of the territory. Coordination can be done through face-to-face communication, messaging, or by observing where the other players are on the map. If the players encounter a critical water level in different checkpoints they are instructed to start an evacuation (triggering the evacuation quest for all the players).

Quest 3: Evacuation. When a flood warning is given, the appropriate action is to evacuate the area and go somewhere with a higher elevation. In this quest the players must go to an evacuation zone (displayed on the map) before the time runs out. If they are not able to reach the evacuation zone, they will die. The main learning goal is to teach that evacuation is the right course of action when a flood warning is issued. Players will also get sensitized to the geography of their local area as they have to move to a location with a higher elevation. This can make them aware of where it is smart to go if a real flood occurs.

During the Flood. This is a passive moment in the game as once the flood hits the river will overflow and the player will not have the possibility to stop it. The flooding is represented on the map as a transparent blue color that broadens the actual river (Fig. 2(b)). How much the river overflows is dependent from how well the players were able to perform in the previous quests. When the flood hits, if a player is into a flooded zone, he will die. This emotional aspect is used to sensitize about the dangers of flooding and the impact of flood protection.

Quest 4: Search and rescue. In this quest the players will look for missing people by exploring different zones. The missing people are non-player characters (NPCs) and they can be dead, injured or dehydrated. The NPC remains hidden until the players are within 25 meters of his location. When the players get close enough they can interact with the NPCs to discover their condition. Once done, they must notify the correct authorities about the situation (for examples describing which kind of help the NPC needs, or if the NPC is dead). Obviously, to have a better chance of finding all the missing people the players should coordinate about who goes where. Again, because the players must physically search the area to find people, they could learn more about their local area. It's worth to note that if a player walks into a flooded area he dies. I spent a long time describing the game dynamics of Flooded as the game elements play (willingly) with rational and emotional elements. At this point of the work on this set of prototypes I was fully convinced that the element that one of the problems with the MoDo game was the extremely bound and rational approach to territory exploration, even if sensory and physical elements were present. For this reason when designing Flooded, the emotional elements were strictly linked with the rational ones, and actuated through physical and sensory interactions.

DESIGN AND EVALUATION:

The evaluation was performed in 2 iterations, with 8 players in total. The participants were all in their mid-twenties. The evaluation involved a game session, a short questionnaire, and a focus group. More quantitative data were collected tracking players' movements and actions during the game. An in depth description of this evaluation can be found in VI of the author's publication list.

Hereafter some consideration on the affordances proposed by Flooded.

Cognitive and Functional affordances:

The game design contains several features that aim to achieve an acceptable level of realism. All the players understood the goal of the quests and how to complete them. The game led to learning in situation about flooding management aspects. One player -who died (in the game) because he tried to walk into the flood to interact with an NPC that was already dead -during the debriefing phase said: "I did not really think too much about it when I did it, but in retrospect it was obviously not a smart decision."

The evacuation quest was regarded with interest: "It can educate people on where it is smart to go when a flood arrives, to push them to go to an elevated area." One of the players experienced firsthand importance of knowing the territory. "While I was trying to evacuate, Google maps lured me into a dead end. On the map it showed a path you could take to get to the main road. But there was a 5 meter tall wall blocking it." When asked if he learned something from this he answered: "Now I know where not to go if I have to evacuate from this area in real life." Physical affordances: Cognitive understanding and learning were possible because of the subjacent physical affordances: visual clues given during the game (alerts, displaying of the zone on the map, iconic elements and so on). For this particular game we decided to not use sound, as being in outdoors running implied to be attentive to the environment (we preferred not to ask users to use earphones for their safety).

Emotional affordances: One of goals of Flooded was trying to put the players in stressful situations, both to mimic how stressful it's a real flood but also to make the game more engaging. In one of the sessions, none of the players was able to reach the evacuation zone in time. When asked if they understood what they should do if a real flood warning occurred, they answered with great emphasis: "Get out (of the area)!"

One player was able to reach the zone at the last second. He commented: "I was there. I ran and I saw that I had one second left exactly when I reached the zone, but it didn't update in time. I was going to do a MacGyver and throw myself into the zone."

Finally, another player commented: "The time aspect was motivating, both negatively and positively. It was stressful after a while; I didn't really grasp the gravity of the situation at the beginning. I just walked calmly to the crate. And when I was measuring water, suddenly the flood was coming and I had to get out of there and had just a few minutes to do so. This was both stressful and engaging."

During the first evaluation one of the players was stuck in the flooding during this quest. This fact seemed to have a huge impact on him: "If you make bad preparations you will most likely die if there's a flood. It's a good lesson." Another player commented: "I think the level of flooding was based on the flood protection score. If we had been able to protect the area better, we could potentially save even more people." (I want to remember here that "people" were Non Playing Character with a scripted behavior). The game also raises an awareness of personal physical conditions: "I think we would have made it if we were in better shape and spread out. The steep hills from the river and towards the evacuation zones made it difficult."

The feedbacks from the players suggest that the game was highly successful in creating an engaging experience because of the location-based aspect, the physical and sensory clues, and the emotional aspects.

SOME GENERAL CONSIDERATIONS

Table 1 summarizes the different affordances used in the above-mentioned researches. As we can see, in the above described works the accent was put on the sensory elements, and on the channels/interaction means conveying them. The emotional element is still present, as an element for hindering or facilitating interaction. The social gameplay, the sensory aspects, and the emotional aspects, where conceived trying to limit the possible "situational impairment".

As a result of this analysis we can raise some interesting research questions:

1) The role of emotional aspects as sensory enablers seems worth of investigation. Yet, the role of the interaction with the other players in the creation of the emotional affordances is an open question (e.g., will the music effect be the same, and have the same role in reaching the functional goal in a solo game?). 2) How many sensory stimuli the brain can afford before entering in overloading mode in such a rich environment (here bodily interaction is not limited to tapping but to the whole experience of running and acting)? Is the number of channels (e.g., the number of objects) used to conveying them influencing it? Is the number of occurrence of sensory stimuli per second influencing it? And so on.

SOME ANSWERS TO THE INITIAL RESEARCH QUESTIONS

The initial research questions described in the chapter can be regrouped in 3 categories. 1) A first group of questions is linked with the role of multi-modal interaction and its impact on learning. 1.1) In this first group of question, a significant role is played by tangible objects for learning (basic knowledge about a subject or soft skills). 2) A second group of questions is linked with relationship between body and space for learning, where space is intended as both, a physical place and a rich multisensory environment.

3) Finally, the role of social communication/aspects for learning was questioned.

The first group of questions is in some way linked to the role of embodied interaction in learning. The experiment conducted with Don't Panic shows well that a rich environment in which the user is able to choose which communication channel and interaction channel to prioritize, works better for learning than a less rich environment. In particular, manipulating tangible interfaces and acting on them helps with learning. In this first context (in a controlled lab setting) the players were constantly stimulated by auditory and visual clues, in a complex setting where functional and cognitive affordances where of high level. On the other hand, several functional (physical+sensory) affordances where available for the user in a modular, independent way (for example the printed cards, the barriers, and the pawns) so that the user could choose when to give his attention to (or ignore) them.

The importance of giving the user the power to choose whether to interact or not with something in this kind of environment is well shown in the context of the whole body in space interaction described in the chapter with the examples of the mobile games. In the first mobile game example, the game was "forcing" the user to interact with multiple channels simultaneously. As a result, we had a bad user experience and a quasi-null learning outcome. On the contrary, in the second mobile game (the one about flooding), as the user ran around the city, s/he would check the information on the mobile phone only when s/he felt the need for it (and could avoid to use auditory clues).

We can thus say that in the context of multimodal and multisensory interaction for learning, not only is it important to create modular adaptation, but also to play with interdependence and non-interdependence. The interdependence or non interdependence should be constructed so that the system allows for what we could call a "free will" interaction. In this case, it is not a matter of asking oneself if an auditory clue is more adapted than a visual one but to let the user be free to decide when a visual clue is more adapted than an auditory one.

It is worth noting, that none of the experiments described in this chapter address therapeutic applications, where the role of guided interaction is much more important than in games for learning. It would thus be worth investigating the role of this "free will" mode in the context of rehabilitation.

As for the third set of questions, we are saying nothing new if we say that how humans interact with each other face to face (smiling, frowning, pointing, tone of voice) lends richness to communication. In everyday conversation we can (or not) give attention to different clues when interacting with people. This was also true for the Don't Panic experiment described in this chapter (for example when discussing strategies, all actions where stopped and movements were only used as clues to support the discussion). On the other hand, in the case of the flooding mobile game, peripheral attention in social interaction (e.g., awareness of the presence of the others without face to face interaction) was central to the experience. How to support channeling from centrality of social interaction to peripheral social interaction is an interesting design question that needs to be investigated further. I believe that this question brings on a different path that the one on modular adaptation, as in this case we are talking more about a continuum than a fragmented adaptation An early answer stemming from the conducted experiments is that functional affordances play a big role in the movement in this continuum.

SOME GENERAL CONSIDERATIONS ON DESIGN AND EVALUATION TECHNIQUES AND METHODS

Interesting enough in the described projects there were no issues instantiating a Participatory Design approach. Returning on considerations done in last chapter, this result could be due to the Culture of Participation up Civic Protection and Emergency Workers who are trained to work together with different bodies/authorities. Let's think about humanitarian assistance, particularly during crisis. Actually, during a period of humanitarian crisis, hundreds or thousands of people around the world organize themselves to provide logistical, food or medical support for local people in need. Many organizations like the Red Cross or Médecins Sans Frontières are able to quickly provide emergency medical assistance across the world. Participation in this context is not only an "experts' concern": in time of emergency (for example earthquakes) professionals participate together with normal citizens to provide assistance to population. Laypeople across the world support these humanitarian organizations. The domain of Crisis management and emergency work is thus in the core structured around participation. An element that has surely influenced the successfulness of the participatory workshops we held.

CHAPTER 3: PERCEPTUAL SUBSTITUTION FOR VISUALLY IMPAIRED PEOPLE

Phenomenology, the philosophical movement that structures my researches, embraces direct experience in open and dynamic environments. The world as directly experienced is hardly determinable: it is of an ambiguous and transforming nature. In addition, to cope skillfully in the world from day to day, we do not need a mental representation of our goals: our body is simply solicited by the situation to find the right balance so as to gain a maximum grip on the situation [32]. As Lucy Suchman [START_REF] Suchman | Plans and Situated Actions: The Problem of Human-Machine Communication[END_REF] argues, people do not execute an internally created "plan for action" but instead act in the face of concrete circumstances in the world (situated action). Each creature, based on its sensory capacities and its behavioral repertoire, brings forth both itself as well as its environment. The essential element for making sense of the world is thus the sensorimotor coupling between a person and its environment. Thus, meaning is created in interaction.

Systems that acknowledge a phenomenological complexity of being in the world are most likely not to follow the if/then paradigms and much more likely to follow ones that embrace the complexity, holisticity, and continuity of context [START_REF] Stienstra | Embodying phenomenology in interaction design research[END_REF]. Phenomenology-inspired design research thus demands an appreciation for and focus on the unique subjective experience of people without lapsing into generalizations, user models, mathematical descriptions, guidelines, and other Cartesian-inspired elements that emerge from rationalism, objectivity, and reduction. On the other hand, technological design needs to have a reductionist approach. User-model inspired approaches to design tend to reduce people to grouped commonalities for the sake of reproducibility (of the approach, of the result, and so on). One possible solution to this tension is the usage of an adaptive approach (like the ones used in the systems presented in Chapter 1).

However, in my latest researches I'm trying to work on this tension more on the methodological approach to research than on the technological solution side. In addition, as the sense of the world is created in the sensory coupling between a person and its environment, it opens to an interesting research question for impairments (both in the sense of situational impairment and people impairment). What is the delta of constructed meaning in interaction in a particular context when using a "neutral" approach and when using an empowering approach to design?

The two projects I will present in this chapter were carried out at UTT-Université de Technologie de Troyes, and are still ongoing (even if they are going through their final phases). The sensory and emotional elements in these researches are preponderant over the cognitive and functional elements. As a matter of fact, a great amount of work was dedicated to the definition of sensory affordances in the context of perceptual substitution for visually impaired people. In both cases the final results are sensory augmented artifact, even if not necessarily computer-based. Beyond the created artifacts, the usage of different methods in workshops (and even the fact of letting go the "work"shop mindset) lead to the emergence of an initial reflection on the importance of the ability-based approach in the subjective and inter-subjective construction of meaning in the interaction with the environment. This is particularly visible in the exhibition example, where the workshops created a new vision of the world of the interaction with art for the participants. Another important aspect to not forget is that as designers and builders of interactive systems for human use, we also play a central role in defining people's relationship to and experience of the physical world.

SENSORY AND PHYSICAL INTERACTION FOR CONCEPTUAL MAP CREATION

The research I will present in this part of the chapter is a work in progress that does not insert itself in any funded project. All the persons involved in the project worked on voluntary basis because they were convinced of the usefulness and the importance of the final artifact, a 3D map to help visually impaired people to learn about the territory before its physical exploration. A previous regional project, SEEIT, was the bootstrap for the ideas that will be described hereafter. The SEEIT project aimed at developing access to leisure, culture and artistic creation for people with visual impairments, expanding the range of accessible activities for visually impaired persons (VIP) as exclusion and lack of accessibility are as much the result of diminished individual ability than the result of the material and social environment [01]. The experimentation site for the SEEIT project was Grandham, a commune in the Ardennes department in northern France. During the project, a walking trail was defined, which allowed the VIP to explore in pseudo autonomy the actual territory. More information on the SEEIT project can be found in [02] and [03].

At the end of the SEEIT project, we decided to continue our researches creating an augmented interactive map to help the visually impaired persons who will explore the Grandham territory to learn about its configuration before going out for walking. Thus the aim of the work I will be presenting was to develop a multi-sensory map that would be actually located in the Grandham town hall, thus creating a refined object to be used in a real context, and not a research prototype. vision can't be fully restored with medical treatment, surgery, or corrective lenses like glasses or contacts. The term blindness is thus used for complete or nearly complete vision loss. The most common causes of visual impairment globally are uncorrected refractive errors (43%), cataracts (33%), and glaucoma (2%) [07] Refractive errors include near sighted, far sighted, presbyopia, and astigmatism. Cataracts are the most common cause of blindness in old age. Other disorders that may cause visual problems include age related macular degeneration, diabetic retinopathy, corneal clouding, childhood blindness, and a number of infections. Visual impairment can also be caused by problems in the brain due to stroke, premature birth, or trauma among others. While the number of people with visual impairment from birth is diminishing, the absolute number of people with visual impairment is supposed to augment, due to the aging population.

The disappearance of a sense can disorient an individual and can be experienced as a trauma. A long process of adaptation must be thus realized by relying on the other senses. It is worth to note that the more in an advanced age the person losing her sight, the more difficult and long this adaptive process is. In this case we can speak of resilience. Carrière [08] defines it as "the study of the psychic forces of the individual that help him overcome the trauma of disability". It involves many factors such as personality, internal and external protective factors against the environment or risk factors. This resilience is built in interactions with others.

As all perception is based on mental representation, strategies of over-verbalization are often adopted by the visually impaired and their entourage to palliate the lack of visual perception. The other element often used as carrier of meaning in case of visual impairment is the sense of touch. While when we talk about the sense of touch we typically think about reaching out to understand the quality of something, touch is the sense by which contact with objects gives evidence as to certain of their qualities in a larger sense.

As a matter of fact, the sense of touch is a rich medium even in everyday life social exchanges [09,10] and people communicate through it, especially on an emotional level; for example, they form strong attachments or cooperative alliances, they soothe and calm, they negotiate status differences, and they also express romantic and sexual interest. Touch communicates the hedonic tone of emotion predominantly, generally warmth or distress, and touch intensifies the meaning of emotional displays in other modalities.

THE INTERACTION PARADIGMS

Devices referred to as "sensory substitution systems" (or "perceptual supplementation" as used by [11,12]), have been developed since the end of the 1960's for assisting blind people. These devices transform stimuli that are linked to one sensory mode (e.g., vision) into the stimuli of another mode (e.g., touch). Bach-y-Rita's [13] Tactile Vision Substitution Systems (TVSS) is the historical and paradigmatic device used to explain this interaction paradigm.

The TVSS captured a visual image by the mean of a video camera and converts it into a tactile 'image' comprised of a surface of stimulators (a 20 x 20 matrix) placed on the subject's back, chest or even tongue. This kind of system allows the user to perceive space through tactile stimulation.

More specific to visual impairment, several prototypes have been proposed for blind people. They rely on tactile or haptic [14]), audio [15] or a combination of these modalities [16].

For our purposes we are particularly interested in perceptual supplementation interaction paradigms linked to accessible interactive maps.

Following [17] accessible interactive maps can be divided in five categories [18]: 1) Virtual acoustic maps entirely based on verbal and non-verbal audio output 2) Virtual tactile maps making use of haptic (e.g., force-feedback) devices;

3) Braille tactile maps are based on the use of dedicated raised-pin displays; 4) Augmented paper-based tactile maps use a raised-line map as overlay over a touchdisplay combined with audio output; 5) Olfactory and gustatory modalities. While extremely rare this paradigm can be found in [17].

Several works are at the crossroad of these paradigms (see e.g., 19,20,21). This classification of sensory augmentation is particularly interesting for our purposes as the classification is based on sensory affordances more than cognitive ones. However, an exhaustive review of maps prototypes with a different classification approach can be found in [22].

To summarize, accessible interactive maps can be used by visually impaired people to acquire mental representation of space, surpassing the important limitations (limited amount of information, ability to read braille text, etc.) that non interactive maps have. It's however difficult to decide which kind of interaction paradigm is the better suited for perceptual supplementation in an absolute way.

Brock et al. [23] have demonstrated that the effectiveness in the usage of different types of maps was not related to the map type but dependent on users' characteristics as well as the category of assessed spatial knowledge. Starting from this finding, we decided to explore the most appropriated kind of perceptual supplementation for a particular subset of users, the members of the SAVS Michel Fandre who collaborated with us in the SEEIT project.

DESCRIPTION OF THE WORK

In this part I will focus on the different workshops we held and on the participatory approach used for this work, describing in which way these workshops led us to choose an interaction paradigm, to reflect on the sensory and physical affordance linked to it, and to create an actual map. With respect to the other projects descriptions I will spend thus more time on the actual workshops to show how the different affordances evolved through the different phases of our work.

Epistemological position

It is important to note that in our reflection we started from the assumption that it's impossible to find one unique ideal solution for visual impairment by testing different artifacts with the participants. We positioned thus ourselves outside the positivist paradigm, as our aim wasn't to create a general, ideal, solution.

Our approach involved co-building (not only in an intellectual way, but also in a physical way) the map with the participants. On one hand, we considered the workshops we held as semi-structured interviews to guide our choices, on the other we co-prototyped with the visual impaired persons. This approach guided the way we limited the range of possibilities for creating an interactive map for Grandham. It's also important to understand that the consequences of using a non-positivist paradigm are the impossibility to use classical usability evaluation, as we are more into an interpretative process carried out by researchers during the project. During the rest of this chapter I will explain not only our design choices, but also the process that led us to such choices, as I believe they are both very important.

Even if we tried up to the hilt to use an interpretative approach, before starting our workshops we fixed in a non-participatory way one only element: the map should have an objective perspective and not a subjective one. To be coherent with a participatory design approach we should have, in fact, explored the interest of a subjective map (i.e., a map using the personal point of view of a person, for example enlarging some buildings, because considered the most important by the participant). However, a workshop around subjective mapping would have demanded design and artistic competences we were not able to mobilize at the beginning of the project. All the other aspects of the map (size, scale, point of interest and so on) were defined during workshops.

As for our methodological protocol, following [24] in our workshops we tried to differentiated a maximum interaction and materiality.

In HCI, and more specifically in interaction design, the influence of material aspects on interactions is getting more attention, but is still in its infancy. Rosner [25] argues that the design of tangible user interfaces in HCI often just adds material properties onto the designed form, rather than implementing in an authentic way the materiality of the object.

During some of our workshops we thus explicitly separated work on materiality, from the work on interaction. All sessions were filmed to be analyzed later by the researchers.

First workshop: collecting memories

For our first workshop we interacted with three blind and visually impaired people who already participated the SEEIT project. The aim of this first workshop was to collect their memories about their first exploration of Grandham, to understand which elements would be interesting to have on the map. The three participants were matched with three mediators-researchers. We sat in pairs, and the mediators noted on post-it all the proposals coming from the participants in a brainstorming mode. To reinforce memory, the already listed content was read from time to time to the participants.

At the end of this first brainstorming session, all participants reunited and the written postit were put on the table. One mediator worked as facilitator for a regrouping phase in collaboration with the participants, reading aloud all the post-it. In case new ideas appeared in this grouping phase, we simply wrote them down and added them to the group discussion. As we could see, we did not feel the need to create an adapted method in order to conduct this workshop, as slight adaptations were enough. This phase led to the emergence of different kinds of point of interests that could be aggregated around three different paths. The results of this brainstorming sessions were submitted to the persons in charge of the management of the Grandham site, who sent us back a map of the zone with the most interesting paths able to touch these points of interest drawn on it. Figure 1 shows some photos of the first workshop. The main objective of this second workshop was to explore in which way the points of interests should become components of an interactive map that will allow a first discovery of the surroundings of Grandham.

Before starting, to help project the participants into possible interactions, we created bricks of actions and sensory stimulations through various materials to be used.

The second workshop was thus organized around four moments: a) A first phase in which participants should choose particulars points of interest from the ones identified during the workshops, identify what they evoke, then propose interaction scenarios (what this point of interest would be useful for, how to insert it in the paths, and so on) b) A phase of exploration of the materials that will be possible to use for the map and the point of interest, c) A phase of exploration on a mock map (roughly reproducing the real territory) and creation of physical possible point of interests. In this phase, the participants created the shapes and the materials that would be possible to use to indicate the different point of interests; d) A phase on exploring how to add interactivity to the map.

The idea behind phase a) was to is to provide a rich description of the point of interest that goes beyond simple information (describing for example sensations, anecdotes, behaviors, etc. that the point of interest could evoke).

For phase b) we displayed materials chosen on criteria of safety (a visual impaired person should not get hurt interacting with it) and sensory differentiation. We thus put on a table materials ranging from sand to wood, to plastic marbles, to foam. Figure 2 shows some photos of phase a) of the workshop. Interesting enough, and this is something which recurred also in the SmartArt project which I will be describing in next section, most of the time in this phase was spent by the participants trying to understand the kind of material they were confronted with. The materials were explored by tapping + pinching + friction-sliding of the fingers on the material / opening the material when possible (e.g., tissues). It is worth to note that while we did not explore in this workshop the role of sound, this did not prevent the participants to focus on auditory clues: one participant noted that the noise made by the polystyrene during exploration was more interesting than the sensory feedback of touching it.

At the end of this first phase, the participant were asked to choose 2 to 3 materials who could be representative of their points of interests, and to go back to the map, to shape them with the help of a mediator.

In this phase c) the participants tried to use the selected materials to create shapes representing the point of interest (trees, bombshells -the zone was a warzone during WWI -and the like). When needed, to be able to generate shapes easily we used modeling clay. The main idea was to create a simplified model illustrating the idea of the scenario with no need of functional or realistic interaction. Also, starting from materials and not from shapes was voluntary done in order to avoid to immediately going into the functional aspects. In this phase some participants evoked the usage of sound and odors to perceptually supplement the map: sound could evoke the fear of the war (people who scream and cry, explosions and so on) odors could reproduce sweat (for fear). Before the workshop we created quick and dirty prototypes for different kind of tangible interactions, ranging from buttons, to levers, to conductive ink. The main idea was to explore ways to activate audio description or other interactive feedbacks (for example vibration). Figure 4 shows a quick and dirty prototype we created to make the participants to experiment with conductive paint. One of the participants was very interested in the conductive ink (although it did not feel good under his finger) as while he was exploring the sheet of paper, a sound was triggered. The conductive ink (who was made thickest for interaction purposes) was not thick enough for that his exploration with the index was able to detect it. In fact he was unintentionally activating it with it is his little finger, and not his index, several minutes before noticing it. Once finished, the pairs came together and each group took 10 minutes to explain their approach to others.

Researchers work session: map design choice

After the second workshop, each researcher analyzed the videos of the exchanges in which he/she did not participate, and the notes taken during the workshops. During a grouped meeting, the three researchers pooled all the elements collected during the workshops, their impressions, and tried to converge towards the definition of a prototype.

As quick recall, the goal that we set for the map was to encourage exploration of the Grandham territory, to give some elements of understanding of the surroundings, and at the same time to give the "explorers" the will to go further in the understanding of the territory.

Hereafter what emerged from this confrontation phase after the workshops:

 First of all, the very idea of augmented map has been questioned. The question if augmenting the map was more counterproductive than useful, as it could quickly led to information overload, was largely discussed. As a result of this discussion we all agreed to go towards a "minimalist" augmented map, a compromise to avoid confusion and the famous overload. In this way we could allow time for exploration and interpretation of the map.  The workshop led us to renounce at whatever idea of realism. For example, using the model of a church made of bricks could create a false pre-representation if the real church is not made of the same material. The point of interest will thus represented through icons and symbols.  We had a long discussion on the materials to be chosen for the map and the points of interests: for each material there is a lot of sensations and possible interpretations (functions of textures, auditory, symbolic, etc.), and having many different materials can be considered as very "loaded". However, we also decided that we should be careful with this notion of "charge" which is very cognitivist: there is a tendency to consider that a material is loaded because it has many discriminating elements, except it could also be said that two very different materials are easier to differentiate, decreasing the load (the careful effort to differentiate them). On the other hand, as a matter of fact, small flaws of the map due to the way we applied paintings let to over thinking the meaning of this unwanted detail.  On the size of the map aspect, at the end of the workshop we asked the participants to give their opinion on their capability to explore different sizes. The one used in the workshop (1.20mt) was a little too big, too imposing, and therefore one of the reasons for the little exploration of the participants. A smallest one (80 cm) limited the number of information. We decided thus to opt for an in the middle dimension (1mt).  Although it is very interesting to see the discussions around the possible modes of interaction via the ink based touchboard, it seemed to us that whatever technological choice will greatly complicate the learning aspect (because the new way of interacting should be learned too).  An open question at the end of this debriefing session was the concept of the "form" of the information: if the way we provide information is definitive, static, it does not work as incentive for discover (and the very idea of interactive map has to be dropped). We could thus opt for a voluntarily incomplete form of information, guided via some anchoring, that could lead to some form of narration. Regarding the content of these narration we decided to ask/let the local actors to choose which information to insert, as they know the best the territory they live in. In this way the map will be the spokesperson of the inhabitants of Grandham.  As a consequence of the above mentioned reflections and remembering our choice of minimalism we decided to keep auditory stimuli to the minimum (contextualized sounds and explanations around the point of interests). The final map will thus be non-sonified (which adds to its durability and therefore its finality) and only the traces/anchors will be sonified (which resonates all the more with the idea that each trace comes from an experience, so narrated).

The final map will have thus a mono-material base with topography and path in relief (like in classical 3D maps), colored paths (to help visually impaired but not blind people to interact with it), and additional mobile elements will be used as bringers of traces and information. In addition it is important that the final map requires no maintenance, a little but not too much learning, and that it is modular (for example that we can, once a year, detach the elements to replace them) as it will be used in an actual context by nontechnological expert users.

INTERMEDIATE PROTOTYPES

Following this first phase in which we explored the sensory aspect and some cognitive elements to be added to the map, we decided to go for a first set of prototypes to test our choices. In this second phase we worked at two levels, the map as a whole, and the particular components of it.

For the map as a whole, basing on IGN maps 26 of the actual territory (Figure 5 left), we create an abstract and simplified 3D the digital version of the territory to be explored (Figure 5 right). The actual physical version will be carved with a CNC cutting machine (see Figure 6) that will be sculpting the reliefs on a multistrate MDF (Medium Density Fiberboard) plate.

The discussion on which material should be used has been carried with the help of the users of the Paris FabLab we interacted with for carving. The final map will polished using sandpaper, varnished to improve its resistance, and only paths will be painted in yellow, creating a deep contrast between the gray of the card and the important elements, useful for certain kinds of vision impairments. As we still had some open questions on the fixed symbolic elements that should be presents on the map (pasture, cultivated fields, forests), we created different samples with the same technique that will be used for the final map. These different samples carried different iconic elements in different sizes to be explored by the same participants. Figure 7 and 8 shows the 3D models and the actual printing of these samples. The second interaction element that is important for our map is the mobile item that will convey the contextual information. As said, we decided to go for a non-high-tech solution for this item (as it should be quickly easy to use for non-technological experts). We thus decided to use an already existing technology used to help visually impaired people to interact with their environment: the Penfriend. The Penfriend is a vocal tagger used to describe objects. Using this kind of object will made our map easily configurable by a non computer-expert user, will facilitate changes in the content, and made the items we will use to convey the content easily replaceable. While we could have obviously be able to develop a similar object, the commercial product has the stability requirements we need for the final users. As for the objects that will be the carriers of the actual content (that we will hereafter call totems, in the multimedia sense of the term), their form was designed by a designer (Luce Aknin) who joined us on the project. In Figure 11 we can see some of her sketches. After brainstorming on these shapes, the selected form is the one you can see in Figure 12. The main idea behind the totem form is to allow for a simple interaction with it, with a form that will let easily understand that the object is a point of interest in the map. The totems will be 3D printed out, and chip will be added on the top to let the user know which kind of content is available (i.e., if the content relate to the nature, to the history, other elements of the territory). The totem is empty, so that we could slide a magnet inside. Another magnet will be positioned in the opposite side of the map to allow for an easy positioning of the totem in whatever place considered pertinent by the persons in charge of the Grandham site. Again, the main idea was to create something easily manipulable by non-technological expert users. In Figure 14 we can see two participants to the workshop exploring the prototypes. The prototypes (3 different kinds) with points of interest sonified via a penfriend positioned on the map where used to discuss the finishing (as we can see from the photo the prototypes are polished and varnished, and paths are specified in yellow as it will be the final map), as well as the kind of icons to use (each prototype carries a different set of icons).

Workshop outcomes

 The handling of the maps and the finishing of the surface did not present particular problems or obstacles to exploration. Asked specifically about this aspect, the participants' experience seemed pleasant. Participants were able to focus on map content throughout the workshop without adhering to manufacturing flaws, which in fact validates our manufacturing process.  With our three participants, not only we could observe the diversity of ways to explore the map (quickly with two hands for the more experienced, with one hand and more slowly for the less experienced) but also that different imaginary were deployed (view from the top, view in first person, more or less schematic, and so on). Still we were able to define which icons were the best suited for the different aspects. The importance of the right size of the patterns clearly emerged as expected, as a too fine pattern appeared as a texture, and a too big pattern appeared as unstructured. A pattern with a max of 5mm spacing seemed the right choice.  As for the yellow painting on the surface, some larger painted zones disturbed one of the participants. It appeared to us that it was necessary to use the yellow color sparingly. We thus decided to paint in yellow only the paths, and to keep the point of interest as yellow totems.  As for the totems, they were very simple to grasp and interact with, reassuring us with the shape choice. Also the information aspect seems to be well understood. The Penfriend is clearly of a very simple and relevant use to convey complex voice messages.  The workshop discussion opened on the other hand to the problem of what should be the given reference point (so that the users will be able to have a central point from which to start the exploration). In the end we decide to discuss this point with the people actually living in Grandham and who have been participating in the project.

VALIDATION WORKSHOP WITH GRANDHAM CITIZENS

In this final validation workshop we focused more on the global elements of the map (e.g., which zones should be defined with pasture patterns and which with ones as farming fields, and so on). In addition we discussed extensively on the physical actual position of the map in the municipality. In particular the orientation of the map to help creating an absolute reference (i.e., from where the people will enter in the room, and from where they will go out exploring) was widely discussed. As a matter of fact, the map was accepted as a final artifact to be exhibited in the town.

An inauguration of the installation of the final map will be organized by the municipality in February. Consequently, we will held the last workshops in the actual final setting after the inauguration.

SOME CONSIDERATION ON THE PROTOTYPES AND THE INTERPRETIVE APPROACH

Even if the map is not finalized, I can start to relate here some consideration on the affordances. As a matter of fact, the importance of the sensory elements as meaning bringers was explicitly evident in this context in which one of the senses was diminished.

The link between sensory and cognitive affordances was particularly evident in the exploration of the first mock up version of the map, where every little defect in the crafting made the person exploring the map stop, in order to understand if the defect was bringer of a meaning or not. Only in the final version where no defect was perceived the iconic patterns become bringer of an interpretative meaning (and additional unintentional tiredness was avoided). The same goes for the auditory clues. As said, the noise created by a particular material had a particular value in this context.

More difficult is to assert something on the functional level (does the sensory and physical elements helps in creating a mental image/learning about the actual territory?). This aspect should be assessed with the final extensive observation.

Finally, some considerations on the interpretive approach we used to design the different elements of the map can be drafted. In this work what we tried to achieve, was not to test a hypothesis, but to use a thinkering approach to create a possible solution. The sensory interactions (for example the ones with the materials) weren't used to decide which material was "better" than another, but as "topic of discussion" to understand the personal and shared meaning created by the objects, in a narrative perspective (the example of odours/sweat to signify fear is in my opinion a good example of this perspective). Thus the manipulation for creating sense is of particular interest in this case. In addition our role as researchers in this project was not "finding the better solution" but "making sense of what emerged". In our research positioning in this project, each workshop during which we interact with the participants is a moment when the researchers are "malleable": rather than having predefined hypotheses to check, we set up situations allowing us to learn from our interlocutors, and evolve in our interpretation of the problem. As a statement from the beginning we inserted ourselves as translators/interpreters and not as neutral researchers.

For example, the choice to renounce at whatever high-tech solution in favor of a craftsmanship approach, is our interpretative choice to make possible an easy creation of sense, not something scientifically proved by the interaction with the users. Thus questions and statements stem from the epistemological positioning, and have consequences on the protocols/methods used.

The question on how and when to promote this interpretive process (not necessarily through a narrative approach) as researchers is something I would like to delve into in the future.

part, we investigated emotions linked with being "together". Finally, we examined emotions linked the act of creating something. The aim of this first workshop was to see under which situations emotions were generated. The workshop did not limit thus its investigation area to art or to a specific type of emotion: to decide which are and emotion to describe was left free for everyone. From this focus group, different commonalities emerged between the participants, in particular linked to the sources of emotion. In the first place, the relationship with nature becomes evident: the noises, the odors and the substances connected with it, the seasonal changes. Secondly, the need for human contact emerged as an enabler to share pleasant moments. These two points allow the appeasement and the search for well-being. Finally, a real need for autonomy has been described during creation and exploration, with the necessity of movements for a liberating purpose. It is a necessity and a great pride for visually impaired people to show that they are capable of realizing concrete things.

Workshop with and on materials

This second workshop took the form of the discovery of specific materials. A total of ten visually impaired participants attended the workshop. Its aim was to find out what kind of tissue would be the most adequate and the most relevant to materialize different paintings.

To allow for a primitive form of touch-based interaction, a paper representation of two paintings (an interior scene and an outdoor scene) was realized using different layers and different types of papers. This rudimental representation allowed the participants to realize the location of the main characters of the artworks. Interesting enough, in this workshop participants' attention was more focused on the authenticity of the fabrics than on its symbolic representativeness. Participants spent time trying to find a fabric that could be related to the artwork period more than on the represented characters. The question arises thus as to whether this search for identification (and thus the focus on the particular) may or may not lead to difficulties in the recognition of whole image during the final exhibit. Apart from these classical structured workshops we decided to held some non laboratory based experience to better understand how the people we will be designing for act in a normal setting.

Guided tour

We invited the members of the association participating in this research to a guided tour in the author's town: the old medieval town followed by a tools' museum. The research objective was to gain insight about how visually impaired people walk and move around as a group, what kind of cooperation is there with sighted people, how the verbal explanations of the guide are perceived, what aspects are considered interesting (architectural description, historical context etc.). Previous ethnographic studies in museums, based on the observation of visitors' actual practices, revealed the significance of co-participation and collaboration in the museum experience. Indeed, museum visiting is a collaborative process: couples and family groups interact around exhibits; they discuss artifacts with museum guides and other visitors [26,27]. The ways in which people navigate in galleries, discover exhibits, and draw conclusions, arise in and through social interaction. Interestingly, we observed the same phenomenon in the guided tour. Moreover, we could observe how the binomials sighted-blind (there were 3, while the other visually impaired persons walked autonomously with or without white cane) worked: the sighted person described pragmatically what was she was seeing, while the guide's explanations were more on the historical or artistic context. This complementarity, they explained, was very appreciated. The other important insight derived from this guided tour, was that blind people could not create a mental image of the tool inside the museum by the verbal description only. Touching the object would have been necessary (see I in the authors' publication list at the end of this chapter).

Interviews with participants about their needs during a museum visit have shown the essential aspect of a hearing aid, and the secondary importance of tactile help in understanding a work. The tactile interaction, in fact, would serve only as a confirmation of the mental image that the participants would have made through the audio description. For some of the participants, in particular the ones who lost their sight later in life, only the auditory description would be necessary. The feedbacks from the guided tour of the city and the museum, on the other side, help us to guarantee the usefulness of the tactile help. Indeed, regarding the museum, participants regretted the small number of tools they were able to touch, especially the blind from birth participants. However, it was obvious that a tactile reading cannot be performed without a hearing aid helping the visually impaired visitor to understand what they are touching and to know where to move their hands to find the most important elements of the work. The hearing and tactile aids are thus complementary.

RE-CONTEXTUALIZING DISCOVERIES FOR THE ART EXHIBITION

In order to re-contextualize the findings from the workshops, we held a formal workshop and a guided tour of the Museum des Ardennes (Figure 15), the museum in which the exhibition will took place. The experience in the exploration of the museum has shown findings similar to the exploration of the city: the role of narration of the guide was important; the participants aggregated themselves in pairs or groups, and so on.

The quality of the performance of the guide was greeted by his involvement and the transmission of his passion in his explanations. This allowed participants to obtain details they would not have paid attention to during a free visit.

Figure 15. The participants following a guided tour in the museum Following the group visit, a third workshop was organized. The aim of the workshop was to collect the participants' feedback about the visit, to choose the works to be augmented as part of the exhibition, as well as to reflect collectively, in a relevant way, on the works discovered the previous day.

Thus a list of six works was selected for the final exhibition.

To summarize

The above-described workshops and tours allowed us to create a set of recommendations that were used not only to implement sensory augmented versions of the paintings, but also to design the whole exhibition experience. Discussions with the participants to the workshops helped to define very practical elements, such as the augmented paintings inclinations for tactile exploration (information which is normally not available in already existing guides for creating exhibitions for VIP, as for the particularity of our artifacts). The workshops gave us also the information about which medium to use to convey which information.

But, above all, the workshops allowed the members of the consortium to conceive a global experience for the exhibition.

CREATING AN EXHIBITION FROM THE WORKSHOP RESULTS

As a short recall, the aim of the SmartArt project is not to conceive an exhibition for visually impaired people, but to design an exhibition with them for everybody, normal sighted or visually impaired, so that they could all have a common experience. The participants to the workshop knew very well that whatever touch-based experience would have never reproduced exactly the original painting. To use the terms of one of the participants to the workshops: "is the same as trying to reproduce a masterpiece melody through images: you will never have the same experience".

The open question for us here was thus how to look at a painting without the eyes, knowing that whatever you will do is a (good or bad) translation of a piece, and thus an interpretation.

Thus, the project decided to use a global multisensory approach for the exhibition: the original masterpieces will be sided with written comments, audio comments, and a 3D reproduction of the painting (the augmentation will not thus be delivered by a single particular artifact).

The aim is not to make all visitors going through every sensory channel, but to let them to choose the better-suited (for them) combination.

Each visitor will thus choose is favourite way to explore the masterpiece, knowing that the aim is also for different visitors (visually impaired or not) to be able to exchange on their experiences and different points of view. The set of media used becomes thus not only useful as a discovery moment, but also as a sharing moment.

That's the reason why the exhibition was named: L'essentiel est invisible pour les yeux 28 .

The idea behind this global multi-sensory approach was one of the results workshops observations. The sense of touch, when used in an exploratory setting (like the one of a painting reproduction, or a map exploration) appeals to the cognitive, rational part (the user tries first of all to make sense of what is touching, not to feel it, which prevents an emotional response to the exploration). The modular (as the user can choose whatever combination of sensory channels he likes) approach to sensory exploration of a masterpiece, in some way has the potential to bypass the cognitive overloading problem.

Another aspect that emerged from the workshop is the need for autonomy of the visually impaired people (I want to explore with others, but I don't want the others to explore for me). The idea of modular approach was thus applied also to the global exhibition, as there is no pre-constructed path to follow: as the exhibition will held only 6 paintings, the visitors can move around as they want, thus constructing their own path through the exhibition. For the same reason, there is not a global audio guide, but mp3 readers that will be linked to each masterpiece reproduction.

THE ACTUAL PROTOTYPES

After explaining the global principles that guided the design of the exhibition, I want to spend some time explaining how the 3D elements of the exhibition were created.

This work has been done by the IFTS 29 and not by us (UTT) so I will not enter into details on the adopted printing process. However, it's very important to have a visual representation of the elements, and in particular of the affordances they generate.

The fabrication process used by IFTS mixed additive manufacturing (3D printing) coupled with machine based carving. The main idea was to avoid to give the visitors a high relief of the painting (as done in other exhibition targeting visual impairment), but to push to the extreme the interpretation of the paintings, and accepting that whatever reproduction is a "translation", thus an interpretation. The idea was reproducing the way an art expert is interpreting a painting: having a background layer, a central layer, and a front layer. These layers can potentially be read in an independent way, but most importantly they came together to form a unique spatial combination. A 3D (in the space sense, not the printing technique) exploration of the so-reproduced painting could thus be useful in order to create a mental cognitive representation of the painting.

As said, what we are creating it's a translation, as the created 3D models are interpreting the piece, and not reproducing it as in photography.

Finally, we decided to drop an initial idea of complementing the 3D printed reproduction (tissues and other materials) because of two reasons: possible information overloading, end hygiene issues.

Hereafter (Figure 16) we can see one (particularly well done, in my opinion) example of this approach. "La tailleuse de Chanvre" (the hemp cutter) is a piece of a local painter, Eugène Damas 30 . If we want to give a perspective analysis of the painting, we can see that we have clearly 3 layers. One first background layer is the part with the wooden walls. The second one is the one in which the cutter lady is sitting. And finally to the front we have the woman feet with the small pieces of hemp and some type of jug. To contextualize the painting, the painter wanted to show the difficulties of this work, which can be seen in the tired expression of the woman.

One particularly interesting element from the pictorial point of view is the way the hemp is rendered (see Figure 17). As we can see is a matter of highlighted brush strokes to give thickness to the hemp stack. The reproduction of this painting (see Figure 18) has been realized by carving the background and 3D printing the woman and the front details. Hereafter (Figure 20) the original painting and its translation in the museum context. Still, as we said, the 3D exploration as a cognitive one, is not able per sè to create an emotional response. This is the role of the narration (in the form of audio or textual narration) that will help to "give a soul" to the masterpiece.

The audio comment was thus constructed in the following way:

1) A first musical moment will give some time to explore (visually or in a tactile way) the painting. This first moment with only emotional and no cognitive content is important in particular for the visually impaired people, as they have to create a mental image of the paintings using only their hands. This kind of appropriation implies a first global tactile exploration of the object to interpret, and a second moment when the VIP will return on the object to explore the details. For this reason the mental effort in this moment is very high [31]. The choice of which music pieces to use has been done with one teacher of the music conservatory, who is also used to teach braille for music. To summarize, this audio part has been designed so that each visitor, at his own rhythm, will have the time to enter in the painting atmosphere.

2) The second part of the audio guide is a more rational description of the painting, ranging from what the painting is representing to some contextualizing information on the painter's time.

TESTING THE TRANSLATIONS

Before holding the final exhibition, we went through a final workshop. This final workshop was a mock exhibition, i.e., an exhibition that is in everything similar to the final one, but was opened only for the participants to the project. We used this mock exhibition to discuss in context with the participants of the feelings linked to the different sensory channels, as well as the global approach.

In this setting we wanted to observe if our research hypothesis: a) A multi-sensory environment is useful for a better understanding of a painting by visually impaired people, b) This kind of approach is generator of social exchanges based on emotional and not rational aspects. could be pre-validated.

For the a) research question, the exploration phase of the 3D printed paintings got very different results if the person exploring it was blind from birth, or become blind in older age, as the mental charge for the blind from birth person was higher. This result is compliant with what is already known in research [31]. Thus having at the same time music and touch was very appreciated by visually impaired but not by the blind persons (and I would like to remember that a similar finding in the first project has shown how music -which appeals to complex elements bridging auditory, rhythmic and cognitive aspects -bring easily to cognitive overload in patients deeply affected by stroke). Figure 21 shows the exploration of one of the paintings in which this overloading happened, a Sisley reproduction. In this case the blind person exploring the painting asked for additional (other than the music and the prepared commentary) information, as even with the explanation it was not possible for him to create a mental image of the painting.

This reinforces the need to leave everybody free to use whatever sensory channel they want in whatever moment they want. It leaves us also with an open research question as the music is used also as an element to create emotions. It would thus be interesting to explore in follow up interviews if the emotional impact for blind from birth visitors is the same as the one of visually impaired or blind in old age persons, as they are "seeing anew".

On question b)(is this kind of multisensory translation bringer of emotional and social affordances?) we can make two observations. First of all, when let free to explore the exhibition the participants normally created pairs (visually impaired/blind; non visually impaired/visually impaired; visually impaired/guide) or groups. We could thus observe the same behavior we observed during the different guided tours. This strategy was mostly used for a rational/cognitive understanding of the painting. Secondly, and more interesting in my opinion, once the formal mock exhibition was finished (and thus the "serious" part was finished), the visually impaired people started to discuss about the expression on the face of a Rimbaud portrait made by Picasso, joking about where he was looking ("I'm telling you he was looking over my shoulder", "No, I'm telling you he had strabismus!") generating thus the kind of conversation normal visitors would have over a painting (which evidently was found fun, more than engaging, but this is not strange for a Picasso…)

So, how to you let people "see" a masterpiece? Using the words of one of the researcher of this project: "On donne à voir surtout à travers les émotions, à travers des moments de partage, à travers des commentaires qui font vivre le tableau. On donne à voir bien sûr avec des supports tactiles comme ceux de notre projet, mais surtout en partageant un moment d'échange avec le groupe, pour mieux connaître l'autre". 31

SOME GENERAL CONSIDERATIONS

Table 1 summarizes the different affordances used in the above-mentioned researches. As we can see, in the above-described works the accent was put on the sensory elements as a way to create sense of the world. What is interesting to underline in particular for the latest work is how the thinkering approach (that I would like to remember is a bodily/sensory way of making sense of abstract concepts) was successful in creating another sense of the world (the works of art) for the people participating in the workshops. As emerged in the interviews, most of them where far from the art world which was seeing as something abstract and which did not make sense. During the thinkering approach where they were allowed to reinterpret/modify the piece of art, the same piece started to make sense. It will be interesting to see what will happen with the augmented map, which is an iconic representation of what was already known by the participants.

As a matter of pieces added to my research questions, these two last projects allowed me to 1) deeply experiment with the sensory aspects, 2) start to explore the idea of distributed clues/interactions with sensory aspects. This latter aspect is particularly visible in the SmartArt exhibit where the main artifact is the global exhibit which is composed of different painting reproductions which are composed of different artifacts (the 3D printing, the 2D scaled reproduction, the audio description, …) which could be seen as a whole but also as combined elements.

SOME ANSWERS TO THE INITIAL RESEARCH QUESTIONS

The initial research questions described in the chapter can be regrouped in 3 categories.

1) A first group of questions are linked to the way the content of a multi-sensory artifact should be approached, to avoid information overload. 2) A second group of questions are linked to the way a multi-sensory artifact should offer interaction to avoid cognitive overload.

3) A third group of questions is linked to the adaptation of participatory methods with particular target groups.

In the SmartArt project, two principles described in previous chapter as initial results were tested: (1) modular adaptability through affordances scaffolding and (2) choosing which communication and interaction channel to prioritize in a rich whole-body space interaction. These experiences were the key to generalize questions 1 and 2 for a general setting. However, we can also say that hidden behind these principles there is an additional element. This element is appropriation, which I intend as the act of appropriating something, in a personal and social way. This is particularly evident in the 3D map project, where the modular element is less visible, but the adaptive aspect is characterized by an overspecification (i.e., an important care to details), which is reached through the tinkering workshops. The appropriation of the 3D object was thus generated by the manipulations/modifications created in time with the participatory design workshops. In order to avoid cognitive and interaction overloading, it is apparent that we can opt for two extremes, the modularity/free will approach and the non modular over specification.

Although the final tests on the 3D map are not finished yet, our first results show that the prioritization principle is more important than modularity when working with people affected by a sensory handicap.

As for the final group of questions, an initial answer will be given in the next subsection.

SOME GENERAL CONSIDERATIONS ON DESIGN AND EVALUATION TECHNIQUES AND METHODS

Enabling the inclusion of impaired people in participatory design projects is a research question that challenges the democratic stance of this field of research as well as the way of doing research and design. This involves not only to adapt the used techniques but also to highlight the work the researchers have conducted in order to adapt their own settings (i.e., research lab, University) and their ways of acting.

It's important also to note that the place of participation in research is of special interest: making possible the participation of VIP in places where they wouldn't have come if not for the research project opens to a rich and dense experience, experience that is socially important. In addition, participants of the project felt strongly their role in helping designing an experience 'for the others'. I'm thus convinced of the relevance of action research projects as bringers of social change, if not of technological innovation.

Finally, if we look at results from workshopS and interviews they are sometimes coherent, sometimes not, with what emerges from researchers observations. This delta is linked to the different ways the participants narrate themselves, as well as the way the observer is interpreting the data (the subjectivist result of data analysis is not per sé a new finding for human science).

Thus, possible methodological research questions could be:

 How should we use the delta between these different findings?

 Should we integrate a reflection-based approach, thus accepting again that we have an interpretative approach?

 Is this approach coherent with the methods used to gain knowledge used by the involved persons?

CHAPTER 4: CONTRIBUTIONS AND STILL OPEN RESEARCH QUESTIONS To summarize, one first contribution of this work is a first analysis of the role of sensory elements in technology-based learning. All the described projects relate on the "translation" of different sensory needs into a system with learning purposes. The answers to these sensory needs have been instantiated through different artifacts, and different interaction paradigms. What emerges is a potentially global, systemic structure (affordances, interaction paradigms, methods) to be taken into account when proposing multisensory interactions in learning systems.

Another element that I hope is a contribution of this work is a first draft of reflection on the reflective stance and on the epistemological positioning during the described projects. In particular I underlined that when doing Participatory Design, the researcher is in a malleable position, and he should be prepared for it. I believe this is particular important when training young researchers.

Hereafter, I will detail the questions that were raised by each group of researches, underlining what is still open for an answer. I will return on the methodological reflection later on.

In the first phase of my researches (the ones described in chapter 1) the research questions that bootstrapped the different experiments can be summarized as:

-Which role plays body based interaction augmented by technology in learning? -The personalization of user experience acts as motivator? -The usage of emotional aspects acts as motivators? -Different devices can be used in a seamless way within the same learning action? -Sensory stimuli play a different role if they are working as supplement for interaction or as main interaction clue?

The work described in the first phase however was not able to give an answer to the role of multiple and distributed affordances, being the projects too specifics (like Rhythm workers) or too generic (like Hammer & Planks). The following subsequent experimentations added some more specific answers in particular when using multi-sensory interaction in a combined way (like in Don't Panic) or in a distributed way (SmartArt). In this part, sensory affordances (i.e., a design feature that helps, aids, supports, facilitates, or enables the user in sensing), physical affordances and emotional affordances become more preponderant in the analysis than functional and cognitive affordances.

We can thus list some specific answers to the initials questions:

-Body based interaction plays a role in learning when linked with tangibility and objects appropriation (e.g., thinkering) when sensory affordances are adequately designed (e.g., multiple sensory stimuli in the Don't Panic board vs it's mobile game)

-Body based interaction (e.g., movement) in an open environment (interior or exterior) plays a role for learning because of the naturally sensory richness of the interactions (e.g., Flooded).

-Personalization for learning in this phase becomes more linked to letting the users free to use different sensory channels when needed/wanted (SmartArt).

-Social exchanges works as emotional motivators.

From these findings we can move up one level to define some general principles that should guide the creation of multimodal and multisensory systems:

1) When designing multimodal and multisensory systems we should see them as empowering devices for an action that is held by humans, and not a result per sé. 2) When possible, opt for modular adaptability through affordances scaffolding (on the cognitive, physical, sensory, and so on levels). 3) A rich environment in which the user is able to choose which communication channel and interaction channel to prioritize, works better for learning than a less rich environment. Still the key is in allowing for this free choice. 4) In order to achieve this free choice, the system should be designed for interdependence between aspects (to create a rich environment) but also to stop this interdependence.

Still at stake will be the richness of the experience. 5) Thinkering can play a role in appropriation when is not possible to allow for modular adaptability 6) Strong social interactions and peripheral social interactions should be designed as a continuum to take into account the added richness of social clues for the system.

Still, more experiments and other specific questions needs to be addressed to transform these first assessments in research findings. Hereafter some possible research paths.

ON AFFORDANCES

What is the role of distributed and differentiated affordances for learning?

WHAT HAS BEEN DISCOVERED UNTIL NOW 106

In Chapter 1 of this work, we introduced a first set of projects involving physical training. This first set of projects led to the emergence of some first consideration on cognitive overloading when using different sensory channels (audio and external stimuli by way of the skin). It also led to some considerations in the case of social interaction during a therapy sessions, raising firsts questions on the role of "distributed and differentiated" affordances (e.g., giving different sensory affordances to different users). As for the asymmetric relationship between the patient and the therapist, the same objects will be bringers of different affordances because of the different ways of being in the world. While the SmartArt project started a discussion on letting personalizing the usage of multiple sensory channels, there is no systematic investigation on what is its impact on learning.

OPEN QUESTION(S) Does this means that the more the participants handle and act (thinker) on artifacts the better the learning? Should we distribute through multiple participants the sensory affordances for learning purposes? Should be different artifacts the carriers of these affordances?

How many stimuli can the brain manage before going into overload?

WHAT HAS BEEN DISCOVERED UNTIL NOW

If we look at the four described projects in a general way we can say that coupling different sensory stimuli (for example tactile + visual clues) facilitate learning. However we can also see that under a certain threshold the stimuli are not complex and interesting enough to create long lasting learning, and works more on a behaviorist reinforcement model (is the case of the rehabilitation system). While for the particular case of a motor rehabilitation case the instinctive application of a motor reflex is enough, in other cases (like the one on teaching reactions to floods) a simple conditioned response could not be appropriate.

At the other end of the spectrum too many stimuli create an information overloading that blocks learning. From one hand when too many channels are used, attention is misplaced from the content to channels and the interaction with the channel. It was the case of the printing machine in the Don't Panic board game that, if from one hand created expectations on the arriving information, on the other moved the attention from the actual content of the cards to the act of printing.

In a similar fashion the presence of too many stimuli of the same kind creates information overloading. In the case of the MObile Don't Panic application, the visual clues given by the mobile app (which were mostly based on colour encoding and iconic displaying of the presence of other actors on the building) were updated every few minutes while the user was moving around the building. The users had to make sense of all the displayed information, while at the same time decide for next course of actions. While this is normal for a real expert on the field, it's not appropriate for teaching the right actions to a novice, and creates only a frustration sense. At the other hand of the spectrum we have Rhythm Workers, which played with mono-sensory stimuli. Still, the lack of a comparative analysis of work held in different domains does not help in creating a global answer to the problem of multi-sensory stimuli.

OPEN QUESTION(S)

From these considerations we can extract questions such as: how many sensory stimuli the brain can afford before entering in overloading mode? Is the number of channels (e.g., the number of objects) used to conveying them influencing it? Is the number of occurrences of sensory stimuli per second influencing it?

What is the role of multimodal interaction and its impact on cognitive and functional affordances?

WHAT HAS BEEN DISCOVERED UNTIL NOW During the different works descriptions (for example Don't Panic on tangible interaction, and the MIG experimentation) I initiated a discussion on multimodal interaction as carrier of multisensory affordances. In the case of multiuser interaction in addition (for example in an asymmetric interaction between a healthy and an impaired person), each person can create different functional and cognitive affordance using different physical devices/objects. Here again, being the works evaluated through different research protocols, is difficult to give a unified answer to the problem.

OPEN QUESTION(S)

We thus have a first open question that is: how can we link multi-sensory and multimodal interactions, in particular in a multiuser environment?

What is the role of emotional affordances for learning?

WHAT HAS BEEN DISCOVERED UNTIL NOW

In Chapter 3, we discussed the role of sensory and related emotional affordances to make sense of the world (or even, to create a new sense of the world). The emotional affordances seem to play an important role (again, in particular in a multiuser, global environment).

OPEN QUESTION(S)

One interesting question to explore would be the role of emotional affordances for learning (e.g., will the music effect be the same, and have the same role in reaching the functional goal in a solo game or in a solo?). What is the role of emotional affordances for social learning (confrontations of sense experiences creates an enriched learning?) The experiment Rhythm worker raised the question of using a single sensory affordance that is inserted into an emotional response (the fun and challenge factors). We could thus ask ourselves if what happened is valid for other sensory stimuli or audition is a privilege sense.

MORE HIGH LEVEL OPEN QUESTIONS

From all these questions we can extract some higher-level questions:

1) What is the role of affordances in evolving systems (as games, or learning systems) through time? Until when the affordance complexity is manageable? 2) If thought is shaped by the words we know, artifact shape the way we think and interact?

ON METHODS

From the described projects we can also extract a set of reflections on the used methods.

If we summarize the reflections on the projects we can see that using a wide range of methods and a participatory design approach facilitate communication with the stakeholders and evaluation of the create prototype. On the design phase, observations and low fidelity prototype usage seems to be more effective than scenarios and storyboarding usage for discussion. This is most probably linked with the "everything is possible in the best of the possible worlds effect". When designing with non-technical experts who are not aware of technological limits and possibilities of a technology, whatever scenario that makes sense and makes the stakeholders think of a possible improvement of their actual situation is accepted. In addition, the discussed ideas are often limited to improvements of the ones presented in scenarios or storyboarding. This is not stupefying if we think that most of these stakeholders are not trained in creative design. While manipulating prototypes (even if we are talking about low fidelity paper prototypes) non-technological experts are more at ease with hijacking the object, as they can easily use it in alternative, non-planned ways.

The second interesting aspect on the method side is the well-known and used (in computer science, but not in learning) usage of quick iterative development, and the usage of different kind of prototypes (from very low tech to high tech) as means of validation. When the access to the stakeholders allows it, the usage of quick prototyping does not only allow for improvements but also helps in creating an empathic link with the stakeholders. It was the case of the paper prototype in the Don't Panic board game.

Synthesizing we can say that classical methods for multi-sensory design which are valid in Human Computer Interaction are still valid when mixing Human Computer Interaction with learning, and that combining different methods allows for better results.

As a final reflection I want to be back on the question: What is the role of reflective practices in our research work?

-We can explore the coordination between affordances. For example we could explore if in a particular activity the coordination of sound and vibration could work in an orchestrated way to create engagement.

-On the same topic we can explore when this coordination become overloaded.

-As the objects will be used in an activity, the will be also the devices the users will be interacting with, which could help in analyzing how to translate a sense into a mode.

Long-term research

At the center of my work is a Human Centered Design approach to technical and technological solutions. Putting the human at the center of the interaction does not mean on the other hand forgetting of the socio-cultural and technical elements in designing systems.

In this complex scenario, on the long terms I'm interested in finding a set principles and operative elements to guide development projects in their design, implementation and results measurement that takes into account the evolving role of the researcher.

However, as said, even when working in an empirico-inductive approach inducing and generalizing from collected data, I try to avoid the trap of thinking that the model created to describe the particular context and project is a theoretical thesis describing the reality of the world.

Thus, the result of this systemic approach will not be a set of ready-made solutions to HCI development challenges but a set of principle that encourages us to consider solutions that work best in a given context, by asking critical questions as mentioned above.

As this is a huge challenge to be achieved, a collaborative research approach has to be implemented with researchers coming from HCI but also different domains (psychology, art and design, philosophy, engineering, and so on). In order to be successful this collaborative approach has to be applied in a systematic way.

Figure 1 .

 1 Figure 1. Timeline of research questions.

Figure 1 .

 1 Figure 1. The overall system.

  Figure 2 left). The IR camera (c and Figure 2 right) and pico projector (d) are placed on the top of a support (b). A laptop computer is used to run the game client and to establish a Bluetooth connection with the WiiMote.

Figure 2 .

 2 Figure 2. Other details of the system.

Figure 3 .

 3 Figure 3. Alternative devices.

Figure 4 .

 4 Figure 4. Displaying the game in a vertical way on a larger scale

Figure 5 .

 5 Figure 5. A screenshot from Hammer and Planks, showing the normalized game play.

Figure 6 .

 6 Figure 6. A screenshot from the therapist interface with personalization elements.

Figure 7 .

 7 Figure 7. Different players playing together at the MIG event Within this particular experience, two elements are worth to retain (and both open in my opinion to interesting additional research questions). The first one is the non-conventional (for computer science controlled research) usage of the MIG event as first evaluation (but not validation) field. While non-conventional, this kind of experimentation led to an informal evaluation of the system, which opened in a second time for the clinical validation of the tool (see [III], [XI] in the author's publications list) as the system was considered by clinicians stable enough to go through it.

  IDENTITY CARD: RHYTHM WORKERS (2016-2017) Description and aim: Design and implementation of an auditory game to evaluate the pertinence of rhythm based training for different disorders. Societal aspect/problem: Cognitive and motor training Partners : UTT (Université de Technologie de Troyes, France) + Naturalpad (France) + EuroMov, Montpellier University, France Stakeholders: therapists, patients, Serious Games company Initial Research questions:

Figure 8 .

 8 Figure 8. Actual screenshots from the game: the evolving house Perception version.

Figure 9 .

 9 Figure 9. The Game for Health ecosystem

Figure 1 .

 1 Figure 1. The different interactions paradigms in augmented board games.

Figure 2 :

 2 Figure 2: Double interaction loop in interactive board games: virtual/physical architecture

Figure 4 .

 4 Figure 4. Details on Don't Panic prototype Don't Panic board (Figure4-left) -is a cardboard that visualizes a map portraying a territory divided in nodes, sector and paths. From a learning point of view, having a normal printed out board allows for an easy personalization to link the game to an actual territory. Nodes are edges between sectors and are connected by paths, as in closed cyclic graphs. Nodes feature physical constraints and no degree of freedoms for the hosted tokens; sector and paths provide visual constraints allowing tokens' translation and rotation, within the perimeter.

Figure 5 .

 5 Figure 5. Details on Don't Panic pawns Pawns (Figure5-right) -are active tokens that embody the players' presence on a node, each player interact with a personal pawn during a game. Pawns can be dragged from node to node, as long as a path directly connects the two. Pawns, and intelligent tokens (described later) have been implemented using the 1 st generation of the Sifteo cubes23 . Each cube is capable of sensing accelerations, proximity with other cubes on any of its four sides, and to display graphics on the top surface. In order to make the cubes recognize discrete locations on a board (required to use the cubes as pawns reacting to nodes constraints) we exploited in an unconventional way the data from the 3-axis accelerometer embedded in each cube. We designed sockets for the cubes each of them featuring a combination of unique horizontal tilt angles over two axes; the aggregated value of tilts angles is used as a fingerprint for the socket. We 3D-printed and embedded the sockets into the board as nodes constraints (Figure4-left) and we coded the relation between sockets fingerprints and nodes' location. In this way when a player associates a pawn to a socket, the cube senses the surface tilt over 2 axes using the accelerometer, the aggregated value is matched against socket fingerprints and thus the position of the pawn on the board is updated in in the game engine.

Figure 6 .

 6 Figure 6. Game sessions with Don't Panic low tech version.

Working team: 1

 1 Junior Researcher (me), 1 PhD (Simone Mora), 1 master student Spent time: 1 year Related publications: 2 conferences INTERACTION PARADIGMS

Figure 9 .Figure 10 .

 910 Figure 9. Two mockup from the MoDo game

Figure 11 .

 11 Figure 11. (a) Main screen; (b) A maximum impact flood

Figure 1 .

 1 Figure 1. The classification phase in the collecting memories workshop

Figure 2 .

 2 Figure 2. The materials used for the phase b of the workshop

Figure 3 .

 3 Figure 3. Phase c) of the workshop In phase d) we wanted to explore different elements for interactivity. This phase was conceived to join the physical and the functional experience.

Figure 4 .

 4 Figure 4. A conductive ink prototype

Figure 5 .

 5 Figure 5. The actual digital map and its 3D model

Figure 8 .

 8 Figure 8. 3D models for the iconic elements

Figure 10 .

 10 Figure 10. The usage of a Penfriend to identify objects.

Figure 11 .

 11 Figure 11. A page of sketch for the totem shapes .

Figure 12 .Figure 13 .

 1213 Figure 12. A totem positioned on the map

Figure 14 .

 14 Figure 14. Two participants exploring the map prototypes.

Figure 14 .

 14 Figure 14. Participants exploring different materials

Figure 16 .Figure 17 .

 1617 Figure 16. La tailleuse de chanvre, Eugène Damas

Figure 18 .

 18 Figure 18. The 3D version of the painting.

Figure 19 .

 19 Figure 19. Two particulars of the 3D printing They way these two elements were rendered (a very spiky rendering), was meant to reproduce the non-pleasurableness of the activity. The rendering wanted thus to give an unpleasant feeling at the touch.

Figure 20 .

 20 Figure 20. The director of the Musée de l'Ardenne, Carole Marquet-Morelle with the original and the 3D printed version of the painting

Figure 21 .

 21 Figure 21. A visually impaired person exploring the Sisley reproduction

Table 1 .

 1 A summary of different affordances in the above described researches

	Parts of the	Different	Kinestesis/	Even if no	Created by the
	system	devices create	movement	explicit		interaction
	designed for	different	coordination	content was	patient/therap
	re-learning	affordances		created	as	ist.	The
	motor skills	(Upper limb	Touch stimuli	cognitive		emotional
		body	through	affordance,		affordance
		rehabilitation)	devices	learning	of	works	to
				motor control	continuously
				is a process in	readapt	the
				which motor	interaction
				aspects		between the
				continuously	patient and the
				interact with	therapist
				sensory and
				cognitive
				processes.

  PervasiveHealth 2011, 530-537. IEEE, 2011. VIII. Ines Di Loreto, Liesjet Van Dokkum, Abdelkader Gouaich, Isabelle Laffont, Mixed Reality as a Means to Strengthen Post-stroke Rehabilitation. International Conference, Virtual and Mixed Reality 2011.

	CHAPTER 2 -MULTISENSORY LEARNING FOR
		Lecture Notes in Computer Science 6774, 11-19.
	IX.	Di Loreto, I. (2017). Cultures of Participation in the Healthcare Field: Could a VideoGame-Based
		Perspective Be Useful?. In eHealth 360° (pp. 65-74). Springer, Cham.
	X.	Abdelkader Gouaich, Ines Di Loreto. Post Stroke Social Gaming. RR-12009, 2011, pp.0-9. 〈lirmm-
		00679185〉
	XI.	Guillaume Tallon, Antoine Seilles, Geoffrey Melia, Sébastien Andary, Ines Di Loreto, Blain Hubert,
		Intérêt du serious game Hammer & Planks pour améliorer l'autonomie fonctionnelle chez la personne
		âgée institutionnalisée. ICEPS 2015 -Congrès international sur la recherche interventionnelle non
		médicamenteuse. 2015.

  Societal aspect/problem: In Italy most of the emergency workers are volunteers and 20 http://www.zmangames.com/boardgames/pandemic.htm21 http://www.hasbro.com/monopoly/en_US/ not trained as professionals→ differences in training levels, learning on the field What is the role of tangible artifacts in learning soft skills?-What is the role of multimodality? -What is the role of social communication in learning soft skills?

	Partners : NTNU (Norwegian University of science and technology) + Piedmont civil
	security (Italy)
	Stakeholders: trainers (experienced emergency workers), junior emergency workers
	Initial Research questions:
	-Working team: 1 Junior Researcher (me), 1 Senior researcher (Monica DIVITINI); 1 phd
	student (Simone MORA); 6 master students
	Spent time: 1 year
	Related publications: 1 journal, 4 conferences
	SOME REFLECTION ON INTERACTION CHALLENGES WHILE IMPLEMENTING A COMPUTER
	BASED BOARD GAME:
	Description	and	aim:	Mirror	(EU	project	-
	http://cordis.europa.eu/project/rcn/95667_en.html)
	Creation of a serious board game for training emergency workers on panic
	management/soft skills in a simulated city.

IDENTITY CARD OF THE PROJECT: DONT'PANIC -BOARD

VERSION

(2012)(2013)(2014)

Table 1 .

 1 A summary of different affordances in the above described researches

		Functional	Physical	Sensory	Cognitive		Emotional
		affordances	affordances	affordance	affordances	affordances
	Don't Panic -	Game	Create by the	Touching (	Soft skills (telling	Created by the
	board	dynamics,	different	kinestetic),	the	right	interaction/di
		game objects, gameplay, content of the game	gaming objects and by the token/constra ints approach	hearing, seeing	information at the right person at the right moment)	scussion (voice, contact, deictic/indexic eyes
							al	pointing,
							moving	and
							constraining
							objects)
							Created by the
							scenario of the

Table 1

 1 summarizes the different affordances in the previous described works.

		Functional	Physical	Sensory	Cognitive		Emotional
		affordances	affordances	affordance	affordances	affordances
	Augmented	the	pair	Created by the	Created by the	Create	a	??	To	be
	map	shape/materia	size, the shape,	material of the	mental	map	explored
		l		etc, of the map	map	(learn	the
				and the totems	(touch/kineste	topology of a
					tic);	place before
						going	to
					Created by the	explore it)
					audio
					description
					(hearing)
	Smart Art	Shapes		Created by the	Created by the	Create	a	Created by the
				size, the shape,	material of the	mental		"being
		Audio content	etc, of the	map ity of the	representation	togheter"
				painting	reproduction;	of	the
						paintings
					Created by the
					audio
					description
					(hearing)

I use here the term translation as defined by Latour, that is: a process of arranging heterogeneous interests into a new order, thereby creating something new (as described in Pandora's Hope[13] Chapter

2: Sampling the Soil in the Amazon Forest). 2 It is worth to note here, that I use here paradigm in the sense of the technological paradigm used to implement a particular interaction (and thus in a loosely way with respect to the more Kuhnian approach used in[11]).

https://en.wikipedia.org/wiki/The_English_Patient

VS4 The building images are courtesy from Guillaume Ducellier and Thomas Dautreppe work on a UTT building digital twin.

I'm totally aware that I'm overgeneralizing, as there are more and more good examples of serious games.

A video of the system is available at https://www.youtube.com/watch?v=NXTer-ZltsI

In vertically scrolling shooters, the action is viewed from above and scrolls up (or very occasionally down) the screen.

For an in depth discussion on the topic and its link with game see[I] in the author's list

https://www.guitarhero.com/fr/game

https://scholar.google.fr/citations?hl=en&user=N6KeA-MAAAAJ&view_op=list_works&sortby=pubdate

For an in depth analysis of cultures of participation in the game for health context see IX in the author's publications list

http://cordis.europa.eu/project/rcn/95667_en.html

http://teseolab.org/

A low-tech version of the game was designed and tested before going through the development of the game, and validated the need for a computer based version. See the Methods and techniques part of this section.

https://www.youtube.com/watch?v=dF0NOtctaME

Blast Theory, Can you see me now?, http://www. blasttheory.co.uk/bt/work_cysmn.html, 2012.

https://www.blasttheory.co.uk/projects/uncle-roy-all-around-you/

http://www.ign.fr/ IGN is a website who allows to buy detailed maps describing a territory.

http://musees-de-france-champagne-ardenne.culture.fr/musee_ardenne.html

What is essential is invisible to the eyes, an iconic phrase in Antoine de Saint-Exupéry's book, The Little Prince.

Institut de Formation Technique Supérieur (IFTS). http://www.univ-reims.fr/ifts

https://fr.wikipedia.org/wiki/Eug%C3%A8ne_Damas

"You can let people "see" through emotions, through moments of sharing, through comments that make alive the paintings. We can let people sees of course with tactile supports like those of our project, but especially by sharing a moment with the group, and better knowing the other persons.".

the players have to learn to use their competences taking into account also the other players and the environment they are playing in. -What is the role of tangible artifacts in learning soft skills? -What is the role of multimodality? -What is the role of social communication in learning soft skills? -Analyze the role of the body (movement and physical actions) for learning (supposed adding movement→ better learning) -Analyze the role of learning in actual environment

EMOTIONAL AND PHYSIC INTERACTION IN MOBILITY

With the global climate changes we can expect an increased occurrence of floods, even in areas not previously affected by flooding. Having the possibility to add to the traditional crisis management organization workforce also a higher number of citizens is thus very important. Despite of this, the general public seems to be ignorant of the possibility and consequences of a major flood. We thus decided to create a game for promoting citizens' preparedness to flooding situations, called Flooded. Flooded was designed using actual flooding management elements, but having has main objective citizens' sensitizations. In this case I co-designed with a master student the game dynamics and interactions, as the student was particularly interested in serious games design.

Learning elements: increasing preparedness. Flood preparedness is used to describe any precautionary measures taken against impending flood(s). To increase the preparedness of citizens the most effective strategy is to inform them of the consequences of a flood. [12] The first step to promote flood preparedness is to sensitize citizens to the dangers and consequences of a flood. Awareness of appropriate actions to take in case of flood warnings is also vital. [13] A key element is knowing where and when to evacuate and to know where to evacuate the citizens must be aware of their local area. This means that citizens need to know where it is safe to go and how to get there. For a citizen to be effectively prepared for a flood he also needs to have the required soft skills (collaboration, communication, and the like). This includes listening and correctly responding to flood warnings, and communicating effectively when helping with flood management. -What content, what interactions, and what relevance for a multi-sensory map as a medium of mediation to make a rural area discoverable by a visually impaired audience?

IDENTITY CARD OF THE PROJECT:

-How to adapt the brainstorm and the co-creation workshop for a visually impaired audience?

Working team: 1 Researcher (me), 1 young researcher (Tom Giraud), 1 engineer (Melanie Araujo Martins), 1 designer (Luce Aknin), Hervé Bertin (CNC carving),

Spent time: 1 year

Related publications:-----ongoing

SOME EXPLANATION ON VISUAL IMPAIRMENT

As all pathologies involving the body, visual impairment is complex and collects disparate forms of impairment.

Visual impairment, also known as vision impairment or vision loss, is a decreased ability to see to a degree that causes problems not fixable by usual means, such as glasses [04][05]. Visual impairment is often defined as a best corrected visual acuity of worse than either 20/40 or 20/60 [06]. When one or more parts of the eye or brain that are needed to process images become diseased or damaged, severe or total loss of vision can occur. In these cases,

MULTI-SENSORY ARTIFACTS FOR SOCIAL AND EMOTIONAL INTERACTION

In this part I will talk about another project who aims at empowering visually impaired people through social and emotional interaction (based on physical objects). The aim of the SmartArt project is to reduce inequalities by allowing access to art and culture to visually impaired people. The objective of the project is to design an art exhibition that makes sense to them, in particular by appealing to the emotional aspect created by art, and not to the rational ones, and that could be lived togheter with non visually impaired people. In this context, new ways of making art accessible are explored, examining not only ways of presenting artwork in a manner that can be appreciated using all five senses, but also emphasizing what people value when visiting a museum. In order to design these new museum experiences and interaction modalities with art, the project adopts a participatory design approach, examining aspects of art, technology and social inclusion. At the end of the SmartArt project (july 2018), an actual exhibition took place in the Musée de l'Ardenne 27 . It will thus be the opportunity to share with a larger public of visually impaired and non impaired people the artifacts created in the project. -Could a multi-sensory environment be useful for a better understanding of a painting by visually impaired people?

-Is this kind of approach is generator of social exchanges based on emotional and not rational aspects.

Working team: 1 Researchers (me), 1 Junior researcher (Karine Lan), 1 trainee (Frederique Champigny)

Spent time: 2 years

Related publications: 1 journal

The design objective of SmartArt is not uncommon and an important number of ethnographically informed studies in museums (e.g., [26,27]) has already been held. In addition, based on the observation that both work and non-work settings involve socially organized joint activity of mundane practices, some researchers explored how the methods of collaborative organization of joint activities in work environments can be transposed to investigate non-work environments [28]. However in the case in the SmartArt project:

1. There are no actual practices to observe (visually impaired people usually do not go to museums, and the social innovation of the project is precisely to reverse that situation); 2. What we will be designing is not known beforehand/at the beginning of the project (it can for some part be technological, or not at all) and will precisely emerge out of the requirements analysis done as part of our PD research. As our objective was to gain an adequate understanding of the relationship between design, use, and the social and political context we thus used, amongst other qualitative orientation to design, ethnography, keeping in mind some guiding questions such as how do we do to capture the details of 'real-time real-world' actions [29]? and how do we study phenomena in their natural setting [30]?

Our position was thus an interest in gaining insights about the experience of visually impaired people in different situations, how they express their emotions and feedback, how they interact with each other in the workshops, and by extension, what matters to them, to inform what should be included in the art exhibition.

Focus group on emotions

The first action we took for the project was to held a focus group with seven visually impaired people and divided in three parts. A series of specific questions was proposed to the participants in the first part of the workshop. The questions asked were mostly related to memories, and in particular to "pleasant" (and unpleasant) personal emotions. In the second Enabling people inclusion through participatory design projects challenges the democratic stance of this field of research as well as the way of doing research and design. This involves not only to use the appropriated techniques, but also to question our positioning as researchers, for the malleable position we are when doing participatory design. Thus learning in this context is not only what is conveyed by the system we are creating, but also our own (as researchers) learning, in a reflexive fashion.

The open question in this case would be: how we do integrate a reflective approach in our everyday research?

For researchers, reflecting on research principles and methodologies is a crucial experience, because it helps moving from a pure « academic» sometimes «rhetoric» sometimes «instinctive» activity, to a full personal reflection on the limits and opportunities for our disciplines of the methods we use. Historically, two types of reflection in which individuals engage are described: reflection-on-action and reflection-in-action [01]. Reflection-on-action is defined as a process in which individuals reflect on actions and thoughts after they have taken place. Reflection-in-action takes place as the action occurs. Later, the concept of reflection for action [02] a reflection guiding future action basing on past thoughts and actions and takes place has being introduced.

How to foster all these types of reflection for our research practices is in my opinion an open question that is worth to explore.

Another more practical methodological question would be a deeper analysis of nonconventional validation/evaluation methods (like the MIG experience for Hammer and Planks, or the guided tour of the city for SmarArt), as bringers of non-expected information.

One last reflection could be done on the importance of the size of the impact of our research projects: what is the success rationale of a project? The number of potential users like for the Don't Panic project, or the number of actual users at the end of the project like for the augmented map or the SmartArt project?

OPENING FOR THE FUTURE:

If we want to paraphrase anew the questions we detailed in previous section under a more applied HCI point of view we could say they can be resumed to:

➔ What are the possible combinations of affordances for the creation of artifacts for multisensory learning? ➔ Which channel (or combination of channels) conveys better which learning element, and which channel on the other hand augment information overloading in a multisensory technological system? ➔ How we do translate these aspects in (multimodal?) interaction? ➔ What is the role of contextual (situated) action in multisensory learning? ➔ What is the role of distributed and differentiated affordances for learning?

➔ How can we take into account multisensory elements in an evolving technological system?

➔ What is the role of emotional affordances for learning systems? ➔ How we do integrate a reflection-based approach in our everyday research?

This last methodological question needs in my opinion a collective reflection, as it has not only ethical consequence, but also ask for actions to train young researchers to it. In this case it is not only a matter of reflecting on the "the position of the researcher" (should we be trying to be neutral at maximum when engaging in research practice?) but how to manage the fact that we are changing in the same exacting moment we are conducting our research, involving thus all the kind of reflection we discussed above. Again and just to be clear, it's not a matter of keeping a reflective journal, but a matter of integrating change in a scientific protocol.

The other questions inform a more practical and less ideological approach. If we want at least be able to create some "best practices" to design multi-sensory systems for learning then a global approach to these questions is required. The answer to these questions also demand a more systematic positivistic evaluation approach (for example isolating the variables to test, in particular in the case of the sensory aspects).

Short-term actions

A first step to explore these questions will be the PhD thesis I will be co-supervising from September 2018 with Denis Mottet, professor on human movement science at Université de Montpellier (France). The thesis proposes to use of augmented objects -i.e. technologically augmented objects that take the form of everyday objects -to help people learn and apply healthy habits. In order to achieve this objective, the augmented objects will be used to create active learning (action based training and learning). The thesis project is particularly interesting in order to answer the previous listed questions as:

-Each augmented object can be structured to use a particular affordance (or set of affordance) in a controlled setting, to see which sensory affordance is better adapted to which activity.