Le Mathieu

Pierre Goc

Samuel Dragicevic

Jeremy Huron

Jean- Daniel Boy

Fekete

Smarttokens

Mathieu Charles Perin

Romain Di Le Goc

Jean-Daniel Vozzo

Pierre Fekete

Dragicevic

S Y N T H È S E

La récente émergence des smartphones nous a conduit à voir les interfaces tactiles prendre une place importante dans notre façon d'interagir avec le monde virtuel. Celles-ci offrent de nombreux avantages : en effet, la technologie tactile est mobile et polyvalente, et s'incorpore facilement pour créer toutes sortes d'appareils interactifs. Ceux-ci ne se limitent désormais plus aux seuls smartphones, mais aussi aux montres connectées, ordinateurs portables et autres tablettes numériques, dans nos voitures ou encore dans nos appareils électroménagers. Les interfaces tactiles permettent de manipuler les objets virtuels directement sous le doigt dans nombre de contextes. L'interaction est plus variée et plus riche qu'avec les interfaces traditionnelles comme la souris et le clavier, notamment en utilisant plusieurs doigts ou grâce à différents gestes comme le pincer pour zoomer ou le glisser. Toutefois, les interfaces tactiles contemporaines sont loin d'être parfaites. Une de leur principale faiblesse réside dans leur manque de physicalité : l'interaction consiste uniquement à faire glisser les doigts sur la surface uniforme et lisse qu'est la vitre de l'écran. Il est alors impossible d'utiliser le sens du toucher pour se repérer dans l'espace comme nous le faisons tous les jours dans le monde physique. L'attention repose alors seulement sur la vision, négligeant les autres capacités sensorielles. Une approche différente consiste à s'inspirer du monde physique dans lequel nous vivons, et de la façon dont nous interagissons avec celui-ci.

Au coeur de cette interaction avec le monde qui nous entoure se trouvent nos mains. Les mains humaines sont le fruit de millions d'années d'évolution et permettent de manipuler les éléments du monde physique. L'humain a appris, à travers les âges, à maitriser ce formidable outil que sont les mains dans différents domaines comme la communication, les arts, l'artisanat ou encore pour les tâches quotidiennes, à mains nues ou en utilisant des outils. Cependant, les mains restent souvent négligées et sous-utilisées dans les interfaces homme-machine. De la recherche en réalité augmentée et en informatique ubiquitaire sont apparues, au début des années 1990, les interfaces dites tangibles. Elles visent à utiliser le monde physique et les objets qui le composent pour interagir avec le monde virtuel. Ces interfaces peuvent ainsi mieux mettre à profit les facultés naturelles humaines. Le monde de la recherche explore depuis des technologies et des méthodes aspirant à combler le vide entre le réel et le virtuel.

Toutefois, bien que prônant davantage de physicalité dans les interfaces Homme-Machine, la plupart des systèmes tangibles existants sont basés sur du contenu graphique projeté ou sur des écrans en utilisant seulement quelques objets passifs. De plus, la majorité de ces systèmes sont conçus pour être utilisés dans des contextes bien précis. Ceux-ci sont développés sur mesure en utilisant des objets dédiés. Bien que cela constitue une force i des interfaces tangibles, en les rendant plus abordable et simple d'utilisation, cette spécificité peut aussi être une faiblesse importante car elle nuit à la polyvalence. Les nouvelles technologies évoluent rapidement dans divers domaines tels que les objets connectés et l'Internet des objets ou encore les technologies mobiles et portables. Je pense que nous avons désormais l'opportunité de repousser les limites de l'interaction tangible, vers des interfaces à la fois dynamiques et polyvalentes, pour nous aider à penser l'interaction homme-machine au-delà des écrans.

Cette thèse propose une nouvelle direction vers des interfaces tangibles à la fois plus physiques et plus polyvalentes. Influencé par la robotique et la notion de matière programmable, je présente dans ce manuscrit mes travaux visant à rendre les interfaces tangibles plus polyvalentes et plus physiques afin de réduire l'espace entre le réel et le virtuel. Pour ce faire, j'étudie et conçois des dispositifs technologiques permettant d'interagir avec le monde numérique exploitant au mieux le potentiel de nos mains. Je commence par examiner l'état de l'art et souligne le besoin d'approfondissement dans cette direction. J'y observe la rigidité des systèmes existants, limitant leur utilisation et leur diffusion, de même que l'utilisation récurrente d'écrans et autres dispositifs de projection comme moyen de représentation du monde numérique. Tirant les leçons de la recherche existante, je choisis d'orienter mes travaux autour de dispositifs physiques constitués uniquement de collections d'objets génériques et interactifs. Mon but est d'apporter plus de polyvalence aux interfaces purement tangibles. J'articule pour cela ma recherche en quatre temps.

Je mène tout d'abord une étude comparant les interfaces tangibles et tactiles, dans le but d'évaluer de potentiels bénéfices de l'utilisation de collections d'objets physiques. J'étudie conjointement l'influence de l'épaisseur des objets sur la manipulation. Les résultats suggèrent tout d'abord de modérer les conclusions de nombre d'études existantes, quant aux avantages de la tangibilité en terme de performances. Ces résultats confirment toutefois l'amélioration de l'expérience subjective lors de l'utilisation de dispositifs physiques, expliquée par une plus grande variété ainsi qu'une plus grande fiabilité des manipulations réalisées.

Je présente dans un deuxième temps SmartTokens, un dispositif à base de petits objets capable de détecter et reconnaître les manipulations auxquelles ils sont sujets. J'illustre les SmartTokens dans un scénario de gestion de notifications et de tâches personnelles. Je poursuis en introduisant les Interfaces en essaim, une sous-catégorie des interfaces tangibles, constituée de collections de nombreux robots autonomes et interactifs. Pour les illustrer, je présente les Zoïdes, une plateforme ouverte pour développer des interfaces en essaim. Je démontre leur polyvalence avec un assortiment d'applications, et clarifie les règles de conception des interfaces en essaim.

Dans un quatrième temps, je me concentre sur le domaine de la visualisation d'informations dans le but d'illustrer la polyvalence des interfaces en essaim. Je définis les physicalisations composites de données comme un sous-ensemble des visualisations physiques, et en implémente plusieurs exemples en utilisant les Zoïdes. Je termine en discutant les perspectives et futures directions, et en tirant les conclusions des travaux réalisés au cours de cette thèse.

A C K N O W L E D G M E N T S

Before starting this dissertation, I would to acknowledge everyone who made this adventure possible.

The first and very important ones are Pierre Dragicevic and Jean-Daniel Fekete. Thank you for giving me the opportunity to pursue research on a topic I am passionate about. Thank you Pierre for being such a great advisor and guide, always available, and pushing me when I was losing motivation. I will never look at writing and statistics the same way again. Thank you Jean-Daniel for bringing wisdom, valuable input and advices all along. You two trusted me and gave me wonderful conditions to carry out my work, I am truly grateful for it.

I would like to thank the member of the jury -Laurence Nigay, Brygg Ullmer, Céline Coutrix, Michel Beaudouin-Lafon and Sean Follmer -for reviewing and attending (physically and virtually) my defense.

I would also like to thank Aviz, the team I had the chance and the pleasure to be part of. I cannot think of a better place to pursue my PhD. Thank you to all the wonderful people I had the chance to meet and who made these three years very special: Yvonne Jansen for all the fruitful discussions, Charles Perin for being such a good friend at the lab, at the bar, during conferences; Pascal Goffin for being my companion in misery all along; Wes Willet and Lora Oehlberg for all the great conversations; Lonni Besançon, Petra and Tobias Isenberg, Benjamin Bach, Nadia Boukhelifa and Emmanouil Giannisakis for joining in the stimulating environment that is Aviz; Katia Evrat for being so nice and helping me getting through Inria's administrative maze; Frederic Vernier, André Spritzer, Evanthia Dimara, Paul Lapides, Alice Thudt, Manfred Micaux, Kai Lawonn, Amira Chalbi and Conglei Shi for being such great office mates and coping with my music every day. I want to thank Romain Di Vozzo dearly for being such a great colleague and friend. It was truly a pleasure to be part of the FabLab with you, I enjoyed it a lot and learned so much working with you.

On the Californian side, I would like to thank my second team, Shape Lab, for having me during three wonderful months; thank you very much Sean Follmer for this great collaboration and for offering me the opportunity to continue this adventure, I am really looking forward to it; Renée Chao for always being kind and helpful. I had the great pleasure to join the Shapies, I would like to thank them for their warm welcome. I enjoyed working with you all very much: Lawrence Kim, Inrak Choi, Shenli Yuan, Alexa Siu, Ernesto Ramirez and Evan Strasnick for making every day at the lab so enjoyable. I would also like to thank all the Lofties, a wonderful group of people and in particular Doruk Gurel, Dickson Chow, So Yeon Park and v vi Philipp Dupree with whom I had the pleasure to share the kitchen with, and so much more during these three short months.

I would like to thank all the members of Inria's In/Ex Situ and Ilda for all the discussions at the lab and during conferences, Fanny Chevalier and Stéphane Homer Huot from Mjolnir for making conferences so much more fun.

I want to thank Olivier Bau for being the first to give me a chance to step into the world of research, for sharing his experience and guiding me to create my own research path, and without who I would not be where I am now. I also want to Franck Crison for being the most inspiring professor and mentor, for passing me his passion for Electronics, supporting and guiding me in my projects and endeavors all along my engineering studies. Thank you my friends who were there for me even when I was struggling to find time for you: Pierre Abi-Aad, Yann Pichon, Benoit Gergaud, Clément Riant, Laurent Chane-Law and Florent Coïc.

Thank you Jeremy Boy for helping me damage my liver, and most importantly for being always present, supportive and helpful. Thank you Pauline Gourlet and Can Liu for being great confidants, enduring my bad temper, always cheering me up and encouraging me in hard times. Thank you Samuel Huron for being always so kind and enthusiastic, motivating and always willing to help, and your wife Tiphaine Rivière for all the great dinners and discussions. Thank you Christoph Kinkeldey for being cool and refreshing, for listening to my many complaints, dragging me to Bar Ourcq to cheer me up, but above all for correcting my English. Thank you Hye Jin Song for all the interesting discussions, cooking sessions and Hangul classes. You all made my time in Paris unforgettable. Thank you Xuemei Yu for being so affectionate, kind and supportive. I am truly sorry I was not present for you.

Thank you Shu Yuan Hsueh for making the last months of this adventure so special and making every day beautiful, for broadening my horizons and challenging my ideas, for bringing me so much. I would also like to thank dearly my family -Simon, Fanny, Nicolas et André -for supporting me during these 28 years. Finally, I would especially like to thank my mother Marie-Line for always being there in the difficult moments, inspiring me and supporting me in my choices and giving me strength when I needed it the most.

introduction

1 A comparison of possible interactions with "Pictures under Glass" [START_REF] Victor | A brief rant on the future of interaction design[END_REF] with typical manipulations we do everyday. . . . . . . . . . . . . . . . . . . . . Figure 1.2 metaDESK [START_REF] Ullmer | The metadesk: models and prototypes for tangible user interfaces[END_REF] allows to interact with a map by manipulating miniatures of the buildings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 1.3

The tension between physicality and versatility in nowadays interfaces. Current technologies focus on one of these, but ideal TUIs would combine a high degree of physicality while supporting versatility. . . . Figure 1.4

The BRETAM Model [START_REF] Gaines | Modeling and forecasting the information sciences[END_REF] describes in six steps the evolution of information sciences. . . . . . . . Figure 1.5

On the left: a LEGO brick. On the right: A polar bear sculpture [START_REF] Kenney | Art with lego bricks: Polar bear[END_REF] Flowblocks [START_REF] Zuckerman | Extending tangible interfaces for education: digital montessori-inspired manipulatives[END_REF] are Montessoriinspired building blocks letting children experiment with abstract concepts of dynamic behavior. . . . . . . Figure 2.8

Perlman's TORTIS Slot Machine [START_REF] Perlman | Using computer technology to provide a creative learning environment for preschool children[END_REF] (left) allow to program a robot using physical cards. LEGO Mindstorms [LEGO, 2017] (right) combine LEGO bricks with a programmable brick, sensors and actuators to create interactive assemblies. . . . . . . . . . . . Figure 2.9

Pico [START_REF] Patten | Mechanical constraints as computational constraints in tabletop tangible interfaces[END_REF] is an interactive tabletop surface capable of tracking and moving small objects to solve spatial layout problems, here laying out cellular phone towers. . . . . . . . . . . . . . . . . . . . . . Figure 2.10 Tangible Query Interfaces [START_REF] Ullmer | Tangible query interfaces: Physically constrained tokens for manipulating database queries[END_REF]] allow to interact with visualizations using tangible controllers. Figure 2.11

Physicalizations [Jansen, 2014a] are objects representing data using physical properties such as shape or material. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.12

The reacTable [START_REF] Jordà | The reactable: exploring the synergy between live music performance and tabletop tangible interfaces[END_REF] is a tangible music interface using tokens to create electronic music. . . . . Figure 2. 13 Using particle jamming [START_REF] Follmer | Jamming user interfaces: programmable particle stiffness and sensing for malleable and shape-changing devices[END_REF], Tunable Clay can modify the stiffness of its material support 3D modeling. . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.14

Lumen [START_REF] Poupyrev | Lumen: interactive visual and shape display for calm computing[END_REF] is a low resolution shape display that can communicate 2D graphical images and augmented with dynamic physical shapes (left). inFORM [START_REF] Follmer | inform: dynamic physical affordances and constraints through shape and object actuation[END_REF] is a high resolution shape display and can create complex shapes and senses user actions (right). . . . . . . . . . . . . . . . . . Figure 2 Madgets [Weiss et al., 2010a] are actuated physical widgets on multitouch tabletop. . . . . . . . . . . . . . . Figure 2.17

Tangible Bots [START_REF] Esben | Tangible bots: interaction with active tangibles in tabletop interfaces[END_REF] (left) provide physically actuated controls to interact with multitouch tabletop surface, here used for musical creation . Thumbles [Patten Studio, 2014] (right) are robots that can be used as handles to interact with digital media, but also provide dynamic tactile feedback. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.18

In the movie X-Men, a surface capable of changing its shape dynamically (Image from [START_REF] Dragicevic | List of physical visualizations[END_REF]). . . . . . . . . . . . . . . . . . . . . . . . . Figure 2

.19

BitDrones [START_REF] Rubens | Bitdrones: Towards levitating programmable matter using interactive 3d quadcopter displays[END_REF] are flying interactive devices supporting direct manipulation, allowing room-scale tangible interaction . . . . . . . . . . . . . . . The continua between compositeness and monolithicity (top), and between homogeneity and heterogeneity in TUIs (bottom). . . . . . . . . . . . . . . . . . . . . Figure 3.1 Chips (left) and pucks (right) as used in our study. . . Figure 3 Frequencies of occurrence of manipulation gestures and manipulation errors, for Chips and for Pucks. . . . Figure 3.9

Four chips lost during a bulldozer push (top), while no loss happens with the pucks (bottom). . . . . . . . . Figure 4 A short strip illustrating the mail notification system . Figure 5.1

Zooids can be held as tokens, manipulated collectively or individually, behave as physical pixels, act as handles and controllers, and can move dynamically under machine control. They are building blocks for a new class of user interface we call Swarm User Interfaces. The continuum between "things" and "stuff". . . . . . Figure 5.9

Alien from the game Space Invaders from Taito (1978) and main character from the game Mario Bros by Nintendo (1983)

I N T R O D U C T I O N

There is more information available at our fingertips during a walk in the woods than in any computer system, yet people find a walk among trees relaxing and computers frustrating. Machines that fit the human environment, instead of forcing humans to enter theirs, will make using a computer as refreshing as taking a walk in the woods. -Mark Weiser [START_REF] Weiser | The computer for the 21st century[END_REF] Technology occupies a growing part of our everyday lives, as we spend more and more time using digital media. This is partly due to democratization of smartphones, tablets, and other multitouch devices.The popularity of multitouch technology started to rise in 2007, with the release of Apple's iPhone: a very well designed product combined with a usable and efficient operating system led to a major commercial success. Other manufacturers quickly followed, benefiting from this success. Sales have been multiplied by more than 10 since 2008, leading to over 1.4 billion units sold in 2016 [START_REF] Han | Characterizing the technological evolution of smartphones: Insights from performance benchmarks[END_REF]. Touch devices have rapidly expanded beyond smartphones and are now omnipresent in our everyday lives, from watches to laptops, tablets, cars, even kitchen appliances, and menus in restaurants.

Multitouch technology offers several benefits: mobile and easy to integrate, multitouch devices are now omnipresent in our everyday lives. Interaction has become richer and more diverse, since multiple fingers can be used at the same time or abstract gestures such as zooming or swiping are facilitated.

But one of most the important benefits of multitouch interfaces is versatility: the unique combination of a fully programmable graphical display with the ability to manipulate virtual objects provides versatility. This allows for a wide variety of use-cases, ranging from traditional voice calls and messages to music listening, Internet browsing, photography and so on. Touch interfaces are called soft interfaces, meaning they can be reconfigured "on-the-fly".

Yet, multitouch technology is far from being perfect. One of the weaknesses of the technology is the lack of physicality. For instance, when typing on a physical keyboard, the edges of the keys help locate them without looking at the keyboard. Both the sight and the sense of touch provide feedback on the actions performed. But with nowadays multitouch interfaces, sliding fingers on a flat and smooth surface does not provide tactile cues, impairing the sense of touch and transferring all the attention on sight.

More research is necessary to keep enriching multitouch technologies and improve user experience. Yet, as Bret Victor observes that touch interaction approximatelly amounts to sliding fingers across the "glassy" surface of a screen. Actions are disconnected from the objects being manipulated, and the lack of physicality remains. He calls this phenomenon "Pictures under Glass". In his rant [START_REF] Victor | A brief rant on the future of interaction design[END_REF], he suggests to take inspiration from the physicality of the world we live in, and especially the way we interact with it.

He argues that touch interaction is only a transition, as it ignores the vast manipulative potential of hands as examplified in Figure 1.1. Human hands are the result of millions of years of evolution. As humans became bipedal, their hands were free to be used for other purposes than locomotion. Compared to the ape hand, human hands have grown to become larger, more precise and mobile, with a fully opposable thumb and strong fingers [START_REF]Evolution of the human hand: the role of throwing and clubbing[END_REF]. This opposable thumb forms the most beneficial quality of the human hand; it enables to grasps objects in various ways. Humans have mastered using them in many different ways for various uses. From arts, crafts, communication or simple everyday actions, bare hands or using tools, hands remain one of the keys to interact with the world. And yet, hand capabilities are still overlooked by user interfaces.

introducing tangible user interfaces

In the 1990's, Graspable User Interfaces [START_REF] George W Fitzmaurice | Bricks: laying the foundations for graspable user interfaces[END_REF] and Tangible Bits [START_REF] Ishii | Tangible bits: towards seamless interfaces between people, bits and atoms[END_REF] started leveraging the benefits of the physical world for Human-Computer Interaction (HCI), laying the theoretical foundations for Tangible User Interfaces (TUI). Making use of human capabilities to interact with the physical world, TUIs use physical objects as handles into the digital world to manipulate information. For instance, metaDESK [START_REF] Ullmer | The metadesk: models and prototypes for tangible user interfaces[END_REF] was used to showcase an interactive map of the MIT campus. Physical miniatures representing the buildings were placed on a table and the map was projected, accordingly to the position of the buildings (see Figure 1.2 left). Manipulating the miniatures was caused the map to adjust automatically to preserve the consistency between the physical and the digital worlds. By pulling two buildings apart the user was able to zoom in and rotate the map(see Figure 1.2 right).

Figure 1.2: metaDESK [START_REF] Ullmer | The metadesk: models and prototypes for tangible user interfaces[END_REF] allows to interact with a map by manipulating miniatures of the buildings.

Motivating TUIs

Nowadays, Tangible User Interfaces (TUIs) constitute an important part of the HCI research community, and a large amount of technologies and prototypes, interaction techniques, taxonomies, and frameworks to improve TUI has been presented.

The physicality of TUIs was shown to provide benefits over Graphical User Interfaces (GUI) [Fitzmaurice and [START_REF] George | An empirical evaluation of graspable user interfaces: towards specialized, space-multiplexed input[END_REF][START_REF] Michael S Horn | Tangible interaction and learning: the case for a hybrid approach[END_REF][START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF]. Taking advantage of the physical design and affordances, characteristics like shape, color, weight or material can be used to convey the meaning and the function of an object, or how to signify the way how to interact with it. Objects no longer need to be generic but can on the contrary be specific and iconic, tailored to one function only. The specificity of objects improves the association between actions and effects, making interfaces more discoverable. Moreover, physicality also enables both hands to be used simultaneously [START_REF] George W Fitzmaurice | Bricks: laying the foundations for graspable user interfaces[END_REF], providing better conditions for collaboration.

In a comparison between GUIs and TUIs, Zuckerman et al. found that people still preferred TUIs over GUIs despite slower and less accurate interactions. TUIs were considered more enjoyable, stimulating, and overall more realistic nature [START_REF] Zuckerman | To tui or not to tui: Evaluating performance and preference in tangible vs. graphical user interfaces[END_REF]. [START_REF] Donald A Norman | Introduction to this special section on beauty, goodness, and usability[END_REF] states that the use of TUIs lies in the behavioral beauty that he defines as "the home of highly learned skills", claiming that "we must move beyond products that perform well toward products that deliver pleasure, beauty and fun" [START_REF] Donald A Norman | Introduction to this special section on beauty, goodness, and usability[END_REF]. These results support that users can favor the comfort and enjoyment of TUIs over usability and performances of GUIs.

The limitations of research in TUI

Tangible User Interfaces are very promising as they leverage human lifelong experience to interact with the physical world. Yet, research prototypes and demonstrations we have seen so far remain hybrids: they are mostly anchored in the digital world, combining tangibles for input with digital display (e.g., screens or projection). Moreover, existing TUI systems use other interaction modalities such as multitouch devices or mid-air and gestural interaction. While these may help overcoming technological barriers, the research community has not put enough efforts in pushing TUIs in the purely physical direction it has always been promoting. Researchers have also been advocating specific designs, using dedicated physical objects to help improving the accessibility of TUIs. But at the same time, using these excessively can be harmful as there is the danger that an each interface may be trapped in the situation it was designed for.

Physicality and versatility: a difficult combination

The benefits of physicality have been established and are now recognized by the research community. Using physicality in HCI takes advantage of human natural abilities to manipulate and interact with the physical world. Versatility is a strength of devices such as multitouch devices or desktop computers and laptops. It allows us to perform diverse tasks with the same device. Opposed to this, TUIs do not typically show a high degree of versatility.

Looking back at the large body of research in the field, current trends seem to show that a high degree of physicality entails high specificity whereas great versatility comes with a low degree of physicality. Illustrated in the Figure 1.3, I describe this relation as the tension between physicality and versatility. More research is needed to try to decrease this tension. But combining physicality and versatility is difficult and rises numerous challenges, both from technological and interaction design perspectives.

From the technological perspective, new devices and systems supporting physicality are needed while providing more versatility. Devices need to sense user manipulations, and potentially perceive several input channels

Existing User Interfaces

Figure 1.3: The tension between physicality and versatility in nowadays interfaces.

Current technologies focus on one of these, but ideal TUIs would combine a high degree of physicality while supporting versatility.

such as grips or gestures of different users. At the same time devices need to feature output capabilities to communicate information physically. Altering physical characteristics such as shape, colors or weight dynamically, or modifying its spatial organization are potential directions to this end. Using these new output channels, novel expressive and meaningful ways to represent digital information are needed to rise the degree of physicality of TUIs. This may be a way to substitute workarounds such as extra screens and graphical representations.

From the interaction design perspective, such new technologies raise major questions. Interacting with purely physical technology is not common, and only a few platforms support both input and output through physical media. New techniques have to be invented, evaluated and released to allow users to become accustomed with interactive physical objects.

TUI Technologies out of reach

To support versatile TUIs with a high degree of physicality, researchers as well as practitioners have to be able to access existing technologies and prototypes. Learning from the evolution of multitouch technology over the last decades could help avoid potential obstacles in research on TUIs.

Since the very first prototype presented by E.A. Johnson in 1965, multitouch interfaces went through many technological iterations and refinements. Yet, it took over 25 years to observe the step-up that constituted the commercial success of smartphones [START_REF] Buxton | Multi-touch systems that i have known and loved[END_REF].

In his essay "The Long Nose of Innovation", [START_REF] Buxton | The long nose of innovation[END_REF] looks at the history of the computer mouse. Invented by English and Engelbart in 1965, this device needed 30 years to evolve from an early demonstration prototype to a mainstream product. Despite the fact that the benefits of the mouse were obvious, Buxton states that "it still wasn't ready for prime time". The mouse as a product had to be refined to become successful. But only the emergence of GUIs with Microsoft Windows 95 and the widespread use of desktop computers led to the mouse becoming ubiquitous as an input device.

In the meantime, as TUIs started gaining interest after the first prototypes at the end of the 1970s and theories emerging during the second half of the 1990s, only a few tangible platforms have evolved from research prototypes to commercial products.

In 1991, Gaines presented a framework to describe the evolution of information sciences and determine "the key events in the development of computing" [START_REF] Gaines | Modeling and forecasting the information sciences[END_REF]. He introduced the BRETAM model, consisting of six different temporal steps: breakthrough, replication, empiricism, theory, automation and maturity (see Figure 1.4). Once the breakthrough has been achieved

Figure 1.4: The BRETAM Model [START_REF] Gaines | Modeling and forecasting the information sciences[END_REF] describes in six steps the evolution of information sciences.

in form of research prototypes, the evolution of the replication phase principally relies on the accessibility of the new technology. Considering touch technology, the major evolutions have happened as touch devices became more available, through commercial products but also thanks to movements such as the Open-Source Initiative1 by providing free material to rebuild existing devices. However, when it comes to TUIs, replication as not yet been fulfilled to this extent.

Despite a few commercial products for TUIs (e.g., Sifteo Blocks [Sifteo]), virtually no solution has been offered to reproduce TUIs research prototypes and pass the step of proof of concept. Consequently, it is very difficult to replicate user studies conducted in particular conditions, when using specific platforms. Similarly, when promising platforms for TUIs are presented in the research community, material to replicate prototypes is generally not available hence preventing further iterations.

goals

The drawbacks of TUI research identified above set the foundations for the major goals of this thesis. These are threefold: favoring physicality in TUIs, promoting versatility with TUIs and finally developing ways to make technology available.

More physicality in TUIs

TUIs have already provided evidence for potential benefits of physicality. But increasing the level of physicality of TUIs, further benefits may be unveiled. Emerging technologies are opening new opportunities to explore widely neglected fields of research such as fully physical interactive TUIs, where users would interact exclusively with physical objects.

As mentioned earlier in Section 1.1, the use of TUIs resides in the behavioral beauty of Norman's theory [START_REF] Donald A Norman | Introduction to this special section on beauty, goodness, and usability[END_REF]. This makes users favor enjoyment, stimulation and realism over usability and performances [START_REF] Zuckerman | To tui or not to tui: Evaluating performance and preference in tangible vs. graphical user interfaces[END_REF]. Motivated by these results, my major interest for this research lies in exploring the potential of Tangible User Interfaces with a high level of physicality.

Inspired by visions presented by Mark Weiser, Bret Victor, or Hiroshi Ishii, I aim at using physicality to create accessible and inviting Tangible User Interfaces to go beyond existing interfaces that can be seen as hybrids. To this end, I intend to base interaction on hand manipulations of interactive physical objects, to promote more physical ways to interact with computers.

Supporting more versatility in TUIs

Researchers advocate specificity [START_REF] George W Fitzmaurice | Graspable user interfaces[END_REF] and realism [Jacob et al., 2008] to facilitate the use of TUIs. This approach is beneficial when it comes to a dedicated application.

But focusing on specific and iconic objects closes the doors to potential opportunities that versatility could bring. Thus, versatility should be given more attention during future development of TUIs.

Making TUIs design more easily available

It can be highly beneficial to make technological advances for TUIs publicly available. It could enable the research community to move faster, by benefiting from otthers' work. Designs can be improved, extended, or even used for new purposes. Allowing others to benefit from my work is one of the major goals in order to make replication and further development possible.

approach

This section contains the approach I adopted to address the goals introduced previously.

Figure 1.5: On the left: a LEGO brick. On the right: A polar bear sculpture [START_REF] Kenney | Art with lego bricks: Polar bear[END_REF] made of 95,000 LEGO Bricks by the artist Sean Kenney. 1.3.1 Supporting complexity using collections of generic objects I gain inspiration from a well known method humans have used for millenia to build complex structures: construction. From The Egyptian pyramids of the Giza Necropolis to the Roman Coliseum, humans have created great things by assembling together smaller elements that were easier to handle.

LEGO construction kits are an example for collections of building elements: bricks. Bricks have generic shapes and colors, and do not carry a specific meaning. Until 1978, LEGO bricks were limited to a single shape, with two possible colors: white and red (see Figure 1.5 on the left), and while they are now available in a variety of shapes and colors, the basic principle remains the same: the bricks are interlocking, and can be bound together mechanically. Rigid assemblies of bricks are the result. The design of the brick is generic, while offering a simple affordance: interlocking. However, once combined, they allow to create very complex and intricate structures (see Figure 1.5 on the right), offering a large potential for diverse designs and new affordances to be created. As this example shows, the nature of construction kits relies on simple and generic objects to create complex assemblies.

Genericness allows for two different perspectives on objects. An object can be an anonymous unit intended to be part of a larger entity, for instance each LEGO brick forming the polar bear in Figure 1.5. Objects can also have an identity, a meaning (for example a time period in a calendar, see Figure 1.6). They can be associated be with a digital information or computational functions (i.e., tokens [START_REF] Ullmer | Token+ constraint systems for tangible interaction with digital information[END_REF]). This duality makes it possible to support different usage contexts with a single platform, making them more versatile. Inspired by this constructive approach, I decided to focus on collections of generic objects. As demonstrated with the previous example, such collections allow to create complex and specific assemblies of diverse nature, thus providing increased versatility compared to traditional TUIs.

Making objects interactive

Interaction requires systems capable of supporting input and output. Considering output, the most common and versatile output channel used in any interactive device is a graphical display. Graphical screens represent the most commonly used output channel to display digital data. They are made of a regular and static 2D grid of pixels and can display any kind of visual representation. Extending the concept of screens, researchers in Robotics created physical displays [START_REF] Alonso-Mora | Multi-robot system for artistic pattern formation[END_REF] using collections of color changing robots, as enhanced pixels with locomotion capabilities. Physical displays are, just like a traditional screen, a spatial arrangement of color changing physical points (see Figure 1.7). Yet, this arrangement is dynamic and no longer regular. Systems such as Phidgets [START_REF] Greenberg | Phidgets: easy development of physical interfaces through physical widgets[END_REF] or OP [START_REF] Coutrix | Op: a novel programming model for integrated design and prototyping of mixed objects[END_REF]] have aimed at making TUIs more accessible and flexible. They provide toolkits allowing everydays objects to be digitally augmented for prototyping and designing TUIs.

The system has to be able to perceive manipulations in order to provide input. While several approaches exist to detect user actions in the context of TUIs (detailed later in Chapter 4), I choose to provide each object with sensing capabilities in order to perceive manipulations To summarize, my approach consists in using collections of interactive and generic objects to creating dynamic and composite TUIs. Dynamic and composite TUIs support increased physicality as well as versatility, as each object of the collection provide both input and output capabilities.

Open Technology

Drawing from Buxton's and Gaines' reflections, I see two obstacles limiting the rapid development of TUIs: lack of maturity and unavailability. Drawing on Buxton's history of the mouse, TUIs do not seem to be ready for "Prime Time" yet. The ecosystem still has to evolve, and Tangible User Interfaces still need improvement to better fit. In order to shorten this refinement phase, it is necessary to make technological advances for TUIs as available as possible. To this end, all the technology presented in this thesis is completely openhardware and open-source, and made available on the web use open-source licenses.

thesis statement

This thesis proposes new directions and opens new doors for research towards more physical and versatile Tangible User Interfaces. Research on tangible interaction has emerged in the light of Ubiquitous Computing and Augmented Reality and has since gained a lot of attention. Instead of pushing users into a virtual world, TUIs try to enhance and augment the physical world with computation to better leverage the human capabilities to interact with their natural environment. Researchers have envisioned and investigated a number of technologies, seeking seamless transitions between "the real" and "the digital".

Yet, I see two main limitations in the current direction of research on TUIs. First of all, researchers in the field of TUIs have been advocating for more physicality in user interfaces, but existing TUIs have mostly relied on graphical content on screens or using projection. Even though available technology does not allow yet to realize purely physical TUIs, they remain for the most part, graphical representations coupled with passive physical handles and proxies. TUIs only provide a limited one-way mapping between physical controls and the digital content. Secondly, one claim of TUIs is to take advantage of space multiplexing, by emphasizing objects' specificity and leveraging physical affordances to efficiently serve a specific context. Numerous technological solutions for TUI have been developed, but have always focused on specific contexts and particular applications. While I believe this specificity is one of the strength of TUIs, it is also its main weakness at the cost of versatility.

Technologies are nowadays rapidly evolving in numerous fields such as connected objects, Internet of Things, wearable and mobile computing. I believe we now have the opportunity to push further towards more versatile dynamic TUIs, helping us to think interaction beyond screens. Influenced by Robotics and the vision of programmable matter, I focus in this thesis on interacting with collections of interactive physical objects. I first motivate this direction by observing how one manipulates collections of objects, to assert the benefits on physicality over touch interaction as well as confirm the design choices for technology. I then start investigating composite TUIs by embedding sensing capabilities in generic objects. I then augment these objects with actuation and output capacities. I introduce Swarm User Interfaces, TUIs composed of collections of autonomous robots that handle both display and interaction. I implement Swarm User Interface with Zooids, the first tabletop versatile and composite TUI allowing to interact only through physical objects. Using Zooids, I seek to offer new opportunities to further investigate more versatile and physical TUIs. I finally present Information Visualization as a use-case for Swarm User Interfaces to demonstrate improved versatility.

thesis overview

Chapter 2 -Background. This chapter provides an overview of the state of the art of Tangible User Interfaces. In a first phase, it describes the emergence of TUIs and their evolution, as well as the efforts to understand the underlying benefits of TUIs. In a second phase, this chapter presents the main application domains for TUIs. It finally reviews the existing approaches to provide actuation to TUIs, and emphasized the need for further research to support more versatility and physicality in TUIs, as motivated in Chapter 1.

Chapter 3 -Studying manipulations of collections of small generic objects.

This chapter identifies directions motivating TUIs and their physicality over multitouch interfaces for 2D manipulations of collections of small generic objects on tabletop surfaces. This chapter presents a novel method using physical proxies to conduct user experiments, investigating fundamental differences between TUIs and multitouch interfaces. This method uses physical chips to emulate the flat, non-graspable objects that make up touch interfaces, in a way that supports direct comparison with TUIs. Results support the benefits of thickness and physicality to manipulate collections of generic objects.

Chapter 4 -Small generic objects as input devices.

This chapter presents an approach to augment collections of generic objects with sensing capabilities to support user input. It introduces SmartTokens, small-sized generic objects that can sense multiple types of motion and grips, and communicating wirelessly input events. This chapter describes the design and implementation of SmartTokens, and how they extend the design space of TUIs by contributing an open platform for embedding sensing within small objects. To illustrate SmartTokens, this chapter finally presents a novel TUI for event notification and personal task management.

Chapter 5 -Introducing Swarm User Interfaces: versatile composite Tangible User Interfaces.

This chapter investigates versatile composite TUIs by introducing Swarm

User Interfaces, a subclass of TUIs consisting of many autonomous robots, capable of handling display and interaction. Zooids are presented as a TUI composed of collections of small and fully interactive robots to instantiate Swarm User Interfaces. Zooids support both input and output and allow to create purely physical TUIs. The potential and versatility of Zooids is demonstrated through a set of application scenarios, and then describes their implementation and design as an open-source open-hardware platform. This chapter finally clarifies design considerations for Swarm User Interfaces.

Chapter 6 -Designing dynamic physical visualizations using Swarm User Interfaces.

This chapter focuses on a more specific use-case for Swarm User Interfaces: Information Visualization. It first presents and defines composite physicalizations, and then surveys existing composite physicalizations. It introduces two dimensions to help classify composite physicalizations: manipulability and actuation. This chapter lastly presents adaptations of typical visual representations using Zooids as fully interactive composite physicalizations.

Chapter 7 -Summary and Perspectives. This final chapter summarizes the results of this thesis, and discusses the outcomes with regards to the established goals. It then provides perspectives and openings for future research.

This chapter provides an overview of the current situation of Tangible User Interfaces. It first reviews the emergence of TUIs and their evolution, followed by the work carried out to better understand the qualities and assets TUIs provide. This chapter then shows the main application fields of TUIs, illustrating the unique advantages of TUIs. Driven by the motivations detailed in the Introduction (Chapter 1), this chapter finally surveys actuation in TUIs, and identifies the unexplored areas, highlighting the need for further research to improve versatility and physicality in TUIs.

This chapter takes inspiration from [START_REF] Shaer | Tangible user interfaces: past, present, and future directions[END_REF]'s monograph Tangible User Interfaces: Past, Present, and Future Directions, which provides an extensive overview on the evolution of TUIs. This subsection presents Tangible User Interfaces from a historical perspective, by detailing the first implementations of TUIs as well as early efforts to conceptualize TUIs.

Early Prototypes and Founding Visions

2.1.1.1 First attempts to create TUIs First prototypes and platforms of TUIs came into view before researchers had started theorizing about User Interfaces involving physical objects. In 1976, Perlman invented the Slot Machine [START_REF] Perlman | Using computer technology to provide a creative learning environment for preschool children[END_REF] to help children discover programming languages. Other prototypes emerged in the early 1980s and in particular two applied architectural CAD systems. Both [START_REF] Aish | 3d input for caad systems[END_REF][START_REF] Hamilton Frazer | Intelligent physical three-dimensional modelling system[END_REF] tried to simplify the use of CAD systems, which were laborious and puzzling to handle. Both decided to adopt physical blocks to create assemblies, and used computers to recognize the topology and create digital 3D models accordingly (see Figure 2.1). Users could create, iterate and refine digital 3D structures without having to manipulate numerical data. From that, the system could finally generate plans and drawings.

In 1992, Durrell Bishop proposed a new concept for a phone answering machine [START_REF] Polynor | The hand that rocks the cradle[END_REF]. It used physical marbles to represent voice messages, by releasing a marble each time a new message was recorded. The marbles fell down a slide and were stored in a gutter, preserving the chronological order. One could then listen to a message by placing the marble in a cavity on the machine, while putting the marble on the phone initiated a call to recorded number. Marbles put aside indicated calls for other people.

Marbles could also be left aside in a dedicated container if aimed at someone else (see Figure 2.2). With Bishop's answering machine, marbles were physical manifestations of voice messages and handling intangible phone messages could be handled by manipulating physical objects, thus making them tangible for the user. There were diverse reasons that motivated the transition from Graphical User Interfaces (GUIs) towards what would later become Graspable and Tangible User Interfaces were diverse, but one appeared to be recurrent among researchers: User Interfaces were lacking physicality and thus the connection to the real world.

2.1.1.2 Evolving from prototypes to theories [START_REF] Weiser | The computer for the 21st century[END_REF] presented his vision for the future of computing and interaction in "The Computer for the 21st Century". He introduced the concept of Ubiquitous Computing, where computing would be embedded and interconnected in our everyday environment, making interaction with computers and the digital world seamless. Weiser's vision of Ubiquitous Computing was not focusing on tangible interaction, but rather on a new approach of physicality in computing, and a new way to envisage technology. His vision inspired numerous researchers to follow the direction of new tangible interaction.

Similarly, a few years later, [START_REF] Wellner | Computer-augmented environments: back to the real world[END_REF] disputed that desktop computers and virtual reality force humans to enter a virtual world that is unnatural to them. Thus, instead of pushing users into a digital environment, unfit for their natural capabilities, their natural environment should host computation and be augmented with computation.

Fitzmaurice et al. [1995] presented Graspable User Interfaces. This was the first interaction paradigm where physical objects (i.e., Graspables) were used to manipulate digital counterparts synchronously (see Figure 2.3), intending to improve directness and manipulability. Graspables were associated with dedicated functions and allowing users to use both hands at the same time to interact in a richer way.

A few years later, [START_REF] Ishii | Tangible bits: towards seamless interfaces between people, bits and atoms[END_REF] introduced Tangible Bits. Their vision consisted in using the physical world as an interface, by "coupling the bits with everyday physical objects and architectural surfaces". The physical world allows to manipulate digital information through tangible handles, leveraging human manipulation capabilities. Concurrently, the digital world can also act on the physical world to communicate information using ambient media such as light or sound, leveraging human awareness of the surrounding environment. Ishii and Ullmer called this an ambient display.

Tangible User Interfaces were presented as an evolution of Graspable User Interfaces. Graspable User Interfaces focused only on input, using handle manipulations to interact with the digital world. Building on this, Tangible User Interfaces also included tangible representation of digital information.

An early prototype incarnating Tangible Bits was the metaDESK [START_REF] Ullmer | The metadesk: models and prototypes for tangible user interfaces[END_REF], physically instantiating well-known GUI, mapping for example a window into a physical lens, a widget into an instrument, or a menu into a tray. Demonstrating metaDESK, the Tangible Geospace allowed interacting with a physical map of the MIT campus using physical miniatures of the buildings (see Figure 1.2). As explained in Section 1.1, miniatures were handles to digital buildings on the map. Manipulating the miniatures allowed to pan and rotate the map. Bringing miniatures closer together or moving them apart allowed to zoom in and out. An arm mounted display was used as a lens and could provide 3D representation of buildings.

Theories and paradigms

This section covers the existing theoretical work in TUIs, from frameworks to taxonomies, and finally summarize benefits of TUIs.

Defining TUIs

Building on Tangible Bits, [START_REF] Ullmer | Emerging frameworks for tangible user interfaces[END_REF] presented a new framework in which they defined TUIs:

Tangible interfaces give physical form to digital information, employing physical artifacts both as representations and controls for computation media.

They adapted the MVC (model view controller) interaction model used for GUIs, and introduced the Model Control-Representation (physical and digital), or MCRpd model. While the MVC model separated the view (i.e., output channel or graphical representation) component from the control component (i.e., input channel), the MCRpd model accentuated the combination of the physical representation and the control (see Figure 2.4), merging input and output channels.

Historically, physical objects have been used to represent intangible quantities and pieces of information for ages. They were used as tools to count, measure or perform abstract operations grouping. For instance, in Mesopotamia (between 2,700 and 2,300 BC) the abacus was used as calculation tool, using beads on rods. It evolved across Eastern Europe, Russia, China and Africa, before the emergence of numerical systems. The Chinese variation, the Suanpan (see Figure 2.5), allowed to perform basic mathematical operations efficiently such as additions and subtractions, multiplications and divisions, but also more complex such as square and cube root. Even though in decline, the Suanpan is still used in China.

Inspired by the abacus, board games and other systems associating physical objects with intangible information, researchers introduced the concept of tokens in computing and interfaces, as physical representations of digital data. Physical tokens can be used in very different ways with computers; [START_REF] Holmquist | Token-based access to digital information[END_REF] identified three classes of physical objects used within the frame of TUIs to represent digital information and functions.

• as Containers, generic objects providing versatility as they can accommodate any type of digital information,

• as Tools, objects representing computational function,

• as Tokens, objects that physically look like the information they represent. [START_REF] Ullmer | Token+ constraint systems for tangible interaction with digital information[END_REF] approached tokens differently. They defined tokens as "discrete physical objects which represent digital information". These can take any form as long as they are associated with digital information, and work coupled with Constraints, "confining regions that are mapped to digital operations". They usually take the form mechanical elements guiding and restraining the movement of tokens such as slides or rails, to be able to "invoke and control a variety of computational interpretations". [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF] adopted a broader approach they called Tangible Interaction. By considering social interaction, Tangible Interaction covers research on whole-body interaction, interactive spaces and gestural interaction. It is articulated around four topics:

• Tangible Manipulation concerns physical representations and their characteristics and manipulability,

• Spatial Interaction deals with properties of the space and the interaction in it,

• Embodied Facilitation focuses on the arrangement of objects and spaces, and how it affects interaction,

• Expressive Representation targets the expressiveness of physical representations used for Tangible Interaction.

Classifying TUIs

Comparing, characterizing and classifying are difficult tasks, as TUIs come in many different forms for many different domains. Researchers formulated several taxonomies to help classifying systems. [START_REF] Ullmer | Token+ constraint systems for tangible interaction with digital information[END_REF] isolated three different types of TUIs:

• Interactive surfaces are flat surfaces on top of which physical objects are placed and their position is processed by the system,

• Constructive assemblies are inspired by construction kits, and consist of collections of objects that can be put together to create structures which are then processed by the system,

• Token+Constraint systems use physical affordances of objects to create constraints and restrain mechanically possible movements of tokens.

With his taxonomy, [START_REF] Kenneth P Fishkin | A taxonomy for and analysis of tangible interfaces[END_REF] presented a new way to analyze and compare easily the different directions in TUIs. Using the dimensions embodiment and metaphor, he created a 2D space where TUIs would sit according to their capabilities. A system is considered more tangible as it sits further away from the origin (see Figure 2.6).

The first dimension, embodiment, relates to the quality of embedding computation inside physical objects and devices, consequently making physical manipulations of the object itself an integral part of the interaction. Embodiment ranges from distant (i.e., the output is physically far away from the input, e.g., a remote control) to full (i.e., output and input form one single object, e.g., sculpting clay). The second dimension, metaphor, evaluates the similarities of the system with the physical world. Fishkin distinguishes two kinds of metaphors: metaphor of noun and metaphor of verb. The metaphor of noun describes the realism of physical properties (shape, appearance, etc.) of the input object (e.g., the miniatures representing buildings in Figure 2.3). The metaphor of verb relates to the realism of result of actions performed on the input object (e.g., a shake gesture to clear a screen). The metaphor level combines both kinds and varies from none (i.e., no correlation exists between the action or the nature of the input and the effect of an action) to full (i.e., the virtual system is the physical system, e.g., using pens on a tablet to draw). [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF] included other domains such as Design or Arts and Architecture to define Tangible Interaction. They raised three standpoints: data-centered view (i.e., traditional TUIs, using physical objects to represent and interact with digital data), expressive-movement-centered view (emphasizing bodily interaction with objects) and space-centered view (based on spatial interaction associating spaces and objects with digital media).

Understanding the benefits of TUIs

TUIs are often compared to other user interfaces, in particular multitouch interfaces as both share similar assets for interaction. Among others, TUIs as well as multitouch interfaces afford direct manipulation of objects, and allow to use both hands simultaneously. Multitouch interfaces show superiorities over TUIs especially when it comes, as discussed earlier, to versatility. Current TUIs also require more development and engineering to create new applications. It is thus important to understand the unique and fundamental qualities of TUIs, in order to benefit from them in an optimal way. Space-Multiplexing. TUIs do not rely on a single input device to interact. In traditional environment such as GUIs on desktop computer or touch devices, users are limited to use one function after another: GUIs are time multiplexed [START_REF] George W Fitzmaurice | Graspable user interfaces[END_REF]. One the contrary, TUIs rely on multiple devices for input and output, interaction is then distributed over space allowing multiple objects to be used simultaneously: TUIs are space-multiplexed. Therefore, space multiplexing provided by TUIs naturally supports multitasking.

Affordances. TUIs have a unique advantage provided by their physicality: affordances [START_REF] Donald A Norman | The design of everyday things: Revised and expanded edition[END_REF]. By carefully designing tangible objects, characteristics such as shape, color, weight or material can be used to convey meaning, function of objects or ways to interact with it. Moreover, this mapping is persistent as anchored in the physical design of the object, thus avoid a learning process. Objects no longer need to be generic, but can on the contrary be specific and iconic as they are dedicated to one function only. The specificity of objects improves the association between actions and effects, making TUIs more discoverable.

Experience. Interacting with physical representations exploits experience from the physical world including "Naïve Physics", "Body Awareness and Skills", "Environment Awareness and Skills" and "Social Awareness and Skills". Jacob et al. [2008] argue that interfaces should be designed to be as close as possible to reality, to take advantage of knowledge users have of the physical world. This close connection to reality allows users to discover and understand interfaces more easily, making them more accessible. For Dourish, the situatedness is one of the main strengths of TUIs [START_REF] Dourish | Where the action is: the foundations of embodied interaction[END_REF]: tangible objects remain in the physical world and the interface they constitute are part of our environment just like any other object.

Collaboration. When it comes to collaboration, it seems legitimate to hypothesize that a physical environment can provide better conditions than a virtual one. [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF] identified several factors upholding face to face collaboration. For instance, the knowledge of the physical world and the interactions performed everyday with it facilitate engagement and contribution. Manual interaction with objects is easily observed and understood, advantaging group coordination and awareness.

Cognitive help. Another positive aspect of TUIs is the cognitive help they provide. [START_REF] Fernaeus | Finding design qualities in a tangible programming space[END_REF] explain that in some concrete contexts such as engineering or architecture, practitioners often have recourse to physical artifacts to support complex reflection and reasoning. TUIs help leveraging the connection between body and cognition, using physical manipulations, bodily actions and tangible representations. By leaving room for "offline" interactions, i.e., "interactions directed at the social/physical setting instead of at the computer" [START_REF] Shaer | Tangible user interfaces: past, present, and future directions[END_REF], TUIs facilitate epistemic actions. Epistemic actions are "actions performed to uncover information that is hidden or hard to compute mentally" [START_REF] Kirsh | On distinguishing epistemic from pragmatic action[END_REF], while pragmatic actions have functional outcomes. Epistemic actions such as pointing, moving or rotating an object can help decrease the cognitive load, and allow to explore possibilities, remember anterior actions or support memory. The external representations that physical objects can form can also help reduce the cognitive load. According to [START_REF] Zhang | Representations in distributed cognitive tasks[END_REF], external representations are essential for many cognitive tasks as they guide and constrain the cognitive behavior.

All these characterisitcs make TUIs a unique modality for Human-Computer Interaction.

tuis for various application domains

Finding compelling application domains for TUIs is crucial, as good usecases allow to leverage advantages TUIs have to offer. Several fields emerged from research as well as from practitioners. This section develops four typical contexts in which TUIs thrive.

Learning with TUIs

TUIs have been largely used as learning tools and support in educational contexts. Building on work from Jean Piaget, Maria Montessori or Friedrich Fröbel, researchers in education developed physical systems to leverage the benefits of TUIs and improve educational practices.

In the late 1840s, Friedrich Fröbel created physical objects to help illiterate children learn and understand mathematics. His educational materials, known as Fröbel Gifts, were kits of objects of different forms including wooden blocks and soft balls [START_REF] Froebel | The education of man[END_REF]. Those objects could be used from young age to discover concepts such as gravity, speed and time, or movement through simple manipulations. Exercises were explained in a manual to explore mathematics, geometry or modeling [Huron, 2015, 45].

More recently, the Lifelong Kindergarten Group at MIT MediaLab has focused on creating and studying creative ways to support education. Among others, FlowBlocks and SystemBlocks [START_REF] Zuckerman | Extending tangible interfaces for education: digital montessori-inspired manipulatives[END_REF] are TUIs using construction kits of digitally augmented objects based on Maria Montessori's work [START_REF] Montessori | The Montessori Method: Scientific Pedagogy as Applied to Child Education in "the Children's Houses" with Additions and Revisions by the Autor[END_REF] Other examples include construction kits to experiment with graph theory [START_REF] Schweikardt | A tangible construction kit for exploring graph theory[END_REF] and mathematical notions such as volume and surface area of 3D objects [START_REF] Girouard | Smart blocks: a tangible mathematical manipulative[END_REF].

Programming with TUIs

The examples above demonstrated how TUIs can be a good medium for education through hands-on activities, especially for abstract concepts. Programming, another abstract activity, allows, to control computational systems. For instance, augmented toys such as simple robots allow children to discover and understand the basic constructs of programming, in a playful and interactive way, and abstract concepts such as the functional principles of computers.

As early as 1976, Radia Perlman invented the TORTIS system (Toddler's Own Recursive Turtle Interpreter System) to help children discover programming languages [START_REF] Perlman | Using computer technology to provide a creative learning environment for preschool children[END_REF]. Building on work of [START_REF] Papert | Mindstorms: Children, computers, and powerful ideas[END_REF] on education, children could control the behavior of a turtle robot using the educational language LOGO. The TORTIS Slot Machine (see Figure 2.8 (left)) used plastic cards to represent constructs of the programming language. Using several types of cards, children could create sequences of instructions to control the turtle robot. [START_REF] Suzuki | Algoblock: a tangible programming language, a tool for collaborative learning[END_REF] pursued this direction further with the "AlgoBlocks" system. Similar to TORTIS, AlgoBlocks used a set of physical blocks to represent elements a LOGO-like programming language. Blocks were assembled to create a program operating a submarine in a video game displayed on a screen. Besides the systems described above, Topobo [START_REF] Hayes Solos Raffle | Topobo: a constructive assembly system with kinetic memory[END_REF] was a construction kit to create animated characters. It illustrated the programming by demonstration [START_REF] Cypher | Watch what I do: programming by demonstration[END_REF]. Another example was Tern [Horn et al., 2008], a tangible block programming language for children.

Planning and Problem solving

Section 2.1.2.3 showed that TUIs provide cognitive help by supporting epistemic actions, making them suitable for complex tasks such as planning or solving.

An influential system for this purpose was Urp [START_REF] Underkoffler | Urp: a luminous-tangible workbench for urban planning and design[END_REF], which aimed at assisting urban planning. Users could manipulate buildings while environmental conditions (e.g., wind flows or cast shadows) were simulated in real time and overlaid on the surface. Urp also provided physical tools to manipulate intangible phenomena such as wind or time. A similar system, Pico [START_REF] Patten | Mechanical constraints as computational constraints in tabletop tangible interfaces[END_REF], also enabled urban planning, in particular the task of optimizing the location of mobile phone towers. Pico used actuation and could move the objects on its surface (see Figure 2.9). The system could automatically optimize the placement, but was also supporting the user's intervention with constraints the system was not aware of.

Based on the same platform as Urp, Illuminating Light allowed to manipulate simulated light rays, using virtual prisms and mirrors to help understand principles of optics while avoiding manipulating fragile and expensive equipment. Illuminating Clay [START_REF] Piper | Illuminating clay: a 3-d tangible interface for landscape analysis[END_REF] was exploring landscape modeling. Users could form landscapes by directly sculpting raw clay. The topography of the model was monitored and analyses were projected on top of the model. Other systems were presented for applications such as collaborative IP network design [START_REF] Kobayashi | A tangible interface for ip network simulation[END_REF] or warehouse logistics [START_REF] Zufferey | Tinkersheets: using paper forms to control and visualize tangible simulations[END_REF] Figure 2.9: Pico [START_REF] Patten | Mechanical constraints as computational constraints in tabletop tangible interfaces[END_REF] is an interactive tabletop surface capable of tracking and moving small objects to solve spatial layout problems, here laying out cellular phone towers.

Figure 2.10: Tangible Query Interfaces [START_REF] Ullmer | Tangible query interfaces: Physically constrained tokens for manipulating database queries[END_REF] allow to interact with visualizations using tangible controllers.

Physical and Tangible Information Visualization

Physical representations have been used by humans to convey information for millennia. For example, the Lebombo bone (35,000 BC to 30,000 BC) and the Ishango Bone (around 22,000 BC) are the oldest known physical representations [Jansen, 2014b, p10]. From Mesopotamian clay tokens (8,000 BC to 3,500 BC) to board games in Ancient Egypt (3,100 BC) [Huron, 2015, p43], physical visualizations have evolved in many different ways. Probably the most banal physical visualizations used across the ages are coins, used to represent currencies to facilitate exchanges.

Tangible Query Interfaces [START_REF] Ullmer | Tangible query interfaces: Physically constrained tokens for manipulating database queries[END_REF] demonstrated the use of TUIs to interact with relational databases. Using different interchangeable physical wheels and bars, the user could perform real-time queries on the database, with the results being then displayed on a screen. A range of different wheels and bars allowed to perform queries on different dimensions of the database. Each wheel and bar was a token representing a dimension of the database and the position of the cursor the value of the query.

Using physical objects to visualize information is now a growing interest in the Information Visualization (InfoVis) research community, and is being explored in various ways [Jansen et al., 2015a]. Recent work from [START_REF] Jansen | Physical and tangible information visualization[END_REF] investigated how to use physical objects can be beneficial in the context of InfoVis. She compared physical representations of 3D bar charts with graphical counterparts (see Figure 2.11), to better understand how physicality can help perform Information Visualization tasks [START_REF] Jansen | Evaluating the efficiency of physical visualizations[END_REF]. She coined the term of data physicalization as a physical artifact whose geometry or material properties encode data [Jansen et al., 2015a]. This emerging area of InfoVis intends to support exploration and assimilation of data using physicalizations augmented with computation.

Making music with TUIs

Music is an attractive and compelling application domain for TUIs. Bringing together the incredible potential of computers for musical composition with the expressive power of hand manipulations, several musical tangible platforms arose and started to grow interest from the music community.

Combined with multitouch interfaces, [START_REF] Jordà | On stage: the reactable and other musical tangibles go real[END_REF] explained that TUIs offer interesting characteristics for music performance. Much like traditional musical instruments, TUIs allow for continuous control and support realtime interaction, while providing multi-dimensional control. TUIs also offer opportunities for collaborative performance, supporting complex and explorative interaction. The most famous example for tangible music interface may be the reacTable [START_REF] Jordà | The reactable: exploring the synergy between live music performance and tabletop tangible interfaces[END_REF]. It consists of multiple cubic tokens sitting on a multitouch surface (see Figure 2.12). Each token has a dedicated function such as producing and filtering audio, adjusting parameters like volume, pitch or effects. By simply combining and controlling tokens, re-acTable is inviting and attractive, and allows multiple users to create music in a playful way.

Other examples of TUIs for music applications include AudioPad [START_REF] Patten | Audiopad: a tag-based interface for musical performance[END_REF] or BlockJam [START_REF] Newton-Dunn | Block jam: a tangible interface for interactive music[END_REF].

Figure 2.12: The reacTable [START_REF] Jordà | The reactable: exploring the synergy between live music performance and tabletop tangible interfaces[END_REF] is a tangible music interface using tokens to create electronic music.

bringing actuation to tuis

Despite the numerous advantages of TUIs illustrated in the previous examples, most existing TUIs suffer from the one-way mapping between the digital and the physical. As objects are physical handles in the digital world, manipulating the physical objects impacts digital data but changes in the digital world cannot be passed back to the physical part of the interface. This leads to inconsistencies between the controls and the content [START_REF] Ishii | Radical atoms: Beyond tangible bits, toward transformable materials[END_REF]. The digital has to be able to act on the physical to maintain the coherence between the two worlds. To bridge this gap, researchers have explored ways to associate computation and actuation, allowing to control the motion of physical objects. [START_REF] Poupyrev | Actuation and tangible user interfaces: the vaucanson duck, robots, and shape displays[END_REF] defined Actuated interfaces as "Interfaces in which physical components move in a way that can be detected by the user". Bringing actuation to TUIs has always been a motivation and a part of the founding visions of TUIs. However, implementing actuation stretches across domains such as Robotics, Electrical and Mechanical Engineering or Computer Science. Earlier sections have shown the broad spectrum of TUIs, and how each case requires a different approach to fit constraints and requirements. Embedded displays, multitouch sensing, multiple objects or mobility are as some of the factors constraining the actuation methods. This section focuses on the two dominant kinds of actuated TUIs: shapechange and self-rearrangement. Extending on the presented examples, this section finally presents the vision for interaction that is Programmable Matter.

Shape-Changing interfaces

Rasmussen et al. [2012] presented a taxonomy to systematically classify shape-changing interfaces. They characterize them as user interfaces involving "physical change of shape as input or output", and identified eight different categories based on the transformations the interfaces provide. Among them, two major trends manifest: modifying continuous surfaces and flexible materials, or approximating shapes with arrays of rigid elements.

Actuating continuous surfaces

Traditionally, TUIs have been using rigid objects to interact with digital content. But recent progresses in material allow to create deformable and malleable objects. In particular, flexible displays and sensing technologies have led to the emergence Organic User Interfaces. Organic User Interfaces (OUI) were defined as computer interfaces that use a non-planar display as a primary means of output, as well as input. [START_REF] Holman | Organic user interfaces: designing computers in any way, shape, or form[END_REF] OUIs rely on deforming objects with co-located input and output channels to convey information through manipulation or actuation. As TUIs, OUIs advocate for specificity so the shape can help estimate the function of the object, Figure 2.13: Using particle jamming [START_REF] Follmer | Jamming user interfaces: programmable particle stiffness and sensing for malleable and shape-changing devices[END_REF], Tunable Clay can modify the stiffness of its material support 3D modeling.

while taking advantage of actuation and flexibility to adapt their shape to the context. Following are some examples of OUIs. Surflex, a system by [START_REF] Coelho | Surflex: a programmable surface for the design of tangible interfaces[END_REF] provided control over the deformation of surfaces by embedding actuators made of shape memory alloys in soft materials such as foam. Morphees [START_REF] Roudaut | Morphees: toward high shape resolution in self-actuated flexible mobile devices[END_REF] were selfactuated flexible mobile devices capable of adjusting their physical shape to their current use, to provide optimal affordances. Morphees were actuated following techniques utilizing shape memory alloys and dielectric electroactive polymers. The presented concept of shape resolution, alongside with a conceptual framework, allowed to characterize the resolution of shape changing devices according to multiple dimensions. [START_REF] Follmer | Jamming user interfaces: programmable particle stiffness and sensing for malleable and shape-changing devices[END_REF] used particle jamming to create a flexible shapechanging user interface, offering a digital control of the stiffness of the material could be computationally controlled. It allowed to control the haptic feedback and the deformation of flexible and malleable object. Shape sensing capabilities were embedded to create interactive deformable devices, such as 3D modeling surface simulating "programmable" clay (see Figure 2.13). PneUI [START_REF] Yao | Pneui: Pneumatically actuated soft composite materials for shape changing interfaces[END_REF] utilized pneumatically-actuated soft materials to build shape changing interfaces. The interface could shift between multiple pre-defined structures using pneumatic actuation, while providing input sensing. This system enabled the exploration of new types of interactive objects such a shape-changing tablet cases or height changing tangibles icons.

MudPad [START_REF] Jansen | Mudpad: tactile feedback and haptic texture overlay for touch surfaces[END_REF] allowed to create dynamic haptic textures on a multitouch screen using an array of electromagnetic fields and a magnetorheological fluid. Modulating the electromagnetic field, in particular its shape and frequency, allowed to create a wide range of dynamic textures.

Emergeables [START_REF] Robinson | Emergeables: Deformable displays for continuous eyes-free mobile interaction[END_REF] consists of mobile surfaces that can change shape to afford actuated tangible controls. Emergeables combine the advantages of both modalities, i.e., the flexibility of touch screens and affordances of tangible controllers. However, Emergeables remain so far at a conceptual stage as no contemporary technology allows to implement such combination.

Approximating shapes with discrete elements

Shape displays adopt a different approach to create dynamic physical topographies. Graphical displays discretize images into aligned pixels, shape displays approximate shapes by controlling the height of pins arranged in a grid. In order to provide this functionality, each pin is able to move up and down.

An influential shape display was Lumen [START_REF] Poupyrev | Lumen: interactive visual and shape display for calm computing[END_REF], a low fidelity shape display, but capable of communicating 2D graphical images with dynamic physical shapes. These shapes brought physicality to graphical content as they could be touched as well as observed from different angles. Lumen consisted of a 13× 13 array of illuminated rods, where each rod was actuated using shape memory alloys. The created motion was smooth and organic (see Figure 2.14 (left)). Despite its low fidelity, Lumen could detect user actions thanks to embedded sensing and could convey rich content combining dynamic shapes with graphical content. [START_REF] Leithinger | Relief: a scalable actuated shape display[END_REF] later introduced Relief, a shape display using motorized rods. Relief could control the height of each pin to create more complex shapes. Relief also presented new interaction techniques leveraging gestures and direct manipulation, taking advantage of the technology. inFORM by Follmer and Leithinger followed, and pushed shape displays to a new level. Using a similar technological approach as Relief, inFORM used 900 motorized pins to provide a higher resolution [START_REF] Follmer | inform: dynamic physical affordances and constraints through shape and object actuation[END_REF]. Coupled with projection, inFORM allowed to display rich and elaborate topologies, but also supported user sensing (see Figure 2.14 (right)). Building on Norman [2013], inFORM enabled to create dynamic physical affordances and constraints for direct physical manipulation. Shape displays have also been explored outside of the research community. The ART+COM design studio created several kinetic sculptures made of arrays of objects suspended in the air by a cable. The nature and shape of objects changed across sculptures, but all relied on the same idea: actuating suspended objects vertically. Objects were precisely choreographed to represent surfaces such as cars, birds or planes (see Figure 2

.15).

These examples show that deformation and shape-change allow to create rich and diverse interface, yet they require a heavy equipment and only allow limited control over physical geometries and topologies. None of the previous systems supports physically detached objects.

Self-Rearranging Tangibles

As presented in Section 2.1.2.3, one important benefit of TUIs is to take advantage of space-multiplexing [START_REF] George W Fitzmaurice | Bricks: laying the foundations for graspable user interfaces[END_REF] using collections of objects and their location in space to represent digital information. This chapter has shown that numerous contemporary TUIs have followed this direction. To avoid inconsistencies between digital models and physical representations, tangibles need to be able to move from one location to another. Systems with these capabilities are called self-rearranging displays, defined by [START_REF] Poupyrev | Actuation and tangible user interfaces: the vaucanson duck, robots, and shape displays[END_REF] as: devices that consist of multiple parts that can dynamically re-arrange themselves in space.

Two strategies are possible to meet this requirement:

• External actuation uses mechanisms external to the objects to move them in space,

• Internal actuation embedds actuation capabilities in each object.

External actuation

External actuation implies that objects remain static and passive and do not require any embedded technology. Several technologies were implemented to actuate passive objects.

The first technique consists in creating multiple dynamic magnetic forces across a surface. Arrays of electromagnets create localized magnetic forces to attract or repulse objects. Controlling each magnetic force independently allows to move objects following 2D trajectories. The Actuated Workbench was the first to implement this technology [START_REF] Pangaro | The actuated workbench: computer-controlled actuation in tabletop tangible interfaces[END_REF]. Coupled with optical tracking and projection, the system could move multiple coin-sized plastic pucks, each featuring a permanent magnet. Pico [Patten and Ishii, Figure 2.14: Lumen [Poupyrev et al., 2004] is a low resolution shape display that can communicate 2D graphical images and augmented with dynamic physical shapes (left). inFORM [START_REF] Follmer | inform: dynamic physical affordances and constraints through shape and object actuation[END_REF] is a high resolution shape display and can create complex shapes and senses user actions (right). 2007], mentioned in section 2.2.3, was a later iteration of this technology. It offered larger interaction space and finer resolution, and introduced new interaction techniques leveraging the potential of mechanical constraints coupled with dynamic actuation. Madgets [Weiss et al., 2010a] provided actuation to independent physical magnetic widgets on the surface, using a similar electromagnets array and an interactive multitouch tabletop surface (see Figure 2.16). The system also supported usual multitouch interaction including gestures. The widgets were made of acrylic and were equipped with permanent magnets. Multiple nature, e.g., buttons, sliders or knobs, could be implemented.

A second technique relies on ultrasounds to actuate small objects. While electromagnets can create both attraction and repulsion forces, an ultrasound transducer can only produce air pressure waves, restricted to push in one direction. Thus, arrays of transducers have to be installed facing each other, to allow bidirectional forces. The "Ultra-Tangibles" system [START_REF] Marshall | Ultra-tangibles: creating movable tangible objects on interactive tables[END_REF] was able to actuate lightweight polystyrene objects in two dimensions on top of an interactive surface. 3D actuation in mid-air was even achieved using this technology [START_REF] Ochiai | Three-dimensional midair acoustic manipulation by ultrasonic phased arrays[END_REF][START_REF] Omirou | Levipath: Modular acoustic levitation for 3d path visualisations[END_REF], but did not support interaction with objects.

A third technique uses the principle of electrostatic induction to actuate lightweight objects. [START_REF] Yamamoto | Planar 3-dof paper sheet manipulation using electrostatic induction[END_REF] and [START_REF] Amano | Tangible interactions on a flat panel display using actuated paper sheets[END_REF] created TUIs where simple pieces of paper could be actuated on flat surfaces. Putting electrostatics to work, electric charges induced in pieces of paper resulted in creating repulsive electrostatic forces. Modulating these forces across the surfaces allowed to move and rotate tangibles.

The main advantage of external actuation resides in the simplicity of the actuated objects. As shown in the previous examples, objects can range from millimeter-scale to hand-size objects, consisting of various types of materials. However, scalability of these systems is often problematic, as they require heavy and complex infrastructure imposing limits on the size of the interaction area. Moreover, these technologies often impose limits regarding objects, as they have to be compatible with the actuation technology. For instance, electromagnetic actuation requires objects to be instrumented with permanent magnets to create the attraction force, while the placement accuracy is usually low, due to the limited resolution of the electromagnet array. Ultrasounds cannot apply strong forces and thus restrict the size and the weight of objects.

Internal Actuation

The alternative consists in augmenting objects themselves with locomotion capabilities. Self-actuated objects do not require special surfaces to function. A common path to provide locomotion capabilities to objects is to equip them with motorized wheels. With two or three wheels, objects are able to move in any direction on a surface. This allows to manipulate the digital world using independent and dynamic physical handles. Rosenfeld et al. introduced, in 2004, the Planar Manipulator Display. Using wheeled robots and projection, the Planar Manipulator Display allowed to manipulate directly pieces of furniture [START_REF] Rosenfeld | Physical objects as bidirectional user interface elements[END_REF] in an interior architecture scenario. The system could move and hold objects in position to respect sets of constraints applied to the layout, such as favoring light.

Tangible Bots created by Pedersen and Hornbaek [2011] was based on of off-the-shelf robots allowing interaction with a multitouch tabletop surface (see Figure 2.17 (left)), using direct manipulation as well as new interaction techniques for group manipulations and leveraging dynamic haptic feedback.

Patten Studio [2014] recently presented Thumbles, a tabletop tangible system using robots illustrated in several use-cases (see Figure 2.17 (right)). In one of them, Thumbles were used as handles in a protein folding simulation and could oppose manipulations to prevent mathematically improbable actions. Patten described Thumbles as "tactile controls that transform a highly dimensional math problem into a simple mechanical system". Thumbles featured three omni-directional wheels to be able to move in any direction regardless of their orientation.

Vibration has been explored as a way of locomotion, to move small robots on a tabletop touch surface [START_REF] Nowacka | Touchbugs: Actuated tangibles on multi-touch tables[END_REF], or to move a larger single object such as a smartphone on a tabletop. These examples illustrate the potential emerging from internally actuated objects for tangible interfaces. Actuation is not limited by the surface, and combining them with other technologies is less difficult. Yet, embedding actuation as well as communication and power requires space, resulting in larger and more constrained objects. robots that can be used as handles to interact with digital media, but also provide dynamic tactile feedback.

Towards Programmable Matter

Imagining an extreme case of a LEGO kit where each brick would be as smaller than a grain of sand pushes the constructive concept towards Pro-grammable Matter. Programmable Matter is a vision in which physical matter could be computationally controlled at a molecular level to create any physical object dynamically. Using large collections of programmable micro or nano-scaled elements, potentially any kind of physical object could be dynamically created and controlled.

Science fiction has used this vision on repeated occasions. In the second opus of the movie franchise Terminator, the antagonist character "T-1000" was a shape changing android. The T-1000 was made of liquid metal, an alloy of nanorobots that could be reconfigured to take any form. The android could change the shape of its entire body or only parts of it, for example, its hand into a weapon [START_REF] Daniel | Terminator 2 1/2: The Constitution in an Alternate World[END_REF].

Similarly in the movie X-Men, a scene shows an apparently flat surface changing its shape into a high fidelity physical reproduction of the Statue of Liberty. It then morphs into the entire southern tip of Manhattan as shown in Figure 2.18.

Figure 2.18: In the movie X-Men, a surface capable of changing its shape dynamically (Image from [START_REF] Dragicevic | List of physical visualizations[END_REF]).

Envisioning Programmable Matter

Early work in the research community on Programmable Matter came from Ivan Sutherland's Ultimate Display. Envisioned in 1965, he depicted it as "a room within which the computer can control the existence of matter" [START_REF] Sutherland | The ultimate display. Multimedia: From Wagner to virtual reality[END_REF].

Goldstein et al. [2005] presented Claytronics, moving towards programmable matter as large collections of so-calledCatoms (i.e., Claytronics Atoms). Catoms are collaborative millimeter-scale robots capable of moving in space in relation with other Catoms to create and maintain 3D shapes. Collections of Catoms would be digitally controlled capable of assembling themselves to create any kind of physical object, imitating the original physical properties such as shape or texture.

Claytronics remains parts of a long term vision involving many different domains. Creating and managing systems with potentially millions of submillimeter-scale robots raises major challenges. From a technological standpoint, designing and manufacturing bevies of autonomous and collaborative small-scaled robots is not yet achievable.

While Claytronics approaches programmable matter from a technical perspective, bringing into focus the implementation aspects, the vision of Radical Atoms concentrated on how to interact with such material [START_REF] Ishii | Radical atoms: Beyond tangible bits, toward transformable materials[END_REF]. Presented by Hiroshi Ishii as an evolution of Tangible Bits, Radical Atoms depicts interaction with imaginary dynamic physical materials that can morph and be rearranged computationally. Radical Atoms are bidirectional materials changing shape according to reflect modifications of the digital world in the physical world while naturally reacting to physical manipulations. This way, Radical Atoms can display digital information in a physical form as well as sense and react to physical actions. Radical Atoms provide dynamic affordances, through their morphing ability, allowing the user to keep track of the state of the interface and how to interact with it.

Inventing Programmable Matter

To make these visions come true, researchers have been exploring Programmable Matter with contemporary technologies. Roboticists are trying to develop hardware and software platforms to begin prototyping envisioned systems. Kirby et al. developed several prototypes of Catoms, where each element can move relatively to one another without using moving mechanism [START_REF] Kirby | Catoms: Moving robots without moving parts[END_REF][START_REF] Kirby | A modular robotic system using magnetic force effectors[END_REF]. Robot Pebbles are for instance small cubes that can create 2D shapes through subtraction [START_REF] Gilpin | Robot pebbles: One centimeter modules for programmable matter through self-disassembly[END_REF]. Starting in an initial state where all Robot Pebbles are latched together, only the required Robot Pebbles remain bound while the others are released.

M-Blocks are self actuated magnetic cubes [START_REF] John W Romanishin | M-blocks: Momentumdriven, magnetic modular robots[END_REF]. A M-Block can attach to others due to magnets embedded in each of its faces, and uses a controlled internal actuator to create momentum impulses and move freely or with regards to others. Groups of M-Blocks can assemble and reconfigure themselves to create various geometric constructions.

Interacting with Programmable Matter

To overcome the limitation of two dimensional surfaces only, BitDrones uses small quadcopters (i.e., flying drones) [START_REF] Rubens | Bitdrones: Towards levitating programmable matter using interactive 3d quadcopter displays[END_REF]. Focusing on interaction, BitDrones breaks into three types providing different modalities such as touch input and high resolution graphical display. BitDrones allow to explore tangible interaction at room-scale (see Figure 2.19), and introduce input techniques to interact with single BitDrones as well as with groups.

Perfect Red [START_REF] Amerigo Bonanni | Perfect red. Online[END_REF] is a concept illustrating Radical Atoms. Perfect Red is made of an imaginary material for CAD design, similar to clay, computationally augmented and programmed with necessary features, Perfect Red can be modeled with bare hands or using hand tools, and reacts to typical manipulations and actions. For instance, rolling a piece of Perfect Red between two hands causes the piece to morph into a perfect ball. Pushing together two balls joins them, and using a knife to cut this ball in its center splits it exactly in two halves. By recognizing manipulations, Perfect Red supports the richness of hand manipulations, while providing the precision of computers Those are early attempts to move towards programmable matter but pave the ground for future work by bringing ideas, experience and insights to the research community. These systems remain far from what had been in Claytronics and Radical Atoms. Presented catoms are large, slow or even still at a conceptual stage, and rarely support interaction.

discussion and lessons

This section aims at analyzing and discussing the current state of the art of TUIs, to highlight the need for further research to improve versatility and physicality in TUIs.

Discussion

As explained in the introduction (see Section 1.2), the goal of this thesis is to provide tools and technological solution to increase versatility for TUIs. One of my major incentive is to broaden the range of possibilities for physical interaction. This Related Work review shows that research has been undertaken in many different directions. Various visions and paradigms have been explored, numerous new technologies have been invented, many studies were conducted, and each has enriched the field of TUIs.

However, this survey confirms that, to the best of my knowledge, only little work has been carried out to explore combined versatility and physicality in TUIs.

motivations, but each couple actually forms a continuous spectrum, out of which two Illustrated in Figure 2.21.

This taxonomy focuses on the construction and the design choices of TUIs as they directly influence the versatility and the level of actuation of TUIs.

Following are examples of existing work classified in this taxonomy.

inFORM [START_REF] Follmer | inform: dynamic physical affordances and constraints through shape and object actuation[END_REF] and other shape displays are purely physical TUIs as they support interaction without other modality. Yet, they are monolithic given that each element of the display cannot be considered independently. Topobo [Raffle et al., 2004] and UltraTangible [START_REF] Marshall | Ultra-tangibles: creating movable tangible objects on interactive tables[END_REF] are also purely physical TUIs as they support input and output. They are composite as both are composed of multiple elements, and heterogeneous given the diversity of the elements. Bishop's marble machine [START_REF] Polynor | The hand that rocks the cradle[END_REF] is a purely physical TUI, and composite as each marble is physically detached. It is homogeneous as all marble are identical. This partial taxonomy depicts only a portion of the entire variety of TUIs, but focuses on the subdivision of TUIs that has been the least explored so far. Indeed, most of the TUIs illustrated in this chapter are hybrids, as they combine tangibles with graphical content using screens and projection. To the best of my knowledge, the few relying only on physicality as a modality for input and output are made of a single or a few elements. To fulfill my goal of exploring user interfaces combining versatility with high level of physicality (detailed in the introduction, see Section 1.3), I decide to focus on the unexplored area that are composite and homogeneous tangible user interfaces (circled in Figure 2.20).

Thus in the rest of this thesis, I will present how I converge towards homogeneous collections of generic interactive objects to create versatile TUIs. 3

S T U D Y I N G M A N I P U L AT I O N S O F C O L L E C T I O N S O F S M A L L G E N E R I C O B J E C T S
This chapter aims at identifying directions in which the physicality of TUIs can be beneficial over multitouch interfaces for 2D manipulations on tabletop surfaces. Strong similarities exist between the two modalitites, especially when it comes direct manipulation of collections of objects. Using a novel method based on physical proxies, this chapter investigates fundamental differences between TUIs and multitouch interfaces with an experiment involving spatial rearrangment of collections of small objects. This method uses physical chips to emulate the flat, non-graspable objects that make up touch interfaces, in a way that supports direct comparison with TUIs. The use of physical proxies allowed us to isolate the effects of object thickness (flat vs. thick), a fundamental difference between touch and tangibles interfaces irrespective of the available technology to implement them. The results highlight the positive influence of thickness on performances, behaviors as well as subjective experience.

Main portions of this chapter were previously published in [Le Goc et al., 2016a]. Thus any use of "we" in this chapter refers to myself, Pierre Dragicevic, Samuel Huron, Jeremy Boy and Jean-Daniel Fekete.

studying manipulations of collections of small generic objects

introduction

Touch interfaces are commonly thought to support rich and engaging interactions, and are often showcased in futuristic movies and concept videos. At the same time, Bret Victor eloquently argues that touch interfaces are nothing but "pictures under glass" [START_REF] Victor | A brief rant on the future of interaction design[END_REF], and that the gestures they rely on are only a tiny fraction of what human hands are capable of doing.

While such viewpoints can be inspiring, it is also important to evaluate claims rigorously, and to try to pinpoint the fundamental differences between touch and tangible interaction paradigms. Granted for many applications, both interaction styles are likely to be complementary, but understanding how they differ will inform design decisions and future research.

Is touch really holding us back? Does it really ignore the expressiveness of our hands? Or does it just tap into another form of expressiveness? And if touch is only transitional, what technology exactly will allow us to "use our hands"? These questions are very broad and it will take time to answer them on scientific grounds.

Many studies have suggested that people are more effective at operating TUIs (e.g., [START_REF] Alissa | Comparing motor-cognitive strategies for spatial problem solving with tangible and multi-touch interfaces[END_REF][START_REF] Lucchi | An empirical evaluation of touch and tangible interfaces for tabletop displays[END_REF][START_REF] North | Understanding multi-touch manipulation for surface computing[END_REF][START_REF] Schneider | Benefits of a tangible interface for collaborative learning and interaction[END_REF][START_REF] Terrenghi | Affordances for manipulation of physical versus digital media on interactive surfaces[END_REF]) -presumably because TUIs support a richer manipulation vocabulary. These findings are consistent with Bret Victor's assertion that touch interfaces do not allow us to fully "use our hands" [START_REF] Victor | A brief rant on the future of interaction design[END_REF]. However, previous studies do not provide robust explanations as for these advantages -they do not investigate fundamental differences between the two interaction paradigms, i.e., differences that are technology-independent and are likely to hold in future implementations.

This type of study can be sensitive to the specificities of touch and tangible setups, leaving room for eventual confounds. In particular, in all previous studies, the superiority of tangibles can be at least partly explained by limitations of the touch setup that can be addressed by technology that is either available now, or may become available in the future.

Our interest is to establish technology-agnostic methods for answering precise questions regarding differences between touch and TUIs. To illustrate such methods, we used flat and thick objects in order to isolate the effect of object thickness on participants' speed and accuracy when performing spatial rearrangements of multiple objects on a 2d surface.

Although abstract, these tasks are applicable to many real-world situations, studying spatial rearrangement of multiple objects, as these tasks in tasks. These tasks often require manipulating several objects at the same time to move them moving groups from one location to another, and are useful in many situations, such as sorting icons in a workspace, controlling an army of soldiers in a video game, or changing the layout of a graph visualization.

We set ourselves to address the following empirical question: how useful is this extra thickness? In other terms, will people make use of the richer hand movement vocabulary? Will it allow them to be more efficient? Will it yield a more satisfying experience? To answer these questions, we conducted a user experiment involving the manipulation of 40 small objects.

As such, our main contributions are: a novel technology-agnostic method based on physical proxies for studying how (and how well) people spontaneously manipulate typical touch and tangible objects, and an assessment of the effect of object thickness on the manipulation of collections of objects, with implications for the design of both touch and tangible user interfaces.

background

Basic Terminology

There is no general consensus on how touch and tangible interfaces should be defined [START_REF] Shaer | Tangible user interfaces: past, present, and future directions[END_REF]. Therefore, we propose the following working definitions for the purpose of this thesis:

Touch interfaces are user interfaces where digital information is manipulated through skin contact with the surface of a single physical object.

Tangible interfaces are user interfaces where digital information is manipulated by changing the spatial arrangement of multiple physical objects.

These definitions are by no means authoritative: they are purely stipulative and meant to clarify upcoming discussions. Although they focus on input, they align well with what is generally understood by both terms [START_REF] Shaer | Tangible user interfaces: past, present, and future directions[END_REF], and capture a broad range of interactive systems. However, our definition of touch excludes all interfaces that employ contactless (mid-air) sensing, as well as all interfaces that have moving parts; and our definition of tangible excludes single-device input (e.g., regular, gyroscopic or 3D mice, or mobile phones with accelerometers), but includes space-multiplexed input device setups [Fitzmaurice and [START_REF] George | An empirical evaluation of graspable user interfaces: towards specialized, space-multiplexed input[END_REF].

Gray Areas

Some interfaces like Sifteo Blocks [Sifteo] combine touch and tangible input. Our working definitions however provide clear classification criteria: a single Sifteo block is a touch interface, while a collection of Sifteo blocks becomes a tangible interface.

Deformable interfaces (see Section 2.3.1) are more problematic. While flexible [START_REF] Schwesig | Gummi: a bendable computer[END_REF] touch surfaces could be seen as variations of touch interfaces, systems that use physical deformation as input [START_REF] Piper | Illuminating clay: a 3-d tangible interface for landscape analysis[END_REF] would be considered by many as tangible. In the future, topologychanging interfaces may even be able to dynamically switch between a touch and TUI interaction paradigm. Although extremely exciting, this territory remains mostly uncharted. Thus we choose to focus on interfaces where shape change is not used as input.

The Varieties of touch and tangible user interfaces

The many different interaction techniques and technologies available for building touch and tangible interfaces make it very difficult to make general claims about their respective merits. The design space of tangible user interfaces is so bewildering that it prompted the development of several conceptual frameworks and taxonomies as demonstrated in Section 2.1.2. While definitions of TUIs in the domain are purposefully kept broad, here we focus on interfaces that involve multiple spatially-aware objects, a subclass of which are the archetypical "tabletop tangibles" or tangible "workbenches" [START_REF] Shaer | Tangible user interfaces: past, present, and future directions[END_REF].

Commercial touch devices are far from having reached their full potential, either. Lots of progress has been made since the early resistive menubased touch interfaces: software and hardware solutions were developed to alleviate precision issues [START_REF] Lee | The performance of touch screen soft buttons[END_REF], while new sensing technologies were developed to support richer hand gestures, the most influential of which being multitouch [START_REF] Buxton | Multi-touch systems that i have known and loved[END_REF]. But a range of other technologies are being developed, such as contact shape sensing [START_REF] Cao | Shapetouch: Leveraging contact shape on interactive surfaces[END_REF]Wilson et al., 2008], pressure sensing [START_REF] Rosenberg | The unmousepad: an interpolating multitouch force-sensing input pad[END_REF], fingerprint sensing [START_REF] Holz | Fiberio: a touchscreen that senses fingerprints[END_REF], near-zero-lag devices [START_REF] Jota | How fast is fast enough?: a study of the effects of latency in direct-touch pointing tasks[END_REF], as well as non-planar [START_REF] Baudisch | Back-of-device interaction allows creating very small touch devices[END_REF][START_REF] Benko | Sphere: multitouch interactions on a spherical display[END_REF][START_REF] Roudaut | Touch input on curved surfaces[END_REF]Weiss et al., 2010b], malleable [START_REF] David | Low-cost malleable surfaces with multi-touch pressure sensitivity[END_REF] and haptic [START_REF] Bau | Teslatouch: electrovibration for touch surfaces[END_REF] touch surfaces. Although touch interfaces can be indirect (e.g.,, laptop touchpads), many modern touch surfaces are "direct" in that they are colocated with the display.

Taken together all these factors, combined with well-designed or physicsbased [Wilson et al., 2008] interaction techniques, may in the future contribute to making touch interaction feel remarkably "real". Combined with a range of elaborate abstract gestures [START_REF] Zhai | Foundational issues in touch-screen stroke gesture designan integrative review[END_REF], such interfaces could become incredibly powerful.

Comparing Touch and Tangible

Given these definitions, how do touch interfaces compare to tangible interfaces? Although some studies have looked at differences in subjective experience (e.g., [START_REF] Alzayat | Quantitative measurement of virtual vs. physical object embodiment through kinesthetic figural after effects[END_REF]), most studies focused on performance. An early evaluation of TUIs by Fitzmaurice and [START_REF] George | An empirical evaluation of graspable user interfaces: towards specialized, space-multiplexed input[END_REF]] demonstrated the benefits of space-multiplexed input (one physical device per function) over time-multiplexed input (a single generic device e.g., a computer mouse). However, tabletop systems supporting multipoint direct touch input rendered this distinction less useful [START_REF] Tuddenham | Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks[END_REF].

Over the past years, several studies have been conducted comparing multitouch tabletop setups with tabletop TUIs on object manipulation tasks. A replication of Fitzmaurice and Buxton's study revealed that tangibles are still superior [START_REF] Tuddenham | Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks[END_REF]. Similar results have been found for puzzle sorting tasks [START_REF] Alissa | Comparing motor-cognitive strategies for spatial problem solving with tangible and multi-touch interfaces[END_REF][START_REF] Terrenghi | Affordances for manipulation of physical versus digital media on interactive surfaces[END_REF], spatial arrangement tasks [START_REF] Lucchi | An empirical evaluation of touch and tangible interfaces for tabletop displays[END_REF][START_REF] North | Understanding multi-touch manipulation for surface computing[END_REF] and collaborative layout optimization tasks [START_REF] Schneider | Benefits of a tangible interface for collaborative learning and interaction[END_REF]. Observed benefits include faster task completion times, less errors and more user satisfaction, and are generally explained by the fact that tangibles support richer as well as more efficient gestures. For example, people spontaneously rotate physical objects using multifinger grips, which is presumably more efficient than pinch gestures [START_REF] Tuddenham | Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks[END_REF].

Besides user performance, many of the previous studies report observational data on the gestures used. For example, North et al [START_REF] North | Understanding multi-touch manipulation for surface computing[END_REF] identified several gesture types in their study, some of which (e.g., lift or stack) were only possible with TUIs.

Past studies seem to confirm the intuition that touch underexploits our motor skills [START_REF] Victor | A brief rant on the future of interaction design[END_REF]. Yet, while informative, these previous studies have so far focused on comparing specific implementations of touch and tangible interfaces. As a result, findings are hard to interpret and generalize.

For example, studies did not use contact shape sensing, thus providing less expressiveness than what touch can really offer [START_REF] Cao | Shapetouch: Leveraging contact shape on interactive surfaces[END_REF]Wilson et al., 2008]. One exception is [Antle and Wang, 2013], but the setup involved large table edges that could be employed in the tangible condition but not in the touch condition, thus introducing another confound. One study [START_REF] Tuddenham | Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks[END_REF] mentioned "exit errors" as a major cause of performance difference, a problem that might be soon addressable by sensing technologies that can distinguish between drags and rolls (e.g., [START_REF] Holz | The generalized perceived input point model and how to double touch accuracy by extracting fingerprints[END_REF]). Researchers also recently discovered that the lag produced by any currently available touch system negatively impacts performance and user experience [START_REF] Jota | How fast is fast enough?: a study of the effects of latency in direct-touch pointing tasks[END_REF].

On the other hand, it is also easy to give touch interfaces an unfair advantage by having them support more advanced features and interaction techniques. For example, in one study [START_REF] Lucchi | An empirical evaluation of touch and tangible interfaces for tabletop displays[END_REF], undo, lasso selection and snap-dragging were available on the touch interface but not on the tangible one. [START_REF] Zuckerman | To tui or not to tui: Evaluating performance and preference in tangible vs. graphical user interfaces[END_REF] compared TUIs with GUIs using almost identical setups. Although not including touch but traditional mouse and desktop GUI, both used the same modeling and simulation system. Their results showed that the TUI was preferred despite similar performances and its lower perceived usability. Interviewing participants revealed that stimulation and enjoyment were the main reasons for this preference, highlighting three characteristic of TUIs that are "physical interaction, rich feedback, and high levels of realism." This study proves that the strength and richness of TUIs may be lie beyond performances, and encourages further investigations in this direction.

To summarize, previous studies are insightful but often suffer from experimental confounds that make it hard to understand the fundamental differences between touch and tangible interfaces -i.e., differences that will persist and remain relevant as both touch and tangible technologies mature. Thus it would seem beneficial to explore complementary methods that do not depend on currently available technology.

Characterizing Hand Gestures

In order to rigorously test claims regarding the richness of hand gestures, we need a more principled way of describing and categorizing them.

Cadoz contributed extensive analyses on hand gestures [START_REF] Cadoz | Le geste canal de communication homme/machine: la communication'instrumentale[END_REF], of which he identifies three key functions: ergotic (modifying the environment), epistemic (learning about the environment) and semiotic (sending information to the environment). Although useful, this typology involves cognitive aspects such as intention that cannot be easily observed experimentally.

From a purely behavioral point of view, hand gestures can be seen from at least two perspectives: i) the input perspective, i.e., what the computer "sees" and ii) the physical perspective, i.e., what happens in the real world. On a touch interface, this amounts to distinguishing what happens "below" the touch surface from what happens "above". HCI gesture taxonomies typically take an input perspective, they ignore what cannot be sensed by the computer (see [START_REF] Kammer | Towards a formalization of multi-touch gestures[END_REF][START_REF] Zhai | Foundational issues in touch-screen stroke gesture designan integrative review[END_REF]).

Psychology has contributed many studies for understanding the mechanisms of hand movements, often with a focus on prehension, i.e., the act of reaching then grasping an object [START_REF] Jones | Human hand function[END_REF]. But few taxonomies were proposed since Napier's seminal work on hand grips [START_REF] John R Napier | The prehensile movements of the human hand[END_REF]. The field of robotics built on psychology to contribute rich descriptive frameworks and taxonomies for hand gestures. One of the most extensive is by [START_REF] Bullock | A hand-centric classification of human and robot dexterous manipulation[END_REF], who identify five binary classification criteria for hand gestures, with 15 possible combinations. Their taxonomy would classify all touch gestures into a single category, of which tactile surface exploration is an example. Six categories involve object manipulation and thus capture TUI gestures, but none considers manipulating collections of objects.

Despite the large amount of work on hand gestures in several disciplines, there is remarkably little work on the manipulation of collections of objects.

In particular, all descriptive frameworks and taxonomies of manipulative actions we know, except [START_REF] North | Understanding multi-touch manipulation for surface computing[END_REF], or assume a single object. Since TUIs as well as many real-world tasks involve the concurrent manipulation of multiple objects, clearly more work is needed in this direction.

methodology

Here, we explain and motivate our proxy-based method for comparing touch and tangible interfaces.

Rationale for Using Physical Proxies for Touch

The goal of a physical proxy is to help empirically investigate user interfaces that are hard to build, or that may only exist in the future. Nevertheless, we stress that a physical proxy necessarily differs from the user interface it emulates.

Physical proxies have already occasionally been used for comparing 2d and 3d visualization techniques. Dwyer has proposed hand-crafted physical models to emulate perfect 3d displays [Dwyer, 2005, pp. 39-45] , while Jansen has suggested to use static physical visualizations in user studies as substitutes for future self-actuated physical visualizations [Jansen, 2014a, p. 133].

Although both touch interfaces and TUIs will keep improving, touch will only ever allow flat objects to be operated from above, while TUIs allow objects with volume to be pushed sideways, grabbed, lifted, etc. Therefore, we use thin discs or "chips" as proxies for touch-operated objects, and cylinders or "pucks" as proxies for tangible objects (Figure 3.1).

The proxy for the complete touch interface consists of a collection of chips placed on a planar surface to emulate GUI icons that can be freely dragged around. This setup necessarily differs from today's touch interfaces in several respects, while sharing several characteristics with an "idealized" touch interface. More specifically, the differences are:

Df1. with enough effort, the proxies can be grasped and lifted, something that is impossible on actual touch interfaces;

Df2. the proxies can slide and fall off the surface if it is inclined-which means that our particular proxy implementation is inappropriate for hand-held and mobile touch devices;

Df3. the proxies' collision behavior is unusual for traditional touch interfaces: the proxies can both push each other and overlap-unlike graphical objects which either interpenetrate or collide, depending on the chosen metaphor; and

Df4. the proxies do not support abstract gestures or advanced commands (e.g., selection, grouping, undo), which are common in touch interfaces.

Nevertheless, this proxy setup can help us learn more about how touch interfaces may differ from tangible interfaces by making the comparison fairer in other respects, since the setup:

• has zero-lag: lag on any current touch interface has been found to be noticeable and detrimental [START_REF] Jota | How fast is fast enough?: a study of the effects of latency in direct-touch pointing tasks[END_REF];

• has infinite resolution, both in terms of input and output;

• provides physically realistic direct manipulation, both for single and multiple objects [START_REF] Cao | Shapetouch: Leveraging contact shape on interactive surfaces[END_REF]Wilson et al., 2008];

• provides realistic rendering: in particular, no light is emitted by the surface (similar to, e.g., an e-ink display);

• provides realistic touch haptics [START_REF] Bau | Teslatouch: electrovibration for touch surfaces[END_REF]: object edges can be felt, and no skin friction occurs when an object is dragged.

Currently, only TUIs have all these characteristics. However, none of them are fundamentally incompatible with touch interfaces, for which both display and sensing technology is quickly improving: a lag of 1 ms has been recently achieved in research prototypes [START_REF] Jota | How fast is fast enough?: a study of the effects of latency in direct-touch pointing tasks[END_REF], as well as tactile feedback [START_REF] Bau | Teslatouch: electrovibration for touch surfaces[END_REF]; but without these characteristics, TUIs have an unfair advantage in comparative studies. The goal of our method is to level both types of interface for comparing them at a fundamental level.

Research Questions

Our goal is to compare how (and how well) chips and pucks are spontaneously manipulated, in order to assess the sole impact of object thickness, a fundamental (i.e., implementation-independent) difference between touch and tangible interaction paradigms. Our two major research questions are:

1. Does the extra thickness of pucks present any manipulation benefits? That is, do people spontaneously make use of the extra spatial manipulation strategies offered?

2. How good is our physical proxy implementation? That is, to what extent do Df1-Df4 affect the way in which chips are manipulated vs. the way we expect them to be manipulated in an idealized touch interface?

Rationale for the Choice of Setup and Tasks

To this effect we chose to have people manipulate objects on a table under two conditions: one only involving thin objectds (i.e., chips), and another one only involving thick objects (i.e., pucks). Since no computation is involved, none of the two setups qualifies as a user interface in the HCI sense. The first condition is a physical proxy for touch, while the second is a (relatively realistic) proxy for TUIs. For the comparison between chips and pucks to be fair, and to address Df2 at the outset, we chose to simulate tabletop touch and tabletop tangible interfaces, which is what has been compared in the vast majority of previous studies [START_REF] Alissa | Comparing motor-cognitive strategies for spatial problem solving with tangible and multi-touch interfaces[END_REF][START_REF] Lucchi | An empirical evaluation of touch and tangible interfaces for tabletop displays[END_REF][START_REF] North | Understanding multi-touch manipulation for surface computing[END_REF][START_REF] Schneider | Benefits of a tangible interface for collaborative learning and interaction[END_REF][START_REF] Terrenghi | Affordances for manipulation of physical versus digital media on interactive surfaces[END_REF][START_REF] Tuddenham | Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks[END_REF].

To address Df4, we limited our study to manipulative gestures [START_REF] Kammer | Towards a formalization of multi-touch gestures[END_REF], which both setups support. Direct manipulation is the main interaction style in TUIs and it is also dominant in modern touch devices; and while in principle TUIs could support symbolic gestures, most currently do not.

Finally, to understand how users manipulate collections of objects (chips and pucks), we chose to use repetitive spatial rearrangement tasks-in a similar fashion to previous studies such as [START_REF] North | Understanding multi-touch manipulation for surface computing[END_REF]. Such tasks are reasonably complex in the motor domain (compared to, e.g., target acquisition tasks), while at the same time devoid of high-level cognitive components that are typically present in real-world tasks. The focus of the study is not on high-level cognition, but on how people naturally and spontaneously "use their hands" [START_REF] Victor | A brief rant on the future of interaction design[END_REF].

Although abstract, such tasks have the benefit to help separating out highlevel cognitive aspects, to focus only on motor aspects. Of course more elaborate proxy implementations and more realistic tasks can be considered in the future, and may help uncover more complex differences between touch and tangible interfaces.

experimental procedure

We now describe in more detail our setups, tasks, procedure, and experiment design.

Physical Proxy Setup

The touch and TUI proxies shared the same setup (see Figure 3.2). The setup included a white plastic support surface, with 3 mm-thick edges to prevent objects from sliding off.

We digitally-fabricated forty 0.2 mm thick disks (chips), and forty 12 mm thick cylinders (pucks); all had a diameter of 25.75 mm (the size of a 2euro coin). The chips were made of transparent laser-cut mylar sheets, and weighed roughly 0.2 g (see Figure 3.1 left). We chose mylar as it remains rigid, even when very thin. This way, chips would be difficult to bend and therefore difficult to grasp. The pucks were made of several layers of lasercut transparent acrylic sheets glued together, and weighed roughly 7.5 g (Figure 3.1 right). We informally verified that mylar and acrylic had similar friction properties. Finally, all chips and pucks were spray-painted, half of them blue and the other half red. Chips were only painted on top, and pucks were painted on top and on the sides. Thus both object types had an unpainted bottom.

Experimental setup

In addition to the physical proxy, the experimental setup included electronic equipment for displaying task instructions and recording participant performance (see Figure 3.2).

A projector 1 was hanging at 1.60 m above the table. The table was divided in five areas: the main area, where participants were manipulating objects 2 and on which instructions and targets were projected with a resolution of 1080 p, two storage areas in upper corners of the table 3 , and two sorting target areas on lower corners of the table 4 . Three recording cameras were used: a HD webcam 5 for taking photo snapshots of the workspace, and two cameras ( 6 and 7 ) for videotaping the entire experiment. Two light spots ensured constant lighting conditions. Finally, a pushbutton 8 we call the timer button was used to let participants navigate through the experiment. No other input device was used.

Tasks

An experimental task consisted of moving all objects from an initial configuration to a final configuration. There were 3 different types of configurations, all involving the same 40 objects (20 blue and 20 red, see Figure 3.4):

• Random-packed: all objects are packed within a disc of diameter 24 cm at the center of the workspace. They fill 46% of the disc area, with an inter-object spacing of ∼0.5 cm.

• Random-spread: same arrangement but within a larger disc of diameter 40 cm (density 17%, separation ∼2.5 cm).

• Sorted: all red objects are placed on one side of the table, while all blue objects are placed on the other. Object location within the area is not important.

From these three configurations we derive four different tasks:

• Packed sorting: from random-packed to sorted.

• Spread placement: from sorted to random-spread.

• Spread sorting: from random-spread to sorted.

• Packed placement: from sorted to random-packed.

Since the final configuration of a task is also the initial configuration of the next task, these tasks could be performed in sequence and looped over without the need for actuation or experimenter intervention. Object colors in placement configurations were randomly shuffled, and sorting areas were randomly swapped. We henceforth refer to a sequence of four successive tasks as a block (see Figure 3.3). A block can start with any of the four tasks above, thus four different block designs are possible.

Tasks were given to subjects by top-projecting the desired final configuration on the workspace, during the entire task duration. Visual targets were made dim enough so as not to interfere with physical manipulation. For placement tasks, individual object targets were shown with red and blue semicircles 120% the size of objects (30.9 mm) (see Figure 3.4). For sorting tasks, a red line and a blue line were displayed on each side of the table. Objects simply had to be moved across the lines.

The use of semicircles instead of circles is due to limitations of the top projection setup. Since the projector could not be oriented perpendicularly to the table, pucks casted a shadow, which pilots suggested might impair puck placement tasks. The use of semicircles addressed this issue, at the cost of rendering placement tasks less well-specified. This was addressed by instructing participants not to focus on accuracy, and by later analyzing both completion times and placement accuracy.

Task Instructions

We wanted to observe spontaneous manipulation behavior, thus no instruction was provided in the way objects were to be manipulated. In particular, the touch proxy was not presented as such, and no reference was made of touch or tangible interfaces. Objects were referred to using the neutral names chips and pucks. At no point did the experimenter manipulate chips or pucks in front of participants.

Participants were instructed to be as fast as possible without committing any obvious placement error, i.e., all objects were required to be approximatively placed within their respective targets (or beyond lines for sorting) with no color inversion. Emphasis on speed was used as an incentive for participants to converge to optimal motor strategies. It is reasonable to assume that similar optimal strategies will be developed in natural settings after extended use of such user interfaces.

The system did not check for correct object placement. Instead we told participants that a photo snapshot will be taken each time they complete a trial, and trials will be later examined for correctness. Correctness was also enforced by the experimenter during practice trials.

procedure

The participant was asked to sit comfortably at the table and was given an instruction sheet. All 80 objects had been previously placed in storage areas ( 3 in Figure 3.2), with chips and pucks stored in their respective areas.

The participant was first asked to bring all objects from one storage area to the main area (chips or pucks depending on the participant) and to complete a discovery blocki.e., four tasks in sequence -where she could get acquainted with the experiment logics without time pressure. Then the participant was asked to switch object type (see procedure below) and complete a training block with time pressure but no performance recording. The participant then switched objects again and went on performing the experimental trials.

An experimental trial consisted in the administration of a single (placement or sorting) task and proceeded as follows:

1. A message invites the participant to press the timer button ( 8 in Fig-

ure 3.2) when she is ready to start, upon which a two-second countdown appears on the table.

2. Two seconds later the configuration to achieve is displayed, and a beep sound is played signifying that the timing started.

3. The participant moves the objects to their final destination ( 4 for sorting tasks or 2 for placement tasks).

4.

As soon as the task is completed, the participant presses the timer button again and a second beep sound is played, signifying that the timing stopped. The projected configuration disappears and a message invites the participant to lean back and get her hands off the table.

5. Two seconds later, a photo snapshot of the workspace is taken (using 5 in Figure 3.2), accompanied with a camera shutter sound. The invite message for the next trial is then displayed (step 1 above).

This trial design together with the unpredictable ordering of colors prevented participants from pre-planning their motor actions. This incentivized participants to interleave motor planning with motor execution, a strategy closer to typical UI object manipulation scenarios. Participants were however allowed to place their hands as they wished during the two-second countdown, as long as they were not touching any object. Thus hand placement, which was irrelevant to the purposes of the experiment, was not included in task times.

Every two blocks, the participant was asked to switch to the other object type (chip or puck). Object switching followed the following procedure: The participant is asked to move all current objects in one of the storage areas, then move all the objects from the other storage area in front of her. Then the final configuration from the last task is displayed and the participant is invited to place all objects, without time pressure.

All experiment instructions were given through short messages and visual annotations, all projected on the table (using 1 in Figure 3.2). Each participant was recorded from different angles using two cameras ( 6 and 7 in Figure 3.2). At the end of the experiment, the participant was asked to fill in a brief questionnaire. The experiment lasted 35 minutes on average.

Design

Our independent variables were object type Object ∈ {Chips, Pucks} and type of task Task ∈ {Packed-placement, Spread-placement, Packed-sorting, Spread-sorting}, both within-subject factors. Each combination of Object and Task was administered four times, i.e., four Block were generated per Object condition. Our dependent variables were time-on-task Time, and placement accuracy Accuracy.

Accuracy was only measured for Packed-placement and Spread-placement tasks. For each trial, the 2D position of all 40 objects was extracted from photo snapshots through a combination of automatic image processing (perspective correction, color segmentation and ellipse fitting) and manual correction. These 40 object positions were then matched with the corresponding (i.e., nearest) slot from the placement template. As a second step, for each slot, the dispersion of object positions was estimated by taking the unbiased estimate of their standard distance deviation (SDD). Four distance measurements were used in computing each SDD (one per repetition). SDDs were then averaged across all 40 slots, yielding a unique measure of placement accuracy per Object and Task ∈ {Packed-placement, Spread-placement} combination.

To reduce learning effects, Object was switched every two blocks. In addition, the presentation order of the first Object was counterbalanced across participants as well as the first task presented in all blocks, yielding 2 × 4 = 8 unique designs. Red and blue colors in task configurations were fully randomized.

Overall the experiment consisted in 2 Object × 4 Task × 4 Block = 32 trials, yielding 32 measures of Time and 4 measures of Accuracy per participant.

Participants

We recruited 8 participants (5 male), all naive to the purposes of the experiment, by email announcement to our lab. Ages ranged from 24 to 32 with a median of 28. All had a university degree, from Bachelor to PhD. They were all right-handed, owned a touch device and were familiar with such technologies (as reported in the post-experiment questionnaire). All participants agreed to be videotaped.

performance and subjective experience

Due to growing concerns about the limits of null hypothesis significance testing for reporting and interpreting results [START_REF] Dragicevic | Fair statistical communication in hci[END_REF], we base our analyses on simple effect sizes with confidence intervals (CIs). See [START_REF] Cumming | Inference by eye: Confidence intervals and how to read pictures of data[END_REF] on how to interpret plots with CIs.

Time

We perform our time analyses on log-transformed data [START_REF] Sauro | Average task times in usability tests: what to report[END_REF]. We thus report geometric means, and all comparisons are expressed as time ratios. CIs assume lognormal distributions.

Completion times were averaged per participant for each combination of Object and Task. The results shown in Figure 3.5 confirm that pucks are overall faster than chips. There is strong evidence for placement tasks, and a possibly similar trend for sorting tasks, but the evidence is much weaker.

Accuracy

Accuracy observations were averaged per participant and per Object condition, and population means (in mm) as well as within-subject ratios (dimensionless) were estimated using 95% BCa bootstrap CI. Results are shown also in Figure 3.6: on average, object placement variability ranges from 6 to 8 mm, with clear signs that chips tended to be positioned more consistently than pucks on placement tasks. Thus it appears that pucks can be manipulated faster than chips under our setup, but their advantage is less apparent in sorting tasks. They also tend to be positioned slightly less accurately.

Subjective Experience and Feedback

Our post-experiment questionnaire included five items to assess participants' subjective experience, and also invited them to elaborate on their responses. Figure 3.7 shows participants' mean responses, given on 5-point Likert scale.

Unsurprisingly, participants strongly agreed that pucks "felt physical". However, participants were also reluctant to admit that chips felt like a touch interface and reported that they also felt physical to some extent. Looking at spontaneous comments, tactile feedback seems to have strongly influenced their experience. One participant commented that "the tactile feedback is different [from a touch interface]", and another that "chips provided a lot more tactile feedback than a glass pane".

Responses revealed a remarkably more positive experience with pucks. Participants reported that they "became increasingly frustrated with [their] inability to pick and place the chips", while "pucks are easier to handle and [...] more comfortable to manipulate". Participants reported that pucks made them feel more efficient and more in control: "You feel more in control with the pucks, they are easier to grasp".

A participant further elaborated "Because pucks could be picked up and placed, they allowed for a wider array of placement strategies [...]. Moving the last few chips into position often required sliding them all the way around the already placed ones, which felt cumbersome. I preferred having the ability to use multiple placement strategies with the pucks and their physicality was satisfying".

Discussion

Participants' comments give us hints as to why pucks were manipulated faster and yielded a better subjective experience. Participants seemed frustrated that the chips could not be grasped and lifted. Indeed, we observed that placement tasks required more caution not to muddle up already placed chips while sliding new ones in.

In contrast, pucks could be easily picked and placed anywhere. Arguably, in many real touch interfaces dragged objects go through other objects, thus emulating "pick and place" operations. This was not possible with chips. Note that it is also not possible with touch metaphors that implement collision detection [START_REF] Alissa | Comparing motor-cognitive strategies for spatial problem solving with tangible and multi-touch interfaces[END_REF][START_REF] Cao | Shapetouch: Leveraging contact shape on interactive surfaces[END_REF]Wilson et al., 2008]. However, this metaphor also generally ensures that objects never occlude each other, which was not the case for our proxy.

Despite the limitations of our proxy, there is still evidence that pucks yielded benefits that even an idealized touch interface could not provide. First, the sorting task is a coarse motor task that should not have suffered from chip collision and overlap -yet, our results suggest that pucks might have been still faster. Second, as we will see with the video analysis, issues with overlap were not that common and chips appear to have suffered from problems unrelated to overlap and collision.

video analysis

We now report on a video analysis of the gestures performed by participants in the Chips and Pucks conditions.

Manipulation Gesture Typology

The first step of the coding process consisted in agreeing on an initial set of gestures. We focus on physical gestures, not gestures in the traditional HCI (input) sense.

We initially considered using the hand manipulation taxonomy from Bullock et al. [START_REF] Bullock | A hand-centric classification of human and robot dexterous manipulation[END_REF], but the classification criteria turned out to be less helpful than we expected: virtually all manipulations involved hand motion, contact and no motion at contact. Although regular touch gestures involve motion at contact, the metaphor is really that of no motion at contact (e.g., pushing a coin), and our proxy implemented this metaphor literally.

We therefore decided to identify recurrent gestures based on video footage. Doing so we noticed common manipulation errors-i.e., manipulations with unintended consequences. We thus agreed on a typology that included a total of 21 gestures and errors, of which we report 13 here (this selection will be explained later on). The manipulation gestures were:

Fine drag: Move object(s) while pushing against their top surface with one or more fingertips. Coarse drag: Move object(s) while pushing against their top surface with another body part (e.g., flat hand). Grasp drag: Move object(s) by grasping and not lifting them. Lift: Move object(s) by grasping them and lifting them. Bulldozer push: Move object(s) by pushing sideways, with the body in contact with the surface. Throw: Move object(s) by grasping / pushing and releasing contact during transport. Separate: Move object(s) in order to break contact with surrounding object(s).

Whereas the manipulation errors were:

Miss: Contact with object(s) is not achieved while intending to initiate a manipulation. Loss: Contact with object(s) is unexpectedly lost. Collision: Object(s) being manipulated accidentally collide with surrounding objects. Overlap: Moving several overlapping objects while intending to move less of them. Stick: Object sticks to skin when attempting to release contact. Escape: Object moves by itself instead of remaining still.

Method

We wanted to focus on close-to-optimal strategies and thus only analyzed two blocks (8 trials per participant) from the video recordings: the last block involving chip manipulation and the last block involving puck manipulation. This represented about 30 minutes of video for all 8 participants. Two coders independently annotated video logs using ChronoViz [START_REF] Fouse | Chronoviz: a system for supporting navigation of time-coded data[END_REF]. Each time any of the 21 gestures or manipulation errors was seen occurring, the coder annotated its exact starting time as well as the hand used. This process took each coder about 30 hours, for a total of 5000 gestures and manipulation errors reported.

Inter-Coder Agreement

In order to estimate inter-coder agreement, annotations had to be temporally aligned. We used a three-pass algorithm. The first pass searched for matching annotations. A match was defined by two annotations having the same category, the same hand and being close in time (±350 ms).

A second pass then searched for non-matches, defined as two annotations having the same hand, being close in time, but having different categories. Remaining annotations were discarded. Finally each remaining annotation was mapped to an annotation with the "none" category.

We assessed inter-coder agreement using Cohen's Kappa (K) [START_REF] Cohen | Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit[END_REF]. K ranges from -1 to 1, with 1 being a perfect agreement, 0 what would be expected by chance.

We added the hand used (left or right) to our 21 categories, yielding a total of 42. The coder agreement was K=0.47, a "moderate" agreement [START_REF] Cohen | Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit[END_REF]. We noticed that one coder annotated more gestures than the other (12% more), suggesting a detection threshold issue whereby micromovements were ambiguously categorized. By micro-movements, we mean very short manipulations that were not necessarily useful. These micromovements were thus discarded in the rest of the analysis, leading to a "substantial" agreement of K=0.79.

Results

Figure 3.8 reports gesture occurrence frequencies per Object condition and per coder. Error frequencies are relative to the total number of gestures. The last column reports per-gesture coder agreements, ranging from "fair" to "substantial".

Most importantly, our effects of interest, i.e., differences in percentages between object types, are generally quite consistent across coders. We only report the 13 manipulation gestures and errors whose occurrence was more than 0.1% for at least one coder and Object condition.

Looking at the diversity of manipulation gestures, fine drags (typical of touch interfaces) represents the vast majority of gestures in the Chips condition. Coarse drags and bulldozer push gestures, are also relatively common. Both are possible with touch interfaces, although not often implemented [START_REF] Cao | Shapetouch: Leveraging contact shape on interactive surfaces[END_REF]Wilson et al., 2008].

Surprisingly, fine drags are also the most common gesture with Pucks: participants typically manipulated pucks as if they were using a touch interface. However, the gestures are also more diverse, with more throw gestures, and gestures like lifts and grasp drags that are impossible with touch. Lifts have been observed only very rarely with Chips, confirming that Df1 was not a problem with our proxy setup.

Despite lower coder agreements (possibly because identifying errors partly requires inferring user intent), both coders reported much more manipulation errors with Chips, possibly because identifying errors partly requires inferring user intent. Approximately twice as many losses are reported (e.g., a few chips are left behind during bulldozer pushes), and misses (e.g., a finger lands next to a chip) also seem much more common.

These results also help us understand to what extent chip overlap (Df3) was a problem. Overlaps caused manipulation errors in only about 3% of all gestures, and only 3-4% of all hand gestures were for separating overlapping chips. This suggests that although the effect of chip overlap is measurable, other issues such as losses or misses are at least as important.

Finally, the dominant and non-dominant hands were used about equally often in both Object conditions.

Understanding Chip Losses and Misses

Video recordings revealed that chip losses happened during bulldozer pushes and coarse drags. Figure 3.9 shows an instance.

We informally experimented with bulldozer pushes by lining up chips so overlap is impossible. It appeared that chips are lost because it is hard to maintain a large contact area between the body and the surface. Full contact is not required with pucks because they are too thick to slip below the arm. Similarly we repeated coarse drag gestures and found that chips are lost because it is hard to distribute body pressure on several chips. With pucks, this is much easier because the empty space around makes it possible to orient and shape the hand in different ways, and the intervening space allowed the flesh to deform in a way that distributes forces more equally.

Misses cannot be easily explained by taking a two-dimensional perspective, since chips and pucks have the same radius. However, since pucks "stick out", they likely facilitate quick establishment of lateral contact. This additional error tolerance on the third dimension may have made puck acquisition both less error-prone and less attention-demanding, considering that bimanual tasks spanning a large surface like ours require split visual attention. Also, failure or success to acquire a puck is signaled with more distinctive haptic and auditory cues, thus likely reducing the need to visually monitor for misses. As one participant mentioned, it was harder to manipulate the correct amount (of chips), especially if you do not look at them.

general discussion

We turn back to our initial research questions and attempt to answer them based on the empirical data we collected.

Are Chips a Good Proxy for Idealized Touch Interfaces?

People spontaneously manipulated chips as if they were using direct touch interfaces, i.e., by dragging one or more with their fingertips. Coarser ges-Figure 3.9: Four chips lost during a bulldozer push (top), while no loss happens with the pucks (bottom).

tures were also used to manipulate many chips concurrently, which is also possible on touch interfaces using contact shape sensing [START_REF] Cao | Shapetouch: Leveraging contact shape on interactive surfaces[END_REF].

A single participant used once a strategy consisting in regrouping chips, lifting them together, and placing them one by one. This can easily be avoided in future experiments through explicit instructions.

The way chips physically interacted may have been unusual for a touch interface. While touch interfaces either always allow object overlap during manipulation or never do (through collision detection), physical chips implement a mix of both. Some participants complained about both unintended overlaps and the necessity to avoid already-placed chips. This issue needs to be addressed in future proxy setups, for example by designing ultra-flat objects that consistently collide.

Concerning self-reported subjective experience, chips felt less "physical" and more "touch-like" than pucks, but they still felt much more physical than current touch interfaces. Participants were highly sensitive to the tactile feedback, while the realistic visual "rendering" and the absence of lag may have also significantly contributed to this experience. All these factors only confirm that chips are not a good proxy for today's touch interfaces, which is not what chips are trying to emulate. Device vendors have been successfully building touch devices that provide a much more physically plausible experience than old touch screens, and this trend is likely to continue.

Does Object Thickness Present Benefits?

Our study seems to suggest this was the case, but not necessarily for the reasons often mentioned in the literature. Pucks turned out to be faster than chips but as often in experiments, speed only tells a tiny fraction of the story. While we eliminated some sources of confound (lag, sensing errors, etc.) by manipulating object thickness only, the overlap and collision behavior of flat objects introduced another confound. Using our new method yielded interesting insights. First, while previous studies find that tangibles afford more accurate manipulation [START_REF] Jansen | Tangible remote controllers for wall-size displays[END_REF][START_REF] Tuddenham | Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks[END_REF][START_REF] Weiss | Slap widgets: bridging the gap between virtual and physical controls on tabletops[END_REF], we found that chips are positioned more precisely than pucks. This result points out benefits of touch interfaces that may have been overlooked. In particular, because they are flat, touch interfaces can be more legible, as objects do not occlude content as much as tangible objects. This may in turn facilitate precision tasks.

Second, it has been argued [START_REF] George W Fitzmaurice | Bricks: laying the foundations for graspable user interfaces[END_REF][START_REF] Victor | A brief rant on the future of interaction design[END_REF] and observed [START_REF] North | Understanding multi-touch manipulation for surface computing[END_REF] that tangibles support richer gestures. Our study calls for more nuance. When repositioning objects in a 2D workspace, finger drag is a dominant gesture even with tangibles. It appears to be the best for positioning small objects. Being "graspable" thus does not necessarily lead to grasping behavior. Naturally, our findings only apply to a specific type of task, and to a particular tangible form-factor appropriate for the task.

Despite this, tangibles were often lifted and grasped, and it is clear that this extra flexibility yielded a much more positive subjective experience. However, it is not clear whether the same difference in satisfaction would have been observed had participants been able to drag chips over other chips, as is often the case in touch interfaces. This finding thus calls for further studies before we can make any strong claim about speed or subjective benefits of thick objects.

A key insight provided by our empirical method is that when multiple objects need to be manipulated at a time, touch interfaces are limited not because objects cannot be grasped, but because human hands and arms are too irregular to interact in elaborate ways with flat surfaces. Indeed, coarse manipulations yielded many unintended chip losses, a problem that is easily addressed by the extra thickness of pucks. Despite previous experimentations with contact shape sensing and coarse manipulation gestures similar to what we observed [START_REF] Cao | Shapetouch: Leveraging contact shape on interactive surfaces[END_REF]Wilson et al., 2008], touch interfaces may be sub-optimal for such tasks. This issue will likely become more apparent as contact shape sensing becomes mainstream, and may require specific interaction techniques and/or mid-air (thus non-touch) sensing equipment.

Finally, we found that thick objects were easier to acquire, likely because i) they "stick out" as targets and ii) they generate haptic "error messages" when mis-acquired. This suggests that touch surfaces may be less comfortable for carrying out complex coordinated tasks over large areas, even with full support for haptic rendering [START_REF] Bau | Teslatouch: electrovibration for touch surfaces[END_REF]. Nevertheless, participant comments suggest that tactile feedback is a key component of the perceived "physicality" of a touch interface.

conclusion and future work

We were interested in comparing touch interfaces with tangible interfaces for repositioning collections of objects on a 2D workspace. The use of physical proxies allowed us to isolate the effects of object thickness (flat vs. thick), a fundamental difference between touch and tangibles interfaces irrespective of the available technology to implement them.

Like previous studies, we observed that graspable objects are manipulated faster than their flat counterparts. However, they were also positioned less accurately, presumably because of the occlusions they entail. Video analysis also revealed that in both conditions, participants favored finger dragging gestures typical of touch interfaces, although they grasped and lifted thick objects when it was more convenient. Meanwhile, we raised two possible limitations of touch that may seem obvious in retrospect but should probably be given more emphasis, as they may be inherent to any touch interface: i) touch objects are more prone to acquisition errors as they are not necessarily acquired orthogonally, ii) touch surfaces are flat but human hands and arms are not, which can cause object losses when trying to drag or sweep many objects at a time. Adding thickness to objects solves this issue, making the manipulation of multiple objects more comfortable, less error-prone and less attention-demanding.

The physical proxy approach definitely has limitations and will never replace traditional evaluation methods, but it can be a useful complement -it provides a totally different way of looking at comparative questions. It helps clearing the space and understand interesting aspects of TUIs. The results are promising and give support further exploration of tabletop composite TUIs. 4

S M A L L G E N E R I C O B J E C T S A S I N P U T D E V I C E S
The previous chapter presented the study we conducted to better understand how collections of small objects are manipulated, and whether thick objects present benefits compared to flat objects as found in multitouch interfaces. The results provide motivation for further exploring composite TUIs using collections of small interactive objects.

The objects from the previous chapter were passive, this chapter focuses on augmenting objects with sensing capabilities. It introduces SmartTokens, small-sized generic objects that can sense multiple types of motion, multiple types of grip, and send input events wirelessly as state-machine transitions.

By providing an open platform for embedding basic sensing capabilities within small form-factors, SmartTokens extend the design space of tangible user interfaces. This chapter describes the design and implementation of SmartTokens, and finally illustrate how they can be used in practice by introducing a novel TUI design for event notification and personal task management.

Main portions of this chapter were previously published in [Le Goc et al., 2015a]. Thus any use of "we" in this chapter refers to myself, Pierre Dragicevic, Samuel Huron, Jeremy Boy and Jean-Daniel Fekete.

introduction

A large amount of previous work on TUIs has focused on tracking tangible objects location with external sensing equipment. Various techniques of sensing are used, for instance interactive tabletop surfaces [START_REF] Ullmer | The metadesk: models and prototypes for tangible user interfaces[END_REF][START_REF] Weiss | Slap widgets: bridging the gap between virtual and physical controls on tabletops[END_REF], or cameras [START_REF] Andrew D Wilson | Depth-sensing video cameras for 3d tangible tabletop interaction[END_REF]. They offer the convenience of using the built-in sensing capabilities, and simple objects can then be easily augmented with fiducial markers [START_REF] Kaltenbrunner | reactivision: a computer-vision framework for table-based tangible interaction[END_REF] or RFID tags [START_REF] Joseph A Paradiso | Tangible music interfaces using passive magnetic tags[END_REF]. In most cases, these installations are heavy and require specific environmental conditions. Other projects have used mobile touch devices [START_REF] Jansen | Tangible remote controllers for wall-size displays[END_REF], and augmented existing devices with extra-sensing capabilities (e.g., magnetic sensing [START_REF] Liang | Gaussbits: magnetic tangible bits for portable and occlusion-free near-surface interactions[END_REF]).

However, these traditional approaches have limitations. Camera-based tracking systems are sensitive to lighting conditions and are prone to hand occlusion, while tabletop tangibles require tangible objects to rest on a horizontal surface. In addition, the use of external sensing devices restricts the space where tangibles can be operated, and makes it hard to move the user interface from one place to another.

It is possible to build TUIs without the use of external sensing technologies. One solution is to use smartphones as tangible objects (e.g., [START_REF] Klum | Stackables: combining tangibles for faceted browsing[END_REF]). Smartphones are widely available but are costly and can be too bulky for some TUI applications, especially when large numbers of tangibles need to be used. More specialized solutions exist like Sifteo cubes [Sifteo]. Although these devices are more compact and cheaper than smartphones, some TUIs may ideally require an even smaller form factor, e.g., tokens the size of a two-euro coin that can be grasped easily, either alone or in small number.

To summarize, there is a gap in available TUI technology. Existing devices with embedded sensing capabilities are generally feature-rich but can be costly and bulky. On the other hand, passive props are cost-effective and can be arbitrarily small but they require external sensing equipment. Neither approach is able to fully tap into our ability to manipulate multiple physical tokens, such as coins or game pieces. Actual tokens are most of the time small and can not be always tracked by external sensing equipments. To fill this gap, we propose a design that embeds sensing and wireless communication capabilities into small tangible tokens, making them "aware" of their manipulation. We introduce SmartTokens, small-sized tangible tokens that can sense information on motion and grip and transmit this information wirelessly. SmartTokens can be used as building blocks for developing sensor network user interfaces (SNUIs), i.e., "distributed TUIs in which many small physical manipulatives have sensing, wireless communication" [START_REF] Merrill | Siftables: towards sensor network user interfaces[END_REF]. The unique combination of small form factor and multiple sensing capabilities makes it possible to build token-based SNUIs. Size constraints offers interaction and manipulation opportunities but introduce several technical challenges, both hardware-and software-related, discussed later in this chapter. We also present a use case of SmartTokens involving notification and task management.

By adopting an object centric approach, we bypass restrictions imposed by external sensing systems. The global system is flexible and the interaction space is no longer tied to a specific static area. Tailor-made sensors are embedded in physical objects to provide only the required capabilities, thus limiting cost and maximizing adaptability in form-factor. Moreover, this device allows to capture manipulations performed by users on a daily basis and not supported by existing system. Stacking, grouping, throwing and similar operations can easily be detected by the device. We illustrate the potential of these interactions by using this device into a mail notification system. We further identify and define a group of novel interactions through fictional scenarios which exemplify the potential of our device.

As such, the main contributions of this chapter are:

A small form-factor TUI device that embeds grip and motion sensing at a low cost.

An open source and open hardware platform to help explore, tinker with, and prototype novel types of TUI designs.

The "tangible notification machine", a novel TUI that illustrates how SmartTokens can be employed for the purposes of peripheral notification and personal task management.

background and motivation

Tangible objects come in many different form factors. Among them, tokens are discrete, usually simple and small physical objects representing a single piece of information [START_REF] Ullmer | Token+ constraint systems for tangible interaction with digital information[END_REF]. For example, a coin stands for a monetary value, while a piece in a board game can stand for a player. Tokens have been used for thousands of years [START_REF] Schmandt-Besserat | How writing came about[END_REF] (see Section 2.1.2), and more recently, HCI researchers have explored how they can be used to interact with digital information in various applications such as voicemail [START_REF] Polynor | The hand that rocks the cradle[END_REF], database queries [START_REF] Ullmer | Tangible query interfaces: Physically constrained tokens for manipulating database queries[END_REF], visualization [Huron et al., 2014a,c] or games [Activision Publishing Inc., 2013]. Tangible tokens can be used as input devices, and occasionally as output devices through self-actuation [START_REF] Esben | Tangible bots: interaction with active tangibles in tabletop interfaces[END_REF]. Both input and output yield major technological challenges, and in this chapter we focus on input. A major challenge for interacting with these objects is to carefully capture the different dimensions of the object manipulation. Many strategies have been used to support input, and they generally fall in two categories: external sensing and internal sensing. Both are discussed to then motivate the SmartToken design.

External Sensing

External sensing involves the use of sensors that are outside the tangible objects to detect and track information on these objects. Different types of sensing technologies have been used to capture various aspects. A common approach is to use a camera and visual markers to track position and orientation [START_REF] Kaltenbrunner | reactivision: a computer-vision framework for table-based tangible interaction[END_REF][START_REF] Weiss | Slap widgets: bridging the gap between virtual and physical controls on tabletops[END_REF]. A related approach is the use of depth cameras [START_REF] Andrew D Wilson | Depth-sensing video cameras for 3d tangible tabletop interaction[END_REF]. Other spatial tracking technologies include capacitive sensing on touch surfaces [START_REF] Jansen | Tangible remote controllers for wall-size displays[END_REF] and electromagnetic sensing [START_REF] Liang | Gaussbits: magnetic tangible bits for portable and occlusion-free near-surface interactions[END_REF]. Finally, RFID tags [START_REF] Joseph A Paradiso | Tangible music interfaces using passive magnetic tags[END_REF] can be used to detect the nearby presence of tangible objects. Henecke et al. [START_REF] Hennecke | Optical pressure sensing for tangible user interfaces[END_REF] as well as Willis et al. [START_REF] Willis | Printed optics: 3d printing of embedded optical elements for interactive devices[END_REF] have developed optical techniques to capture object touch and pressure ontop an interactive surface.

External sensing solutions are able to provide position and sometimes orientation information on tangible objects of arbitrary form factor, including tangible tokens. However, external sensing systems are typically bulky and complex to set up, and often operate under controlled environmental conditions. The interaction space is thus tied to the location of the sensing equipment, which prevents tangible tokens to be seamlessly integrated with our everyday environments [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF]. Moreover, external devices cannot easily sense manipulative information such as touch or grip. Therefore, internal sensing has many potential benefits over external sensing for TUIs.

Internal Sensing

Internal sensing refers to systems that embed sensors within the tangible objects themselves. Many types of electronic sensors can be employed, e.g., light, sound, kinetic, accelerometer, gyroscope, and touch sensors. Systems can be classified according to what information they sense and how they communicate this information. Some systems are entirely self-contained. For instance, "kinetic memory" systems like Curlybot [START_REF] Frei | Curlybot: designing a new class of computational toys[END_REF] and Topobo [START_REF] Hayes Solos Raffle | Topobo: a constructive assembly system with kinetic memory[END_REF] are made of physical building blocks that can record and replay manipulations. Another example is the "cube to learn" [START_REF] Terrenghi | A cube to learn: a tangible user interface for the design of a learning appliance[END_REF], whose building blocks are screens with acceleration sensors. Although stand-alone tangible systems can support specific applications remarkably well, they cannot be used as input devices to prototype TUIs since they cannot communicate with external systems. Furthermore, we are not aware of any self-contained system that is made of tokens, besides Bishop's marble answering machine concept [START_REF] Polynor | The hand that rocks the cradle[END_REF].

A more recent project [START_REF] Jacobson | Tangible and modular input device for character articulation[END_REF] has used these sensor assemblies for real-time animation.

An early attempt to build self-sensing objects with communication capacities was Electronic Blocks [START_REF] Wyeth | Electronic blocks: Tangible programming elements for preschoolers[END_REF], a children game to build "computer programs" made out of LEGO bricks with touch, light, and sound sensors, and that can intercommunicate once physically connected. Later [START_REF] Merrill | Siftables: towards sensor network user interfaces[END_REF] coined the concept of Sensor Network User Interface and introduced Siftables. Siftables combine a small LCD screen and an accelerometer with wireless communication support. They support twodimensional topology recognition through contact sensing. Sifteo Cubes [Sifteo] are a commercial extension with additional capacitive sensing capabilities. Recently, [START_REF] Hosoi | A-blocks: recognizing and assessing child building processes during play with toy blocks[END_REF] introduced A-Blocks, which support threedimensional topology recognition through geomagnetism sensors. All these devices have a rather small form factor and rich internal sensing capabilities, but mostly focus on physical assembly scenarios. For example, they do not sense grip. Bar of Soap [Taylor and Bove, 2008] is, to our knowledge, the unique attempt to combine grip and motion sensing in a mobile device. The device is able to sense orientation and grip. However, it remains bulky and is meant to be used as a single object.

Besides specialized TUI devices, off-the-shelf generic-purpose mobile devices such as smartphones have also been used to prototype TUIs [START_REF] Klum | Stackables: combining tangibles for faceted browsing[END_REF]. Smartphones provide highly elaborate sensing, computation and communication capabilities, but at the cost of price and form factor. Finally, internal and external sensing approaches can be combined. An example of a hybrid strategy is Touchbug [START_REF] Nowacka | Touchbugs: Actuated tangibles on multi-touch tables[END_REF], a tangible object with embedded accelerometer and gyroscope that detects gestures like shake and tip. Touchbugs are coupled with a multi-touch tabletop for sensing their absolute positions.

Motivation for SmartTokens

As shown in Chapter 3, we observed and analyzed multiple video recordings of the participants of our study. Trying to discern all the fine details of the manipulations was particularly difficult, especially when distinguishing how each object was manipulated or how many were manipulated. Some participants were very dexterous and could perform very complex manipulations, involving simultaneously multiple fingers and handling multiple objects with both hands. Parts of the videos had to be analyzed image by image, in order to capture the finesse of the manipulations. As this task ended up being especially long and tedious, it appeared logical to augment the objects to perform this task automatically. The annotations could then be done more accurately and reliably. We thus started to investigate how to make objects more aware of the manipulations they were undergoing.

The previous section showed that many technologies are available to prototype or implement TUIs, each with a different trade-off. A diversity of choices is important for TUI researchers to be able to explore a large array of designs. Internal sensing is receiving more and more attention, and several devices exist that can sense topology, motion, and touch. Yet, form factor remains a bottleneck, as no device exists that has the width of a coin and can support even basic sensing. This limits researchers' access to the range of TUIs that can be prototyped.

Tokens are ubiquitous in our everyday lives (e.g., monetary coins, bus tokens, game pieces) and their form factor has been refined over thousands of years of technological evolutions. Tokens provide an answer to several realworld problems: first, they are easy to transport and easy to store in small containers such as pockets and wallets. They also resist wear and shocks (e.g., when we drop them). Finally, tokens are easy to manipulate (including in-hand) and to hand over, both as individual items and as collections.

Our goal is to provide coin-sized tangible and graspable objects that are smart enough to be used as handles for digital information, at a low cost. We believe SmartTokens can open up new design opportunities for TUIs.

smarttoken design and implementation

In this section we detail the implementation of SmartTokens and the underlying design rationale.

Form Factor and Structure

The SmartToken's shell is a 3D-printed cylinder of 28 mm diameter by 12 mm height (Figure 4.2 1 ). This form factor is comparable to common passive tokens as discussed previously, and is generic enough so it does not carry a specific meaning. A SmartToken is large enough to be easily grasped, while being small enough to allow for group manipulation (about five tokens fit comfortably in a hand). This size lets a user to grasp or manipulate many of them at once. It is made of two 3D printed matching parts, 2 and 3 in Figure 4.2. SmartTokens occupy almost half the volume of Siftables [START_REF] Merrill | Siftables: towards sensor network user interfaces[END_REF] and 5 to 6 times less than Sifteo cubes [Sifteo] (7.4 cm 3 vs. 13.0 cm 3 vs. 42 cm 3 ).

All components of SmartTokens fit on a custom circuit board measuring 22 mm in diameter by 2.3 mm high (Figure 4.2 2 ). The electronics design is Each SmartToken is powered by a 70mAh Lithium-Polymer battery ( 3c in Figure 4.2) supplying enough energy for approximatively three hours of intensive use. This battery takes up most of the space. Batteries are recharged using a custom charger, where the battery block ( 3 in Figure 4.2) of the SmartToken can be inserted.

SmartTokens are relatively cost-effective: when produced in small quantities, components cost about $20 in total, the circuit board costs about $7 to manufacture, for a total assembly time of about two hours.

Sensing

SmartTokens contain touch sensors and a 3D accelerometer and gyroscope. To ensure a good sensing reactivity, both touch and acceleration sensors are read at a rate of 100Hz. Changes in grip and motion are then used to update an internal state-machine described in the next section.

Touch

We established that, for a SmartToken to be able to distinguish the grips of gestures described in Section 3.7.1, it had to be able to detect touch on each of its faces.

We embedded a capacitive touch sensor (Atmel AT42QT1070) in each SmartToken, monitoring six electrodes distributed on the inner faces of the SmartToken's shell (pink areas in Figure 4.2).

To provide the best touch sensitivity, the size of the touch electrodes has to be as large as possible. However, enlarging them excessively would block wireless communication. We found that using rim electrodes of size 10 × 7 mm (10 × 10 mm for the upper and lower electrodes) and spacing them apart by 9 mm worked well in our case. Electrodes are connected to the PCB through thin wires. The touch state of the six electrodes is stored in a byte by the microcontroller and events are delivered to our state-machine when the state of the electrodes changes.

While we opted for a cylindric form factor, the general design remains flexible and can be reshaped in various ways. One can easily imagine a spherical shell instead of a cylindric one, if the token has to roll. The touch electrodes would need to be evenly mapped on the inner surface of the sphere. Six electrodes should be enough to detect grip gestures with a range of shell shape and size, but an electrode spacing of about 10 mm should be preserved to avoid touch blind spots.

Motion

Motion is monitored with a six axes inertial measurement unit (ST LSM6DS0). Data is delivered as three linear accelerations (angular rates are ignored in the current implementation). Since the accelerometer output includes the gravity (g) component, we estimate g through a calibration phase at startup (while the token is still) and then substract it from all subsequent measurements. Thus the output at rest is 0.

Our state machine distinguishes between immobility, motion and free fall by monitoring changes in acceleration and touch events. Motion is inferred when the acceleration exceeds 25 % of g for more than 100 ms, and at least one touch contact is found. When a token falls, the expected output is |0 -g| = g. Thus free fall is inferred when no touch is registered and acceleration exceeds 80 % of g during at least 20 ms.

Communication

For communication, we use IEEE 802.15.4 ultra low-power wireless communication technology [START_REF] Molisch | Ieee 802.15. 4a channel model-final report[END_REF]. The network is created and managed by a coordinator embodied by a USB dongle (see Figure 4.4) and connected to a host computer. The dongle receives state-change messages from all SmartTokens. Each message also contains the state of the touch sensors. On the host computer, a C# server application translates all incoming messages into higher-level events and streams them via a TCP socket to client applications.

Concerning scalability, the IEEE 802.15.4 standard identifies each node with a 16 bits address, so up to 65535 nodes can be used at the same time in theory. Raw signal is processed within the SmartTokens themselves, minimizing the amount of information to transmit. A state transition is encoded in 4 bytes, for a total of 19 bytes per packet: the used bandwidth is thus much lower than the network's bandwidth (250 kbit/s), thus allowing multiple SmartTokens to transmit almost simultaneously. The operating range of an IEEE 802.15.4 network is about 10 meters, which should be enough for most office applications such as our use-case scenario. However, mobile or pervasive computing scenarios would require multiple distributed coordinators, which is possible using the IEEE 802.15.4 standard.

the smarttoken input model

Following Buxton [START_REF] Buxton | A three-state model of graphical input[END_REF], we model the input event vocabulary of SmartTokens as a state-machine (see Figure 4.5). Each SmartToken senses i) whether or not it is touched by the user and ii) whether it is moving or still.

The cross-product of these two state-machines yields a state-machine with four different states (dark grey rectangles in Figure 4.5). Transitions between the states correspond to different manipulative actions (arrows in Figure 4.5).

Eight different transitions are possible, plus four more transitions (not illustrated) if we assume that touch and motion transitions can happen concurrently. Thus monitoring two binary states already provides a rich repertoire of actions. In addition, SmartTokens also sense grip, i.e., whether the user touches the object on one side or more (see Figure 4.3).

use case: tangible notification machine

To illustrate how SmartTokens can be used in practice, we present an application scenario that consists in a tangible notification and task management interface inspired from Durrell Bishop's marble answering machine [START_REF] Polynor | The hand that rocks the cradle[END_REF], detailed in Section 2.1.1.

Walkthrough

The notification machine consists in a set of SmartToken dispensers that communicate with a personal computer (see Figure 4.7). When an event of interest is received on the computer (e.g., a new calendar event, a bug report or email), a token is released. Different dispensers represent different event types, and SmartTokens accumulate in their respective containers. Tokens can also come in different forms to convey different types of notifications through color, shape or material. SmartTokens can be picked up anytime to display event details on the computer screen or to update events' status.

For example, suppose Laura is a software engineer who actively maintains a blog and an open-source software project. She wants to be notified when someone comments on her blog or posts a bug report on her software. She also wants to be notified of any new email, but does not want to be constantly interacting with her email client and prefers to keep her screen free of any distraction. In her fablab she constructed her own notification machine made of three dispensers (see Figure 4 This short interruption prompts Laura to peek at her new emails. She brushes the red tokens still on his desk (see 6 in Figure 4.7), and sees the senders and subjects from all three emails on his screen. One of them is a reminder asking for a prompt reply. She places it in front of his keyboard with a post-it note to make sure she does not forget about it, puts the other back in the dispenser, and gets back to work.

Later, Laura is about to leave when she realizes her email token is still waiting to be dealt with. She pinches it to display the message body again, then decides she will reply from home later tonight. She grabs it and encloses it with her hand while shaking to mark the message as important, and a red star appears next to the message. She puts the token back on the desk and leaves.

Novelty and Benefits

As tangible output devices, token dispensers let users adapt the level of notification by altering their physical arrangement, geometry, and material properties (e.g., a dispenser for important events can be suspended high above a metal container). The level of notification can be adapted over time to reflect evolving task demands. This is in contrast with desktop notification systems which generally employ subtle visual and/or auditory messages but are not flexible enough to account for different levels of notification. Moreover, such visual cues, reminders showing up on the screen for instance, can happen to be distracting when appearing directly in the field of view of the user.

Desktop notification systems also typically deliver short cues, calling for a prompt response. The notification machine makes it easier to delay response, as notification items piled up physically, waiting to be taken care of. Notifications can be ignored or missed, yet the constant presence of tokens remind the user of the presence of pending notifications.

Compared to desktop tools, the notification machine offers a different way of managing notifications and tasks: i) since SmartTokens are tangible, users can arrange them in a way that is meaningful to them (by improvising a Kanban Board [START_REF] Dragicevic | Kanban boards -list of physical visualizations[END_REF]] on a desk for example) ; ii) since SmartTokens are capable of sensing, they support a range of additional tasks: notification items can be previewed, inspected, and their status changed and reflected back on the computer. Furthermore, the notification machine provides richer output and input than typical peripheral and ambient devices (e.g., [START_REF] Jafarinaimi | Breakaway: an ambient display designed to change human behavior[END_REF]) and offers a different way of interacting with tangible objects compared to classical TUIs. Both the marble answering machine concept [START_REF] Polynor | The hand that rocks the cradle[END_REF] and RFID-based TUI systems [START_REF] Joseph A Paradiso | Tangible music interfaces using passive magnetic tags[END_REF] assign predefined semantics to specific containers or specific spatial locations. In contrast, Smart-Tokens only care about what happens to them, which allows users to assign their own semantics to locations and containers, and leaves more room for the opportunistic use of space and creativity. This is further facilitated by the fact that no heavy sensing surface or equipment is needed for SmartTokens to operate.

Assuming that in the future most computers and mobile devices could be equipped to communicate with SmartTokens, it is further possible to imagine scenarios where opportunistic uses extend beyond the desktop and where notification items are carried around, put in one's pocket, and passed on from people to people like coins [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF]]. While we focused on input, nothing in principle prevents SmartTokens from supporting basic output, and a SmartToken connected to a smartphone with GPS could for example vibrate in a user's pocket to remind her to drop by the bakery to buy a baguette.

Notification Machine Implementation

Assuming the SmartToken framework is provided, implementing the notification machine is relatively effortless and accessible to anyone with basic digital prototyping and programming skills.

Each token dispenser is made of wooden laser-cut parts. SmartTokens are stored vertically in the dispenser, and a servomotor arm delivers them one by one. These then fall down a slide. Each dispenser is controlled by a simple Arduino board and connected to the host computer via USB.

The notification manager is a Java program with a GUI to connect notification sources to notification sinks. Each token dispenser is a sink. The driver currently implements a single notification source factory that connects to an email account through the IMAP protocol and lists each folder as a notification source. Many web applications with notification support can be configured to send emails automatically, thus this solution is quite versatile. The library is extensible and can be connected to other notification sources.

The code for associating tokens with notification events and for supporting user interaction with tokens is written on top of i) the notification manager framework and ii) a Java API for registering to SmartToken input events from the state-machine. The events are used as follows:

Initially, when all SmartTokens are stored in a dispenser, they are in state 1 (see Figure 4.5). When a notification is received and a dispenser releases a SmartToken, the token switches to state 4 and a fall event is sent. The notification manager receives this event and maps the ID of the token to the ID of the notification event just received. Then the token moves back to state 1 after the fall. As soon as Laura touches it, the token transitions to state 2 and emits a touch event. The notification's title is then displayed on the screen (email sender and subject) until a release event is received. When Laura picks up a token: a touch event is sent, shortly followed by a grip change event indicating two contact points; the notification details (email body) is then displayed. In case the token was picked up for the first time (i.e., from a container), the move event that follows tags the notification event as read. Enclosing the token inside one's hand triggers a grip change event on 3 to 6 contact points, which adds the tag important. Finally, when Laura drops a token back in its dispenser, the token moves to state 4 as it falls and finally ends in to state 1 . The fall event removes the notification ID from the token.

limitations and future work

This section steps back, observing how SmartTokens could be improved and different directions they could take.

Improving SmartTokens

Extending sensing and performances

We wanted SmartTokens to be able to detect the gestures from the typology introduced in Section 3.7.1. We chose to use capacitive touch technology to recognize the different grips as it provides robust sensing while remaining easy to integrate. However, capacitive touch sensing only supports the detection of human skin contact, and not contacts between SmartTokens. Hence, SmartTokens cannot detect manipulations when moved using a tool, for instance a plastic ruler or another SmartToken. As a result, the SmartTokens undergoing a Bulldozer Push that are not in direct contact with the user cannot recognize the manipulation.

While SmartTokens embed a powerful motion sensor, it is only used to detect freefalls and users' shake gestures. Yet, this implementation is far from exploiting the full potential of the orientation, acceleration and rotation data. Improving this data processing will enable the detection of new interactions such as: stacking, flip, roll between fingers, etc. All these interactions could be used to new functional mapping. For instance one can imagine using a token as a knob to control remotely the sound level in a room, and flip it to change sound track.

Furthermore, combining SmartTokens with external sensing can also provide interesting possibilities, among which the detection of absolute position. Dead reckoning techniques [START_REF] Fuke | Dead reckoning for a lunar rover on uneven terrain[END_REF]] could be used to disambiguate external information on absolute position when occlusion is present, for example when multiple tokens are manipulated in-hand. Similarly, localization based on the Received Signal Strength Indicator [START_REF] Goldoni | Experimental analysis of rssi-based indoor localization with ieee 802.15. 4[END_REF] would only use the already present wireless communication.

Another interesting direction to augment the device is to add additional sensors that could capture: the tree dimensional topology, the proximity between tokens, the sound, light conditions.

Power consumption is an aspect we did not consider in our first prototype, but day-or even week-long battery life can be achieved using software strategies. For instance, a SmartToken can fall into deep sleep when no activity is detected for a given period of time. Similarly, a reduced sensor sampling rate can be used when the user is not interacting with the SmartTokens.

Other form factors

Our current electronics design can in principle accommodate other form factors. However some work is required to adapt the design to various form factors including marbles, assembly building blocks, and games tokens. We are currently working on a spherical version to build marble machine TUIs. Pushing miniaturization further would also give access to interesting TUI designs where large collections of small physical elements can be manipulated. Moreover, we expect that future research in battery component and electronic will help us to reduce SmartToken size. The ideal will be to reduce the thickness of our component to be able to embed it in the size of a coin.

Output

One obvious limitation of the current SmartTokens is the lack of embedded output. Our future work includes augmenting the SmartToken design with output channels (vibrators and LEDs) and with robotic actuation.

Extending the SmartTokens Input Model

Although the sensing support in SmartTokens is simple, the use of touch sensors on all of tokens' faces allows to capture a new set of human interactions, including grips. The presented initial input model allows applications to listen to basic input events related to motion, touch and grip. This model can be extended to support concurrent transitions or sequences of transitions. Turning the state machine into a Petri net would further support complex input events involving multiple tokens.

Other application scenarios

The notification machine scenario presented is one possible application, however SmartToken as an open-source platform can be seen for future tangible user interface applications and research projects. We are considering using SmartTokens to facilitate the logging and analysis of interaction with tangible interfaces in empirical studies. Indeed, SmartToken could help to better understand manipulations of collections of small objects. Despite the fact human hands are using small object that people manipulate on a daily basis, the knowledge in this domain is still limited. SmartToken could be used to gather precise information on manipulations activity.

conclusion

We presented SmartTokens, small-sized tangible tokens that can sense multiple types of motions and various types of grip, and can wirelessly send input events modeled as state-machine transitions. SmartTokens extend the design space of TUIs by providing a cost-effective and versatile open-source platform for developing token-based tangible interfaces. All necessary material and documentation for implementing SmartTokens can be found at http://mathieulegoc.github.io/SmartTokens/.

The token form-factor makes it possible to map multiple kinds digital information such as notifications or emails to individual objects that can persist in our physical environment. Users can manipulate up to about 12 of these objects simultaneously; these objects can be stored or used in machines; they can be exchanged between people and carried around (e.g.,, in one's pocket) across different physical environments. All of these features are difficult or impractical to support with previously proposed TUI devices, which are bulkier and meant for different usage scenarios.

SmartTokens are the only system we know of that combines a small form factor with grip detection, allowing to detect grip even when several tokens are manipulated at the same time. This new combinition of embedded grip and motion sensing into a small form factor opens up new creative opportunities for TUIs.

Currently, SmartTokens provide six touch sensors and a 3D accelerometer. They synthesize three motion states: immobility, motion, and free fall.

The touch sensors allow discriminating between simple touch, grip, and fullhand hold; therefore, SmartTokens can be used to implement a variety of manipulative gestures applied to a single or to multiple SmartTokens.

We illustrated SmartTokens' capabilities by revisiting Bishop's marble answering machine [START_REF] Polynor | The hand that rocks the cradle[END_REF], but the token form-factor is also suitable for a range of other applications, such as board games or physical data storytelling 1 . 5

I N T R O D U C I N G S WA R M U I S , V E R S AT I L E A N D C O M P O S I T E T U I S
The previous chapter presented how small, generic objects can be augmented with sensing capabilities to support user input. This chapter takes another important step towards goals presented in Chapter 1, that is to create versatile composite TUIs.

This chapter presents

Zooids, an open-source open hardware TUI platform relying on collections of small, interactive and actuated generic objects. By providing both input and output capabilities, Zooids remove the need for substitution media such as screen and projection, or mid-air input. This chapter also generalizes the approach by introducing Swarm User Interfaces, a new declination of TUIs comprised of many autonomous robots that handle both display and interaction. This chapter illustrates the potential of Swarm UIs through a set of application scenarios developed with Zooids, and describes their implementation and design. This chapter finally discusses general design considerations unique to Swarm User Interfaces.

Main portions of this chapter were previously published in [Le Goc et al., 2016b]. Thus any use of "we" in this chapter refers to myself, Lawrence Kim, Ali Parsaei, Jean-Daniel Fekete, Pierre Dragicevic and Sean Follmer.

introduction

Despite the many benefits of TUIs [START_REF] Ishii | Tangible bits: towards seamless interfaces between people, bits and atoms[END_REF][START_REF] Shaer | Tangible user interfaces: past, present, and future directions[END_REF] and the technological achievements made in HCI so far, we are still far from Ivan Sutherland's vision of the Ultimate Display, or Hiroshi Ishii's vision of Radical Atoms. For the lack of such enabling technologies, we still have a long way to go before we can fully combine the unique human capabilities for physical manipulation with the power of computing. Several significant steps have been recently made towards Sutherland's and Ishii's visions, particularly through research on actuated tangibles and shape displays as covered in Section 2.3. However, current systems suffer from a number of limitations. First, actuated tabletop tangibles generally only support the manipulation and actuation of a few (e.g., 3-4) solid objects, which is not enough to emulate physical matter that can change form. On the other hand, shape displays try to achieve surfaces that can be deformed and actuated, but current implementations do not support arbitrary physical topologies.

Furthermore, both types of systems traditionally use physical objects primarily as input, while output is almost always provided through separate pixel-based display technology. Although video-projected overlays allows input and output to spatially coincide [START_REF] Kenneth P Fishkin | A taxonomy for and analysis of tangible interfaces[END_REF], they provide only a limited sense of physicality [START_REF] Bennett | The effect that touching a projection augmented model has on object-presence[END_REF]. Likewise, many such systems require heavy hardware or displays to function, and are thus primarily meant to be operated in sterile environments rather than embedded in our own physical world [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF][START_REF] Weiser | Some computer science issues in ubiquitous computing[END_REF].

Robinson et al. [2016]

Our research work fills this current gap in user interface technologies by introducing Zooids and Swarm User Interfaces. A Zooid is a hardware and software system: a small wheel-propelled robot with position and touch sensing capabilities that can be freely arranged and repositioned on any horizontal surface, both through user manipulation and computer control. Zooids extend previoulsy presented SmartTokens (see Chapter 4). Building on lessons learned during the development of SmartTokens, Zooids retain almost the same form factor, while using a similar communication network architecture as well as the same technologies for users' manipulation sensing.

A Zooid is defined in Wikipedia as "a single animal that is part of a colonial animal. Zooids are multicellular; their structure is similar to that of other solitary animals."

Zooids build on work from swarm robotics [START_REF] Dudek | A taxonomy for swarm robots[END_REF][START_REF] Snape | Independent navigation of multiple mobile robots with hybrid reciprocal velocity obstacles[END_REF], adding interaction and speed. Swarm User Interfaces (Swarm UIs) are interfaces built using collections of self-propelled physical objects (e.g., mini robots) that can move collectively and react to user input. Swarm UIs can be seen as a coarse-grained version of Sutherland's and Ishii's futuristic visions of user interfaces based on programmable matter.

Due to Zooids' ability to freely and quickly reconfigure themselves spatially, a collection of Zooids can act as a display and can provide meaningful user output. Their ability to sense user actions allow Zooids to support rich input. For example, users can either move Zooids one by one, or manipulate many Zooids at once using "sweeping" gestures [Le Goc et al., 2016a].

Sophisticated interactive behaviors can be implemented on the application side, e.g., Zooids can act as controls or as handles for manipulating others Zooids; they can even move other light objects. At the same time, since all input and output can be mediated through the same physical elements, the system is able to achieve a complete fusion between input and output and provide a complete experience of physical manipulation. Finally, the system is relatively lightweight and only requires the use of a compact DLP projector (122 mm × 115 mm × 48 mm) for optical tracking. Zooids can operate on any horizontal surface (e.g., a sheet of paper, a messy office desk, a dining table, or a game board), making it possible to blend Swarm UIs with everyday physical environments. To stimulate future research on swarm user interfaces, we distribute our Zooids tabletop Swarm The first open-source hardware/software platform for experimenting with tabletop Swarm User Interfaces, A set of scenarios to illustrate the unprecedented possibilities offered by our system and by tabletop Swarm User Interfaces in general, A discussion of some general design principles and design challenges for Swarm User Interfaces. Furthermore, as benefits, Zooids:

• can coexist in large numbers, in comparison to previous actuated tangible user interfaces,

• can act as individual objects, while being small enough to also act as "pixels" of a physical display,

• can be manipulated either individually or collectively, including with physical tools such as rulers,

• are lightweight, can operate on any horizontal surface, and relatively cost-effective: about 50 USD each now, down to $20 if mass manufactured.

background

These research efforts are partly inspired by visions such as Sutherland's and Ishii's as discussed earlier in Section 2.3.3, where computers would be able to reconfigure physical matter to recreate any physical shape.

Other fields such as robotics and material science have been interested in realizing this dream of "programmable matter", but most of the progress so far has been purely theoretical [Goldstein et al., 2005;[START_REF] Rus | Programmable matter with self-reconfiguring robots[END_REF]. This work is related to several research areas: tabletop tangible user interfaces and shape displays are reviewed respectively in Section 2.3.1 and Section 2.3.1, swarm robotics are detailed here.

Swarm Robotics

Swarm robots draw from natural swarms, where social animals such as birds or ants can produce complex collective behavior by moving and interacting with each other according to simple rules. The largest robotic swarm implemented so far involves as many as 1,000 robots although they move slowly (about 1 cm/s vs. ~50 cm/s for Zooids) [START_REF] Michael Rubenstein | Kilobot: A low cost scalable robot system for collective behaviors[END_REF]. Our paper is inspired from past research in swarm robotics [START_REF] Ducatelle | Self-organized cooperation between robotic swarms[END_REF][START_REF] Dudek | A taxonomy for swarm robots[END_REF], but while the area of swarm robotics has been mostly interested in how to emulate swarm behavior using distributed intelligence and fully autonomous agents, we focus on direct physical interaction with small swarm robots, HCI applications, and employ a centralized system to coordinate robots.

Researchers in robotics have started to develop methods for interacting with swarm robots, but most of them have only been tested on mouseoperated computer simulations [START_REF] Kira | Exerting human control over decentralized robot swarms[END_REF][START_REF] Kolling | Towards human control of robot swarms[END_REF]. [START_REF] Alonso-Mora | Multi-robot system for artistic pattern formation[END_REF] investigated the use of swarm robots as physical displays and recently extended their system to support interaction through sketching [START_REF] Hauri | Multi-robot formation control via a real-time drawing interface[END_REF], hand-held tablet input [START_REF] Grieder | Multi-robot control and interaction with a hand-held tablet[END_REF] and mid-air gestures [START_REF] Alonso-Mora | Gesture based human-multi-robot swarm interaction and its application to an interactive display[END_REF]. Their systems share many features with Zooids, but we focuse instead on direct tangible manipulation of swarm robots and explore a wider range of application scenarios.

As mentioned in Section 2.3.3.2, [START_REF] Rubens | Bitdrones: Towards levitating programmable matter using interactive 3d quadcopter displays[END_REF] described a mid-air 3D physical display system based on drones, with which users can interact by directly manipulating drones. Despite their goal of ultimately converging towards a swarm user interface, each drone is rather large (8.9 cm) and the number of drones that can be simultaneously used is limited due to turbulence issues -their prototype currently consists of 3 drones. The Drone 100 [Futurelab, 2016] project involves a hundred drones called "spaxels". Each is light-equipped and can be positioned in three dimensions, resulting in a choreographed swarm capable of displaying dynamic images. However, the large operating volume prevents direct manipulation.

Introducing Swarm User Interfaces

We propose to refer to swarm user interfaces (swarm UIs) as:

"human-computer interfaces made of independent self-propelled elements that move collectively and react to user input". Independent: the user interface elements need to be physically detached from each other and free to move. Counter-examples include graphical elements on a computer display, which are all part of a single physical object. Articulated models, 2.5D shape displays [START_REF] Follmer | inform: dynamic physical affordances and constraints through shape and object actuation[END_REF] and physical control panels such as mixing consoles also do not qualify, since the moving parts and controls are attached and not free to move.

Self-propelled: the elements need to be able to move without external forces. Counter-examples include passive physical tokens [Huron et al., 2014b;Le Goc et al., 2015a].

Move collectively: by definition, swarming behavior involves collective motion. Thus the elements need to be able to move in a coordinated fashion, either by exchanging information with each other or with a centralized coordinator. In addition, the more elements a user interface contains, the more their motion can be qualified as collective, and thus the more "swarm-like" the interface is.

React to user input: the elements need to sense user input and react to this input. Thus, most swarm robotics systems are not swarm user interfaces, because they lack the user interaction component. A swarm display that is interactive but only takes user input from external sources -e.g., a mouse or a keyboard -is not a swarm user interface either according to our definition, because the elements themselves need to be able to react to user input. Systems that use computer vision to detect mid-air gestures such as DisplaySwarm [START_REF] Alonso-Mora | Gesture based human-multi-robot swarm interaction and its application to an interactive display[END_REF] are in a gray area. For smooth interaction, speed of the system is critical: the elements of a swarm UI need to be fast enough for shape change to occur at a usable rate. The ideal transition time is in the order of one second, because this is about the limit where a system is perceived as interactive [START_REF] Nielsen | Usability engineering[END_REF], and it is also the recommended duration for animated transitions on regular graphical displays [START_REF] Heer | Animated transitions in statistical data graphics[END_REF].

From the partial taxonomy presented in Section 2.4.2, Swarm UIs are thus purely physical as they have to support both input and output. They are also composite as made of constituted of a multitude of small robots, heterogeneous or homogeneous according to the design of the robots, and actuated. The systems that come closest to Swarm User Interfaces according are selfpropelled tangibles [START_REF] Kojima | Augmented coliseum: An augmented game environment with small vehicles[END_REF][START_REF] Mi | Hats: interact using height-adjustable tangibles in tabletop interfaces[END_REF][START_REF] Nowacka | Touchbugs: Actuated tangibles on multi-touch tables[END_REF]Patten Studio, 2014;[START_REF] Esben | Tangible bots: interaction with active tangibles in tabletop interfaces[END_REF][START_REF] Richter | Remote active tangible interactions[END_REF][START_REF] Rosenfeld | Physical objects as bidirectional user interface elements[END_REF] and BitDrones [START_REF] Rubens | Bitdrones: Towards levitating programmable matter using interactive 3d quadcopter displays[END_REF], because they are made of independent self-propelled elements that can move in a coordinated fashion and can be directly manipulated. However, these systems involve few elements (i.e., around 4-5), and are thus at best low-fidelity prototypes of actual swarm user interfaces. While many such systems could have involved more units, a small form factor (e.g., Zooids are more than three times smaller than [START_REF] Rosenfeld | Physical objects as bidirectional user interface elements[END_REF] robots) enables different types of interactions. Users can manipulate many Zooids at once, while several dozens of larger robots may not even fit on a regular table. Moreover, previous work does not discuss or demonstrate swarm user interfaces, which are our focus.

In principle, swarm user interfaces could take many forms and could be implemented in many different ways. For example, a swarm UI can consist of free-floating particles [START_REF] Sue | Dexterous ultrasonic levitation of millimeter-sized objects in air[END_REF] or drones [START_REF] Rubens | Bitdrones: Towards levitating programmable matter using interactive 3d quadcopter displays[END_REF] that are free to move in 3D space, or can consist of objects that evolve on a 2D surface [START_REF] Pangaro | The actuated workbench: computer-controlled actuation in tabletop tangible interfaces[END_REF]. In this chapter, we focus on elements that move on a 2D surface, i.e., tabletop swarm user interfaces. We illustrate the possibilities offered by tabletop swarm interfaces through examples using Zooids, which implementation is then detailed. We finally discuss design principles and challenges for swarm UIs.

swarm ui examples with zooids

In this section we illustrate and discuss possibilities Zooids offer through simple use cases and scenarios, before explaining their hardware and software design.

Swarm Drawing

Freehand Drawing

Inspired from vector graphics authoring tools, we have implemented a swarm version of a freehand drawing tool, shown in Figure 5.2: initially, the freehand drawing Zooid stands in the center of the working surface, while unassigned Zooids wait at the top, in an idle state (Figure 5.2-1 ). When the user drags the freehand drawing Zooid, the previously idle Zooids move to the path of the drawing Zooid to form a physical trail (Figure 5.2-2 and 3 ). When the system runs out of idle Zooids, the trail follows the freehand drawing tool like a snake. The curve can also be deformed by dragging its constituent Zooids individually (Figure 5.2-4 ), or by moving many of them simultaneously, e.g., by pushing them with the side of the arm.

Shapes

We also experimented with tools for drawing lines, rectangles and circles, based on the standard rubber band technique from desktop applications. Each of these tools employs two Zooids as control points. Figure 5.3 shows the example of a circle tool, where two control points are used to define the circle's diameter, and idle Zooids are automatically positioned to complete the circular shape. Zooids are also automatically added or removed depending on how many of them are necessary to construct the shape. Another Zooid at the bottom of the table (not shown) allows users to switch between shapes.

Bézier Curves

In traditional vector drawing editing tools, Bézier curves allow for accurate shaping using discrete control points. We developed a physical curve editing tool where a collection of Zooids are positioned to represent a curve. While shaping using the previously introduced drawing tool requires to manipulate many Zooids at once, this tool uses specific Zooids as control points to adjust the curvature represented by the collection. Each control point consists of two Zooids, where one sets the anchor point and the other adjusts the tangent.

It is important to note that although GUIs currently support far higher information resolution, Zooids enable richer physical gestures. We believe that technology advances will allow higher resolution of swarm UIs in the future.

Stop Motion Swarm Animation

Inspired by traditional stop motion animation tools, we implemented a tool enabling users to author physical animations. The user positions each Zooid to form the desired layout. Moving the timeline Zooid a step forward saves the current layout as a key frame. Once the desired layouts have been recorded, toggling the second control Zooid switches the mode to playback and plays consecutively the different keyframes.

In-the-Wild Scenarios

Although we have not implemented specific applications, we have begun to experiment with in-the-wild scenarios, in which Zooids could be embedded with real-world environments. For example, they could be placed on a user's working desk to act as ambient displays (e.g., to show progress in downloads), extra controls, or as notification devices (e.g., they could hit a metallic or glass object when an important event starts or to remind you to drink water). Enough Zooids can even move objects such as smartphones.

zooids

Elaborating from the examples of uses of Zooids just presented, this section explains their hardware and software design.

Hardware

Robot Design

Zooids are small custom-made robots as shown in Figure 5.4; their dimensions are 26 mm in diameter, 21 mm in height and they weight about 12 g. Each robot is powered by a 100 mAh LiPo battery and uses motor driven wheels. The motors are placed non-colinearly to minimize the diameter. Even though the motors do not rotate around the same axis, the robot has the same net force and moment as would a robot with colinear motors. To drive the robot, a motor driver chip (Allegro A3901) and two micro motors (FA-GM6-3V-25) are used. With this combination, the robot has a maximum speed of approximately 74 cm/s. However, for controllability and smoothness of the motion, the robots move at a slower average speed of 44 cm/s for our applications.

A flexible electrode is wrapped inside the 3D printed enclosure to provide capacitive touch sensing capabilities. An integrated capacitive touch sensing circuit is included (Atmel AT42QT1070) to detect user's touch.

Embedded custom electronics, shown in the PCB layer of Figure 5.4, allows for robot control. A 48MHz ARM micro-controller (STMicroelectronics STM32F051C8) manages the overall logic computation and communicates wirelessly with the main master computer using a 2.4GHz radio chip (Nordic nRF24L01+). As part of the projector-based tracking system (explained in the next section), two photodiodes are placed at the top of the robot. Placed between the photodiodes, a color LED is used for robot identification and feedback.

Most of the power in the robots are consumed by (in order) the motors, radio module, micro-controller, and LED. When stationary, each robot consumes approximately 40 mA and 100 mA when moving. Thus, with a 100 mAh battery, robots are capable of moving for one hour, and can work even longer under normal usage.

Radio Communication

Each robot communicates with the radio receiver using the NRF24L01+ chip. Using a teensy 3.1 microcontroller as the master and Arduino Pro mini as the slave, we tested the total communication times for different numbers of slaves per master and packet sizes. From the experiment, we found that the total time is linearly dependent of both packet size and number of slaves, and that we could have up to 18 slaves per master for a packet size of 12 bytes. Zooids uses 10 slaves per master for a safety factor of about 2.

Projector-based Tracking System

A projector-based tracking system similar to [START_REF] Lee | Moveable interactive projected displays using projector based tracking[END_REF] is used for robot position tracking. As opposed to camera based systems, our projector based tracking system does not add any latency from networking for the local feedback control on each robot, making position control more stable. Our system setup is demonstrated in Figure 5.5. Using a high frame-rate (3000 Hz) projector (DLP LightCrafter) from Texas Instruments Inc., a sequence of gray-coded patterns are projected onto a flat surface. Then, the photodiodes on the robot independently decodes the gray code into a location within the projected area, and sends its position and orientation to the master computer. Due to the number of the patterns, the position refresh rate is approximately 73 Hz (1/(41 images per pattern × 1/3000)). Due to the diamond pixels of the projector, the horizontal and vertical resolutions slightly differ. In the current setup in which the projector is placed 1.25 m above the table producing a 1 m × 0.63 m projection area, the horizontal and vertical resolutions are 1.15 mm and 1.12 mm, respectively.

Calibration

Every robot consists of the same components and thus should behave identically given the same inputs. Then, due to the discrepancies of the hardware, all robots do not exactly behave in the same manner and thus calibration for crucial elements is needed.

Minimum Speed Duty Cycle

Each robot has a minimum speed or Pulse Width Modulation (PWM) duty cycle that is needed to overcome the static friction between the wheels and the ground surface. While the robots have similar minimum duty cycle, they do not behave identically. Thus, during a startup phase, each robot goes through an initialization and calibration process to find their own parameters. This is achieved by incrementing the PWM duty cycle until it achieves moving the robot by 5 mm in 100 ms.

Preferred Speed Duty Cycle For most of their active time, robots move at their preferred speed. Similar to the minimum speed, there is a need for calibrating the preferred speed duty cycle. This is achieved again incrementing the PWM duty cycle until it moves at the nominal preferred speed of 44 cm/s.

Gain between Motors

As each robot behaves differently, the motors within the robot also behave differently and thus, a calibration between the motors is needed. The calibration process is as follows: record the initial orientation, let the robot move for 0.5 s, compare the final and initial orientation and either increase or decrease the motor gain accordingly. This process is repeated until the initial and final orientations are less than 5 degrees apart.

Software

As shown in Figure 5.5, the communication structure consists of four main layers from highest to lowest level: Application, Simulation, Server, and Hardware. At the application level, the desired positions of the robots are computed. These desired positions are transmitted to the simulation layer through a network socket. The application programmer can choose between two control strategies: Proportional-Integral-Derivative (PID) position control or Hybrid Reciprocal Velocity Obstacles (HRVO) combined with PID (these options are explained in the next paragraphs). Based on the chosen control strategy, the simulation layer computes the goal positions of the robots, either final positions for PID or intermediate points for HRVO, and sends them to the server. Finally, the server layer dispatches commands to the individual Zooids, while at the same time monitoring their status and position.

The control procedure for our system consists of three steps:

• Hungarian goal assignment (optional)

• HRVO global control strategy (optional)

• PID position control.

Before any movement, each robot first needs to be assigned its final position. The final positions may be specific for each robot or they can be dynamically assigned to move in a more efficient manner. The Hungarian algorithm [START_REF] Harold W Kuhn | The hungarian method for the assignment problem[END_REF], a well-known optimization method for one-to-one task-agent problems, can be used to assign the goal positions to robots in an optimal fashion. The cost function to be optimized is the summation of the squared distances from the initial to the final positions.

After the goal assignment step, robots need to move toward their goals, while minimizing possible collisions with each other robot. We chose to use the HRVO control strategy [START_REF] Snape | Independent navigation of multiple mobile robots with hybrid reciprocal velocity obstacles[END_REF][START_REF] Snape | The hybrid reciprocal velocity obstacle[END_REF] due to its fast real-time path planning capabilities. With HRVO, a robot moves at the user-defined preferred speed unless it detects possible collisions. In that case, it uses the notion of velocity obstacle, i.e., the set of all robot velocities that will result in a collision with another robot. While HRVO does not guarantee collisionfree, oscillation-free control, it reduces the number of collisions dramatically compared to other velocity obstacle strategies while providing real-time updates, essential to natural and smooth user interactions. To implement HRVO, we used a slightly modified version of the HRVO library created by [START_REF] Snape | Independent navigation of multiple mobile robots with hybrid reciprocal velocity obstacles[END_REF][START_REF] Snape | The hybrid reciprocal velocity obstacle[END_REF].

With the HRVO control strategy, we can derive the incremental goal positions along a path for each robot. These positions are sequentially sent to each robot which independently controls its motion through a PID controller based on the state machine shown in Figure 5.6. Given a final goal, the robot initially turns itself in the right direction and, once aligned, accelerates to its user-defined preferred speed. When it reaches the speed, it maintains it with a PID control on the orientation to ensure its direction towards the final goal. When a new incremental goal is given, it will still move at same speed but the PID control on orientation will direct the robot towards the new intermediate goal. When the robot arrives within 5 cm of the final goal, it slows down to its minimum velocity and once within 1 cm of the final goal, it stops and orients itself as commanded by the application programmer. To enable smooth transitions between the incremental goal positions, robots are given their next position at 60 Hz and will not stop unless given their final goal positions.

swarm uis: design principles and challenges

Swarm UIs radically change the way we think of user interfaces, not only from an end user's perspective but also from an application designer's perspective. We discuss new concepts, and highlight the major differences here. ling many with one Zooid, or with groups), a display aspect, and an environment aspect (operating in a neutral area, in a crowded desk populated with external objects, over a static background layer, or over a dynamic display). We expand on some of these aspects below.

Display: Things vs. Stuff

Designing Swarm UIs requires thinking both in terms of "things" and of "stuff". In our previous examples, a Zooid can stand for an individual object (e.g., a widget) or be part of a larger collection of objects (e.g., a circle). Figure 5.8 illustrates the continuum between these two paradigms: things are physical entities experienced as individual, solid objects; Stuff consist in physical entities experienced as shapes and material that can be reshaped, divided, merged, or temporarily solidified to emulate things. The elements making up stuff can be large enough to be visible (particles) or too small to be visible (atoms). Typical TUIs are located to the left of the continuumthey are made of things. In contrast, Swarm UIs occupy the right half of the continuum. As a low-resolution swarm UI implementation, Zooids stand in the gray area of the continuum and have both the affordance of things and stuff.

Figure 5.8: The continuum between "things" and "stuff".

The thing-stuff continuum also applies to traditional graphical displays. Many computer displays from the 80's were very low resolution (semigraphics from the Sinclair ZX-81 and Tandy TRS-80 were 64 × 48 pixels), thus pixels were discernible particles much like the Zooids in our previous examples (see Figure 5.9). Now with ultra-high resolution displays pixels became practically invisible, i.e., they became atoms. There are however major conceptual differences between pixel-based displays and Swarm UIs, which we discuss next.

Display: Fixed vs. Movable Elements

We are used to program graphics on computer displays where the elements (pixels) are arranged on a regular grid, and only their color is controlled.

Although elements of swarm UIs can also have different colors (in our system, each Zooid embeds a color LED), a major difference is that they can be positioned freely. Even at equal resolution between the two systems, the way elements can be combined into shapes is very different (see Figure 5.10).

In general, free positioning allows finer shape control than simply turning pixels on and off. At the same time, this extra flexibility comes at the cost of slower response time and higher engineering complexity, with algorithmic problems such as collision avoidance and optimal element-target assignment.

In addition, with systems with few elements such as Zooids, designers need to think carefully about how to use every Zooid optimally, the same way designers from the 80's had to think carefully about how to best use every pixel. It will become less of a concern as the resolution of swarm UIs increases, but on the other hand, engineering and algorithmic challenges will likely become harder. In addition, as shown in Figure 5.7, the display elements may be homogeneous, as with Zooids, or heterogeneous.

Display: Fixed vs. Variable Numbers of Elements

On regular graphical displays the total number of pixels is generally fixed, and the illusion of having more or less content on the screen is achieved by simply manipulating pixel color (e.g., having more or less dark pixels on a white background). In contrast, many swarm applications (e.g, our drawing application) require the number of elements to actually change over time.

Zooids cannot be created or destroyed, but as we saw, unassigned Zooids can be placed in a dedicated region and moved to the working area whenever they are needed. This type of object persistence contributes to realism and can help users remain oriented across view changes [START_REF] Chang | Animation: from cartoons to the user interface[END_REF]. As a result, object persistence is often implemented as a metaphor in modern GUIs (e.g., [Microsoft, 2016]). Swarm UIs support these metaphors natively, and they force designers to think about how to animate appearance and disappearance [START_REF] Chang | Animation: from cartoons to the user interface[END_REF]. However, when true appearance and disappearance are needed, swarm UIs may be impractical and the motions produced by elements constantly arriving and departing can be distracting to end users.

5.5.4 Display: Elements with an Identity vs. Interchangeable Elements

One important distinction to be made is between swarm UI elements that have a fixed identity, and elements that are interchangeable. In general, elements used as "things" have a fixed identity, whereas elements making up "stuff" are interchangeable. For example, in our shape drawing application, the Zooids that make up a circle or a line do not have an identity of their own and could be freely swapped. As explained in the implementation section, this makes it possible to optimize the swarm interface so that the motion of Zooids remain minimal even during complex transitions. At the same time, swapping a widget (e.g., one of the handles) with another Zooid is not desirable, as this might be disorienting to a user, especially if she was about to grasp it. Similarly, in systems where each Zooid has a stable meaning (e.g., a visualization system where a Zooid represent a data point), swapping Zooids can produce confusing or misleading animations. Therefore, the designer of a swarm UI should think carefully about which elements are interchangeable, and which elements should be given a fixed identity. Finally, elements that are manipulated should never be reassigned, which is ensured automatically in our current Zooid implementation.

Interaction: Element Manipulation

Regular graphical displays do not allow pixels to be physically manipulated.

Although direct touch displays give a decent illusion of direct manipulation, the subjective experience and the level of expressiveness fall short of true physical object manipulation [START_REF] Victor | A brief rant on the future of interaction design[END_REF]. In contrast, Zooids can be grasped and directly manipulated, allowing to tap into the richness of human hands [Le Goc et al., 2015a]. For example, in our swarm drawing scenario, users can not only manipulate curves using surrogates such as control points, they can also shape the curves directly. Our system explicitly supports such interactions by registering when a Zooid is touched and by constantly updating its goal based on its position. Generally, swarm UI designers should not only focus on the design of "synthetic" interactions, but also consider what is possible in terms of purely physical interactions [Jansen et al., 2015b]. Due to their form factor, Zooids can be manipulated both as collections of objects (stuff), and as individual objects (things). As swarm UI elements get smaller though, affordances will change dramatically. For example, grains of rice can be manipulated individually, but rice better affords being manipulated as "stuff". While object manipulation is supported natively in systems with large enough elements like ours, future swarm UIs will need to be able to coalesce multiple elements into solid objects to be able to support similar manipulations.

Interaction: Differing Roles of Elements

Different swarm UI elements can be given different roles. On the drawing application (see Figure 5.2), Zooids are used both for input and output, although different Zooids interpret input differently. In our circle drawing tool (see Figure 5.3), moving the two control points reshapes the circle, while moving any other Zooid translates it. While a few Zooids are used as controllers, the remaining are operated mostly for display purposes.

Giving different roles to different swarm UI elements allows for more design flexibility, but it also poses the problem of how to convey affordances. In our example applications we assign different LED colors to different functions, but the color mappings are arbitrary and this approach assumes a user who is already familiar with the system. Better affordances could be conveyed by giving different Zooids different shapes, consistent with the tradition of TUI design [Fitzmaurice and [START_REF] George | An empirical evaluation of graspable user interfaces: towards specialized, space-multiplexed input[END_REF]. These different shapes could be clippable, or alternatively, Zooids could change their shape dynamically. For a high-resolution swarm UI however, a more natural approach would consist of producing objects of different shapes by assembling many particles or atoms together, as we discussed previously.

Environment: Extra Visual Feedback

Although the drawing application we illustrated is a pure swarm UI, in practice many coarse-grained swarm UIs would need the display of extra visual information (such as text) to be really usable.

We illustrated two ways of doing this: one can craft a support surface that contains all the necessary annotations, provided these are stable over time (as in, e.g., board games). When annotations need to change over time, Zooids can be placed on a regular graphical display, or alternatively, top projection can be used if optical tracking is in the IR spectrum. Despite the current need for extra display hardware, Zooids can convey more visual information on their own than traditional TUIs that only involve a few tangibles as controllers and typically convey most visual information through additional graphical overlays. One can imagine that future swarm UIs will be high-resolution enough to be able to act as displays of their own, thereby entirely eliminating the need for virtual information overlays that suffer from many problems such as (for top projection) occlusion, difficulty of calibration, and difficulty of projecting on shiny or dark surfaces [START_REF] Gervais | Interaction and introspection with tangible augmented objects[END_REF].

limitations and future work

There are a number of technical limitations with the Zooids system that limit its capabilities and performance as a swarm user interface. These range from the scale and speed of the device to the cost. One significant limitation is that our robots have a non-holonomic drive, meaning that they cannot move freely in two-dimensional space and instead must turn to a specific heading like a car. Having a holonomic system with an omni-direction drive would allow the robots to move more smoothly and more easily respond to user interaction. Unlike the case of using robots as displays, where movement paths can be pre-computed [START_REF] Michael Rubenstein | Kilobot: A low cost scalable robot system for collective behaviors[END_REF], our interactive systems may not be able to find a simple or comprehensible path, especially when the movements are over small distances.

Currently, our sensing of input is limited to capacitive touch input on each robot around its circumference. When interacting with many Zooids at once, not all touch sensors will be activated, only the ones directly touching a user's hand. Sensor fusion techniques could be used to identify and match the motion between two or more robots that are being moved in unison. This would allow for richer interaction techniques leveraging the direct manipulation of many robots at once. Another technical limitation of our system is its use of an external projector for tracking. This requirement adds cost and also requires additional hardware and set up to use the system, impeding the scalability of Zooids. In addition, like all optical tracking systems, our system is limited by occlusions which may often happen when interacting with the system. Finally, our projector and photodiodes operate in the optical light spectrum, making it hard to use with too much ambient light (this could be improved some with the use of an IR projector and Photodiodes). A number of different approaches could improve our tracking. Using rotating IR laser line beacons, similar to Valve's Vive Lighthouse tracker (http://www.htcvive.com) could significantly reduce the cost, and having multiple beacons could solve some occlusion problems. However, we see great potential in wireless tracking, which could reduce setup to adding a small number of fixed beacons (anchors) for localization with received radio signal strength. Alternatively, future work on improving dead-reckoning location techniques with sensor fusion between wheel encoders and IMUs, coupled with either IR or RSSI anchor free localization between elements, could reduce the need for external tracking completely. We believe that advances in technology will benefit swarm UIs, allowing for more ubiquitous installations.

Power and charging management of many robots presents many challenges. Currently, our system relies on individual chargers in which each robot must be placed manually. An automated system, potentially with integrated wireless charging coils in each robot could allow robots to charge autonomously when needed by returning to a charging base station.

The scale and number of elements in our current system limits the type of interaction and applications that can be created -smaller and more elements may allow for radically different and richer styles of interaction with "stuff" instead of "things". In order to achieve smaller elements we will need to move away from geared DC motors with wheels for locomotion to other actuation, such as piezo actuators. Other micro-robots have been developed which utilize compliant linkages with piezo actuation to create locomotion similar to that of small insects at much smaller scales [START_REF] Sahai | Towards a 3g crawling robot through the integration of microrobot technologies[END_REF], however power electronics at this scale remain challenging [START_REF] Steltz | Power electronics design choice for piezoelectric microrobots[END_REF]. Another contributing factor which limits the number of robots is cost. Our current robot design at small scales of production is around $50 USD per robot in cost for parts and assembly. This makes swarms larger than 30-40 cost prohibitive outside of research applications. With further design for manufacturing at larger scales, the price per robot could be reduced, but other fabrication techniques such as printable and foldable robots [START_REF] Samuel M Felton | Robot self-assembly by folding: A printed inchworm robot[END_REF] may ultimately enable much cheaper swarm interface systems.

For smooth interaction, speed of the system is critical: the elements of a swarm UI need to be fast enough for shape change to occur at a usable rate. With our Zooids system, the transition time is typically from 1 to 3 seconds, depending on travel distance (up to 100 cm in our implementation) and the number of Zooids involved in the transition.

Another large limitation is the interaction area, as well as the type of surfaces on which the elements can move. Since Zooids's movement relies on a set of small rubber wheels, our system can only work for relatively flat surfaces with a minimal amount of traction. This limits our system to 2D interactions. Obviously, work on swarms of aerial drones [START_REF] Kushleyev | Towards a swarm of agile micro quadrotors[END_REF] present opportunities to make fully 3D interfaces. However, we see great opportunity in further exploration of ground based swarm interfaces that may be able to reconfigure into 2.5D or even 3D displays. Taking inspiration from ants and other insects, they could form complex shapes by interweaving and connecting, or even rolling on top of each other [START_REF] Cucu | Towards selfassembled structures with mobile climbing robots[END_REF][START_REF] John W Romanishin | M-blocks: Momentumdriven, magnetic modular robots[END_REF]. We also see great potential for different classes of robots which could help construct more 3D shapes, such as a ramp robot, or other passive building blocks that could allow swarm interfaces to form more complex structures similar to recent work in swarm robotics [START_REF] Werfel | Designing collective behavior in a termite-inspired robot construction team[END_REF].

Finally, we want to explore more application domains; now that we have created a scalable platform we can explore and quickly prototype. We believe information visualization is an exciting area, especially for creating engagement and for educational domains. It is also important to better understand the benefits and drawbacks of swarm user interfaces compared to traditional GUIs. For this purpose, conducting user studies will identify favorable conditions for the use of swarm user interfaces. We hope that our open source platform with also encourage other researchers, designers, and educators to explore a range of applications, and will enable further evaluation and study of tangible interaction principles.

conclusion

This chapter introduced Swarm User Interfaces, a new class of user interfaces made of "independent self-propelled elements that move collectively and react to user input". We described the technical implementation of Zooids, a novel open-source platform for building Swarm UIs, and illustrated its possibilities through concrete examples.

All necessary material and documentation for implementing Zooids can be found at https://github.com/PhysicalInteractionLab/SwarmUI/. Portions of this chapter were previously published in [Le Goc et al., 2015b], as well as in [Le Goc et al., 2016b]. Thus any use of "we" in this chapter refers to myself, Pierre Dragicevic, Samuel Huron and Jean-Daniel Fekete or Lawrence Kim, Ali Parsaei, Jean-Daniel Fekete, Pierre Dragicevic and Sean Follmer.

introduction

To illustrate the potential of Swarm UIs, we investigate how information visualization and more specifically data physicalization could be implemented using Swarm UIs. Data physicalization, a nascent research area in the information visualization community, consists in investigating "how computersupported, physical representation of data (i.e., physicalizations) can support cognition, communication, learning, problem solving and decision making" [Jansen et al., 2015a]. Data physicalization provides a compelling context for Swarm UIs, as it relies on rich and concrete tasks.

We refer to composite data physicalizations as physicalizations using many physically detached objects to represent data, consistently with the definition of composite TUIs given in Chapter 2 (see Section 2.4.2).

Many current physicalizations (e.g., [START_REF] Boleslavský | Google eye[END_REF]) are monolithic, static, and not interactive. Others are made of multiple individual objects that can be rearranged to represent a variety of informative configurations. We identify a promising category of physicalizations we call composite physicalizations. Composite physicalizations are combinations of multiple physical objects and can be designed to better leverage both human and technological capabilities.Yet, their design space is not well understood. We show that two important properties have to be considered when designing such physicalizations: their level of actuation and their manipulability. Through examples, we illustrate the trade-offs between these two dimensions, and identify the need for more research in this particular area. We finally adapted two typical representations in information visualization, to create fully interactive composite data physicalizations.

composite data physicalization

Usage of TUIs Information Visualization was tackled earlier in Section 2.2.4. We focus here on using physical representations of data, thus excluding using the use of handles and controls to interact with screen-based visualizations, such as [START_REF] Ullmer | Token+ constraint systems for tangible interaction with digital information[END_REF].

Based on research in cognitive science around embodied and distributed cognition, there has been recent interest in the information visualization field around physical data physicalizations [Huron et al., 2014b;Jansen et al., 2015b;[START_REF] Zhao | Embodiment in data sculpture: a model of the physical visualization of information[END_REF]. Researchers have already shown that there can be benefits to passive physical representations of data to promote engagement [Moere, 2008], to better support data exploration [START_REF] Jansen | Evaluating the efficiency of physical visualizations[END_REF], and for the vision impaired [START_REF] Susan | Tangible graphs for the blind[END_REF].

Less work has explored dynamic data physicalizations because they are more complex to build [Jansen et al., 2015b], but recent work has investigated the use of 2.5D shape displays for data exploration [START_REF] Taher | Exploring interactions with physically dynamic bar charts[END_REF]. However, the range of visualization techniques that can be supported with 2.5D shape displays is limited. Swarm interfaces provide a promising platform to physicalize many traditional 2D information visualizations, as well as newer interactive data visualizations [Microsoft, 2016;[START_REF] Soo | Dust & magnet: multivariate information visualization using a magnet metaphor[END_REF].

Following are existing examples of composite physicalizations, from which emerge two categories: manually arranged and actuated data physicalizations

Manually arranged physicalizations

An easy way to build a composite physicalizations is to arrange multiple elements manually in order to create visual patterns representing data. This type of composite physicalizations is fully manipulable, but not actuated at all. Such physicalizations have been studied by, e.g., [START_REF] Samuel Huron | Constructing Visual Representations: Investigating the Use of Tangible Tokens[END_REF]. In their study, users were asked to use square wooden tiles of various colors (see Figure 6.2 (right)) to build representations of a given dataset. The created physicalizations were fully manipulable as each element can be manipulated freely.

Other examples include an activity logging physicalization built by Hunger [2008] which uses stacks of LEGO bricks of different colors to represent activities carried on during each day, and Jacques Bertin's physical matrices [START_REF] Perin | Diy bertin matrix[END_REF] (see Figure 6.2 (left)). Such manually arranged physicalizations provide the benefits of being highly flexible and requiring little expertise [START_REF] Samuel Huron | Constructing Visual Representations: Investigating the Use of Tangible Tokens[END_REF]. However, constructing and updating them can be tedious and time consuming when manipulated objects are numerous.

Actuated physicalizations

By introducing automatic actuation and computation, it is possible to make composite physicalizations dynamic. Elements can be rearranged automatically to reflect changes in data. Several systems were presented earlier in Background chapter (see Section 2.3.1), and are reviewed here in the context of physicalization.

The ART+COM's kinetic sculptures are made of winch-controlled elements, which height can be accurately controlled to create complex representations (see one example in Figure 6.3). They are not data physicalisations as they do not convey data, but the hardware could be used in principle to build physicalisations. Even though these sculptures allows physicalizations to be updated dynamically, elements cannot be manipulated directly. This limitation is due to the wires used to control the objects and preventing them from being manipulated. Some actuated composite physicalizations can be manipulable but the level of actuation is limited. For instance, Durrell Bishop's Marble Answering Machine [START_REF] Polynor | The hand that rocks the cradle[END_REF] physicalizes incoming voice messages into marbles. Yet, the system cannot be considered as fully actuated. Indeed, as explained earlier, marbles have to be repositioned in the storage container once they have been heard. inFORM [START_REF] Follmer | inform: dynamic physical affordances and constraints through shape and object actuation[END_REF] as well as EMERGE [START_REF] Taher | Exploring interactions with physically dynamic bar charts[END_REF] are dynamic 2.5D shape displays that use a large collection of moving vertical bars. The potential of inFORM and EMERGE is illustrated in a set of physicalization examples (see Figure 6.4). From these examples, it is possible to Figure 6.4: EMERGE [START_REF] Taher | Exploring interactions with physically dynamic bar charts[END_REF](left) and inFORM [START_REF] Follmer | inform: dynamic physical affordances and constraints through shape and object actuation[END_REF] (right) allows to create physicalisations, here 3D barchart.

see that 2.5D shape displays allow to create fully actuated physicalizations, in the sense that each of its objects can move. It is also manipulable as each object can react to gestures. However, each element cannot be considered fully mobile as it can move in only one dimension. Moreover, many of the supported gestures are not direct-manipulation gestures, but rather mid-air abstract gestures. Thus, these systems are not fully manipulable.

challenges

The previous selected examples show that different types of composite physicalizations exist. Two characteristics stand out: actuation and manipulability. However, none of the examples is yet able to fully combine these two properties that we consider important for an ideal composite physicalization. Following are challenges to create such ideal composite physicalizatioms.

Technological considerations

One technological challenge is to support full actuation while ensuring the full mobility of objects. As demonstrated by ART+COM's kinetic sculpture (see Figure 6.3) and the shape displays (see Figure 6.4), one cannot fully manipulate objects that are physically constrained. By contrast, Durrell Bishop's marbles are not constrained in any way, allowing users to grasp marbles and manipulate several of them at the same time. However, supporting full object mobility will in many cases make the design more complex if computation or actuation features have to be embedded in the objects. Section 2.3.2 of Chapter 2 showed that several approaches are possible using external and internal actuation actuation. Yet, internal actuation seems more realistic as it scales up to arbitrary numbers of objects and does not require a controlled environment to operate.

Physical Object Design

One other aspect to consider is physical object design. To encourage users to take advantage of manipulation, the design of the objects is crucial as it will constrain the possible interactions. Even for physicalizations that are dynamic and updatable, objects have to be carefully designed. The choice of the form factor will impact possible manipulations such as grouping, stacking or assembling. For instance, square objects like LEGO bricks can be assembled and stacked easily while round objects cannot. The right form factor is highly context dependent, for example it is often desirable to have objects with a flat base to insure stability, but the roundness of objects can be also be exploited to ease actuation like marbles in Bishop's Marble Machine [START_REF] Polynor | The hand that rocks the cradle[END_REF]. The size and the material will also affect manipulation. Medium sized objects are easy to handle but users cannot manipulate many (dozens) at a time. Furthermore, low friction can make objects slippery and difficult to control, while heavy material can make manipulation tiresome.

physicalizing using swarm uis

Swarm UIs are by essence composite, making them well suited for implementing composite physicalizations. Moreover, the versatility of Swarm UIs allows to represent datasets of many different types. First, Section 5.5.1 showed that the element composing a Swarm UIs can be considered in two ways: as "things" or as "stuff". Both viewpoints are relevant when applied to data physicalization: while "things" allow to create physical marks, physical analogues to Bertin's visual marks [START_REF] Bertin | Sémiologie graphique: Les diagrammes-les réseaux-les cartes[END_REF], "stuff" supports more abstract representations using area or lines. We present here two adaptations of typical visual representations used in Information Visualization, illustrating both approaches using Zooids. The first physicalization shows how scatter plots can be implemented, using single Zooids to represent each data point.

The second example is a line chart representing time-series, aligning many Zooids to form a curve.

Multiple Scatter Plots

Scatter plots are one of the most common way to visualize data points [START_REF] Munzner | Visualization Analysis and Design[END_REF]. Looking more specifically at multivariate data, the ability to represent multiple dimensions at the same time is particularly relevant to better understand the data, identify trends and isolate clusters. The physicalization shown in Figure 6.5 consists of two juxtaposed scatter plots, each representing a couple of dimensions. In this physicalization, axes and labels are printed on simple sheets of paper and laid on a table. Once placed on the sheet, Zooids are automatically associated with the points of the dataset. Depending on which scatter plot they are placed on, Zooids immediately move to their position on space according to their values in each dimension. One or many Zooids can be picked up at a time, and moved from one scatter plot to the other. As soon as they are released, Zooids systematically try to reach back their position. While not implemented in this current physicalization, it is easy to imagine adding color encoding, as Zooids already provide color change capabilities.

From this example, we can see that Zooids allow to create a fully manipulable and actuated data physicalization. Indeed, each point can be independently grasped, lifted, moved in space, or placed on the side of the table to isolate a group. While interaction with this physicalization remains rather simple, direct manipulation of Zooids allows to explore multiple dimensions, and can help isolating trends and patterns in the dataset. Moreover, this physicalization is actuated in two dimensions as each point of the dataset can move on the table. 2D actuation also allows this physicalization to be updated in real-time, if the data changes. However, it cannot be considered as fully actuated as Zooids are not capable of elevating themselves on the third dimension. Alternative technologies to improve actuation could take inspiration from LeviPath [START_REF] Omirou | Levipath: Modular acoustic levitation for 3d path visualisations[END_REF] (although the physicalization would lose its manipulability as LeviPath does not support interaction), or BitDrones [START_REF] Rubens | Bitdrones: Towards levitating programmable matter using interactive 3d quadcopter displays[END_REF] (although only a few drones can occupy the same volume).

Time-Series Navigation

We can also used Zooids to physicalize and navigate in time-series data.

Although not yet implemented, the physicalization illustrated in Figure 6.6 shows with a line chart the evolution of resources usage on a computer. Again, axes and labels are static (e.g., printed on paper), while the data physicalization itself is dynamic and continuously updated. Indeed, the line chart appears to move to the left as new data arrives. At the bottom, Zooids take on the role of widgets to let users customize the display and navigate the data. The two Zooids at the bottom right specify the time range to physicalize -they act like the two thumbs of a range slider [START_REF] Ahlberg | Dynamic queries for information exploration: An implementation and evaluation[END_REF]. If the user moves the left thumb to the left (see Figure 6.6), the line chart stretches to show a wider time range. Moving both Zooids to the left scrolls in the past. Finally, another Zooid (see center of Figure 6.6) lets users switch between CPU usage and memory usage.

Suppose the user loads a different dataset, we implemented the physicalization that now represents the evolution of temperatures in Paris over a year (see Figure 6.7). Zooids move to the bottom to create a range slider. This slider has three control points to select the period of time over which the data is physicalized: a central thumb and two extrema to select the bounds of the time period. One can then use these handles to explore the evolution of temperatures, by scrolling through time. Moving the central thumb allows to manipulate both maxima simultaneously, while preserving their relative distance. Bringing the two maxima will focus the physicalization on a short period of time, giving a more detailed evolution to look at precise values. On the contrary, moving them away from each will represent the evolution of temperature over a longer timeframe, providing overviews and trends over the selected period. These line chart physicalizations, like the previous example, are actuated on the tabletop, thus on two dimensions. In addition, they are fully manipulable. Yet, compared to the previous example, manipulating points of the line chart do not serve any purpose, as each Zooid can be grasped and moved, but these will automatically return to their position. Interaction is thus limited to manipulating the three thumbs of the range slider. Therefore, this physicalization cannot be considered as fully manipulable. One of the limitations of our current implementation lies in the fact that interacting directly with a majority of Zooids has no effect. However, we can imagine that directly manipulating each Zooid of the line chart would allow temporal exploration, similar to DimpVis [START_REF] Kondo | Dimpvis: Exploring time-varying information visualizations by direct manipulation[END_REF]. Zooids could provide haptic feedback to hint unsupported manipulations in an analogous fashion to Thumbles [Patten Studio, 2014] which resist mathematically illegal actions.

conclusion

While many current physicalizations are monolithic and static, physicalizations made of multiple objects can better leverage both human and technological capabilities. We called such physicalizations composite physicalizations and showed that they can be usefully classified according to two dimensions: level of actuation and manipulability (see Figure 6.8. Only a few systems combine both characteristics and none supports both full manipulability and full actuation. We demonstrated how scatter plots and line charts can be physicalized using Swarm UIs, and situated these two physicalizations in the design space introduced. Although currently simple and limited, we see the use of Swarm UIs as a promising platform for providing 2D actuation and full manipulability, allowing to push forward composite physicalizations. Inspired by multiple visions presented along the chapters, I advocate that TUIs may not have received the attention they deserve, and may not have the place they should in our everyday interaction with digital information. Modalities such as multitouch, pen-based or mouse and keyboard, have also shown their strengths. While all these are ultimately not meant to be used alone, a challenge in Human-Computer Interaction is to push each to better understand their respective benefits. Thus, my goal with this work was to push the capabilities of TUIs further, and help better identify their strengths. conclusion 7.1 summary I started this dissertation with an overview of the state of the art of Tangible User Interfaces. I first described the appearance of TUIs and their evolution, as well as the efforts to assess their potential benefits. I then identified the main application domains for TUIs and finally reviewed existing approaches to provide actuation for TUIs. I emphasized the need for further research to better support versatility and physicality in TUIs, as motivated in Chapter 1.

Level of Actuation

Degree of Manipulability

I then used empirical methods to demonstrate possible benefits of TUIs over multitouch interfaces for 2D manipulations of collections of small generic objects on tabletop surfaces. I presented a new method to conduct user experiments to help investigate fundamental differences between TUIs and multitouch interfaces. This method uses physical proxies to emulate both the flat, non-graspable objects of touch interfaces and thick and easy to manipulate objects of TUIs, in a way that made direct comparison possible. The results suggested that thickness of the objects and physicality to manipulate collections of generic objects played an important role. I then introduced SmartTokens, small-sized generic objects that can sense multiple types of motion and grips, and communicate input events wirelessly. SmartTokens allow to augment collections of generic objects with sensing capabilities to enable user input. I described their design and implementation, and how they extend the design space of TUIs by contributing an open platform for embedding sensing within small objects. To illustrate SmartTokens, I presented a novel TUI design for event notification and personal task management.

As a next step building on experience from SmartTokens, I investigated versatile composite TUIs and introduced Swarm User Interfaces. I presented Zooids, a TUI platform composed of collections of small and fully interactive robots. Zooids instantiate Swarm User Interfaces. I demonstrated their potential and versatility with a set of application scenarios, alongside with descriptions of their implementation and design, and general design considerations for Swarm User Interfaces. I finally disscussed how to use Swarm User Interfaces for Information Visualization. I defined the term of composite physicalizations, surveyed existing examples and introduced manipulability and actuation to characterize composite physicalizations. I then illustrated how typical visual representations can be adapted using Zooids to become actuated and manipulable composite physicalizations.

limitations of the chosen approach

Inspired by visions of [START_REF] Weiser | The computer for the 21st century[END_REF], [START_REF] Ishii | Radical atoms: Beyond tangible bits, toward transformable materials[END_REF][START_REF] Victor | A brief rant on the future of interaction design[END_REF], one of my goals was to emphasize physicality in order to focus on hand manipulation and interaction beyond screens. To reach this goal, I presented Zooids which showed that Swarm UIs allow purely physical interaction, as each provides both input and output capabilities.

Investigations of Zooids have nonetheless shown that physicality does not come without difficulties: scalability remains a major limitation of current Swarm UIs. Manipulating large amounts of objects can quickly become physically invading and cumbersome. For instance, interacting with over forty Zooids was challenging. They require more space than we had expected, thus restricting the movements of the group. Moreover, mechanical systems showed problems such as break downs or wear and tear, that do not occur with GUIs. During the development of Zooids, several objects broke down due to defective motors. Those are inherent limitations of physicality, and are challenging to overcome.

Research in TUI has aimed for specificity to make interfaces easier to use, more enjoyable and stimulating. Thus, existing TUIs often only support the few contexts they were designed for. Other modalities (e.g., multitouch interfaces) demonstrate that versatility can be profitable. That is why I argued for more versatility in TUIs. Chapter 5 and Chapter 6 that increased versatility offers new opportunities for tangible interaction.

Yet, I do not claim that there is a single platform that is able to handle all tasks. Desktop computers support certain tasks better than multitouch devices, and similarly TUIs have predilection application domains as well as contexts in which they do not excel. Moreover, generic objects make interfaces less discoverable, as the appearance of individual objects does not convey enough information. A very high degree of versatility may bring more problems than benefits, the challenge thus lies in finding the right balance between specificity and versatility.

perspectives and future work

Swarm UIs allow to create many novel types of physical interactive systems, encouraging the exploration of new paths for research. But this work only started to scratch the surface of what is possible with Zooids, and much still remains to be done. The development and implementation of Zooids to create the first Swarm UIs was a major technological challenge. But bigger challenges arise now that a prototype is operational, on both short and long term perspectives.

Improving interaction with Swarm UIs

Firstly, more research has to be pursued as the questions, as questions raised by Chapter 5 indicate.

As Chapter 5 and Chapter 6 started to demonstrate, the versatility offered by Swarm UIs allows to emulate dynamic physical widgets like sliders and buttons, that are already familiar to users. One of the main objectives of the work presented in this dissertation was to provide more versatile TUIs, using a generic form factor to create objects that can be associated with different types of digital information.

Dynamic affordances identified by [START_REF] Follmer | inform: dynamic physical affordances and constraints through shape and object actuation[END_REF] appear as a promising direction for research on Swarm UIs. But more work needs to be done to support richer manipulations of digital information. While this allowed us to use Zooids in diverse contexts, Zooids lost part of the discoverability TUIs provide when using specific and iconic designs. In the current implementations, Zooids can only change the color of their top surface, but changing the color of the entire body of Zooids would be more communicative and would better reflect changes in the system. Zooids could also provide more advanced output capabilities than motion, through mechanisms to modify their physical appearance. For instance, Zooids could morph from a cube to a cylinder, or increase their size.

Exploring interaction techniques further will allow to better leverage physicality, and provide richer manipulation vocabularies. Existing gestures and techniques can be adapted to Swarm UIs to take advantage of experience of users' interaction with existing devices (e.g., zoom, selection, etc.). At the same time, a new challenge appear in inventing techniques specific to Swarm UIs. Screen based representations have strongly influenced the way we interact with digital information, but trying to think interaction beyond screens requires to disregard existing tendencies and approach the problem with a fresh eye. Touch interaction remained limited until it moved on, from single touch inspired from mouse pointing to leverage unique possibilities of multitouch such as direct manipulation or multi-fingers gestures. Similarly, Swarm UIs need specific interaction techniques that will reveal their potential and expressiveness.

Introducing interactive systems that rely only on physical objects could be disturbing for users at first. Graphical representation of digital information is nowadays the norm, as computing has evolved with this medium for several decades. Just like for screens, interacting with digital information using only physical manipulation would require time for users to learn and become comfortable with them. Users will have to know intuitively how to interact with Zooids, especially with more abstract and less obvious techniques. Such dynamic affordances could help explore how to suggest interactivity [START_REF] Boy | Suggested Interactivity: Seeking Perceived Affordances for Information Visualization[END_REF]. For instance, investigation animated demonstrations seems like a promising direction.

Moreover, extensive research that allowed to understand how we interact using screens (e.g., Fitt's Law for pointing) will have to be adapted for physical objects. It is important to better understand how users perform using Swarm UIs, and their impact on user experience. To this end, it is necessary to conduct user studies, to help evaluate benefits and limitations of Swarm UIs, as well as potential directions to improve Zooids.

Better exploring data using Swarm UIs

Swarm UIs have shown good potential for Information Visualization and Data Physicalization (Chapter 6), and further investigation seems promising. One example Kinetica [Rzeszotarski and Kittur, 2014] can be a strong source To better exploit the richness of Swarm UIs for Information Visualization, interaction techniques different from ones used in traditional GUIs have to be developed. Kinetica showed that it can be interesting to change the representation style to be able to observe different patterns in the data. Zooids could then adapt their behavior to their representation. For example, on the multi-dimensional scatter plots (Section 6.4.1) mat, each Zooid can be manipulated independently. On the bar chart mat, Zooids representing the same bar move collectively when data is updated or manipulated. Using different physical mats could allow to physically change the representation of the data. For example, Zooids could be put on a scatter plot mat to show the distribution of points across the dimensions. The analyst could then isolate a subset of the data points, and place them on a different mat to obtain a bar chart, providing a more precise representation of the subset over one dimension. Other techniques could be imagined to better exploit Swarm UIs in Data Physicalization.

Creating educational tools with Swarm UIs

Swarm UIs are not limited to the applications mentioned so far. It would be interesting to investigate how could Zooids and interactive objects be used as pedagogical interactive tools in educational contexts. Inspired by more Using interactive systems to support teaching children is not new, HE-LIOS [START_REF] Fleck | Marker-based augmented reality: Instructional-design to improve children interactions with astronomical concepts[END_REF] combines Augmented Reality with TUIs to teach astronomy to children of 8 to 11 years old to stimulate active explorations of scientific problem. The system was tested in two primary schools, the studies showed that HELIOS supported children during their explorations. Zooids could be used to explain the celestial mechanics and the solar system, as illustrated in Figure 7.2. Each Zooid would represent a planet, symbolized by its color, and would move following the orbit around the sun. Such a system would allow to explore Newton's law of universal gravitation, in order to help understand how planets attract each other. et al. [2008] discuss how the nature of artificial movement affects our perception and emotions. They explain:

Investigating affect through motion

Parkes

We can easily distinguish artificial motion because of its exact repeatability. In designing kinetic interactions, repeatable exactness is the simplest form of control state, and in many behaviors it is easily identifiable. Introducing a level of variation in kinetic interfaces or perhaps even "noise" can add a degree of an organic natural feeling, usually missing from direct digital actuator control.

Similarly, older work from [START_REF] Heider | An experimental study of apparent behavior[END_REF] investigated the perception of motion. In an experiment, participants watched an animation of three simple polygons, each moving in different fashions. As participants were then asked to interpret these moving shapes, they described "animate beings" with different personalities and motives. This shows that motion offers new opportunities as a rich communication medium.

Indeed, demonstration sessions we conducted with Zooids, as well as online videos have suggested that people often respond with affection when seeing or interacting with them. Due to imperfections in design and controls, the fast but rocky and unstable movements of Zooids are not repeatable. According to Parkes et al., this "noise" can explain the affection users developed when using the system. While this phenomenon was unexpected, the language of motion for tangible interaction appears to be a very interesting area to investigate further. Controlling the accuracy in motion of Zooids could support different degrees of affection in interaction [START_REF] Picard | Affective computing[END_REF], depending on the application domain. For example, communicating playful and positive intentions using fast and slightly out of control movements could be beneficial in educational context to help engage children. Data exploration of complex information would on the contrary require concentration and precision, thus may require neutral and accurate motions.

On the longer run

A last challenge is to anticipate the role of Swarm UIs in a more distant future, i.e., in twenty or thirty years. It is legitimate to believe that programmable matter may come closer to reality. It may invade our everyday lives, where some objects made of programmable matter could fulfill multiple and diverse functions, and the computational nature of these objects would enable to make them fully interactive. But this could also shake the foundations of the knowledge people have of the physical world. As explained earlier in this dissertation, one of the strengths of TUIs is the knowledge users have of the physical world, how it behaves and reacts to their actions. The laws of physics are known and predictable, allowing humans to build up experience with the physical world. With programmable matter, material objects will no longer only be ruled by the laws of physics but also by the computational program controlling the object. Poor design of objects using programmable matter could then lead to a physical world in which humans would not be able to predict the behavior of things. The recent expansion of Artificial Intelligence already started to raise discussions on how it should integrate into our daily lives, as it could blur the lines between what is computing and what is human. Similarly, introducing programmable matter could make it hard to distinguish objects following the laws of physics from others. Thus, design rules have to be established to allow users to differentiate static objects from ones made of programmable matter.

conclusion

While inspirational visions to bring computing in the physical world are still far from being fully met, the work presented in this dissertation is a small step towards truly versatile and highly tangible user interfaces. It contributes new technological tools and concepts to the community, and will hopefully help further exploration of interaction beyond common interfaces such asscreens. A lot remains to be done, but I hope my work will allow researchers and also designers, engineers and practioners of different domains investigate new types of interactions based on highly physical interfaces. Je commence par examiner l'état de l'art et souligne le besoin d'approfondissement dans cette direction. J'y observe la spécificité des systèmes existants, limitant leur utilisation et diffusion, de même que l'utilisation récurrente d'écrans et autres dispositifs de projection comme moyen de représentation du monde numérique. Tirant les leçons de la recherche existante, je choisis d'orienter mes travaux autour de dispositifs physiques constitués uniquement de collections d'objets génériques et interactifs. Mon but est d'apporter plus de polyvalence aux interfaces purement tangibles. J'articule pour cela ma recherche en quatre temps.

Je mène tout d'abord une étude comparant les interfaces tangibles et tactiles, dans le but d'évaluer de potentiels bénéfices de l'utilisation d'objets physiques. J'étudie conjointement l'influence de l'épaisseur des objets sur la manipulation. Les résultats suggèrent tout d'abord de modérer les conclusions de nombre d'études existantes, quant aux avantages de la tangibilité en terme de performances. Ces résultats confirment toutefois l'amélioration de l'agrément lors de l'utilisation de dispositifs physiques, expliquée par une plus grande variété ainsi qu'une plus grande fiabilité des manipulations réalisées.

Je présente dans un deuxième temps Smart-Tokens, un dispositif à base de petits objets capable de détecter et reconnaître les manipulations auxquelles ils sont sujets. J'illustre les Smart-Tokens dans un scénario de gestion de notifications et de tâches personnelles.

Je poursuis en introduisant les Interfaces en essaim, une sous-catégorie des interfaces tangibles, constituée de collections de nombreux robots autonomes et interactifs. Pour les illustrer, je présente les Zoïdes, une plateforme ouverte pour développer des Interfaces en essaim. Je démontre le potentiel quant à leur polyvalence avec un assortiment d'applications, et clarifie les règles de conception des Interfaces en essaim.

Dans un quatrième temps, je me concentre sur le domaine de la visualisation d'informations dans le but d'illustrer la polyvalence des interfaces en essaim. Je définis les physicalisations de données composites comme un sous-ensemble des visualisations physiques, et en implémentent plusieurs exemples en utilisant les Zoïdes. Je termine en ouvrant perspectives et futures directions, et en tirant les conclusions des travaux réalisés au cours de cette thèse.

Title: Supporting Versatility in Tangible User Interfaces Using Collections of Small Actuated Objects Keywords: Tangible Interaction, Versatility, Manipulation, Collections of Object, Actuation Abstract: In this dissertation, I present my work aiming at making tangible user interfaces more versatile with a higher degree of physicality, in order to bridge the gap between digital and physical worlds. To this end, I study and design systems which support interaction with digital information while better leveraging human hand capabilities.

I start with an examination of the current related work, and highlight the need for further research towards more versatility with a higher degree of physicality. I argue that the specificity of existing systems tends to impair their usability and diffusion and induce a dependence on screens and other projections as media to represent the digital world. Building on lessons learned from previous work, I choose to focus my work on physical systems made of collections of generic and interactive objects. I articulate my research in four steps.

Firstly, I present a study that compares tangible and multitouch interfaces to help assess potential benefits of physical objects. At the same time, I investigate the influence of object thickness on how users manipulate objects. Results suggest that conclusions from numerous previous studies need to be tempered, in particular regarding the ad-vantages of physicality in terms of performance. These results however confirm that physicality improves user experience, due to the higher diversity of possible manipulations.

As a second step, I present SmartTokens, a system based on small objects capable of detecting and recognizing user manipulations. I illustrate SmartTokens in a notification and personal task management scenario.

  Figure 1.1A comparison of possible interactions with "Pictures under Glass"[START_REF] Victor | A brief rant on the future of interaction design[END_REF] with typical manipulations we do everyday. . . . . . . . . . . . . . . . . . . . . Figure1.2 metaDESK[START_REF] Ullmer | The metadesk: models and prototypes for tangible user interfaces[END_REF] allows to interact with a map by manipulating miniatures of the buildings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 1.3The tension between physicality and versatility in nowadays interfaces. Current technologies focus on one of these, but ideal TUIs would combine a high degree of physicality while supporting versatility. . . . Figure1.4The BRETAM Model[START_REF] Gaines | Modeling and forecasting the information sciences[END_REF] describes in six steps the evolution of information sciences. . . . . . . . Figure1.5On the left: a LEGO brick. On the right: A polar bear sculpture[START_REF] Kenney | Art with lego bricks: Polar bear[END_REF] made of 95,000 LEGO Bricks by the artist Sean Kenney. . . . . . . . . . . . . . . . . . Figure 1.6 Bit Planner [Vitamins, 2014] is a calendar using LEGO bricks to represent the schedule in a company. Each line of the calendar is a person (left), each brick represents half a day working on a project indicated by the color. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 1.7 PixelBot [Alonso-Mora et al., 2011] are robots capable of moving in synchrony and changing color. Here a collection of PixelBots displays a fish jumping out of the water. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.1 Aish [1979]'s architectural CAD system uses physical bricks to build 3D models. Physical assemblies (left) allow to create equivalent digital 3D models (right). . . Figure 2.2 Bishop's Marble Answering Machine represents voice messages with physical marbles. (Illustration by Matt Jones from Dragicevic and Jansen [2012].) . . . . . . . . Figure 2.3 Graspable User Interfaces [Fitzmaurice, 1996] use physical handles to control directly digital content, here to draw geometric figures. . . . . . . . . . . . . . . Figure 2.4 Ullmer and Ishii [2000]'s MVC interaction model for GUIs (left) and the MCRpd model for TUIs (right). . . Figure 2.5 The Suanpan is the Chinese abacus and allows to perform mathematical operations (Image from Wikipedia [2016]). . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.6 Fishkin [2004]'s 2D taxonomy classifies TUIs according to metaphor and embodiment. . . . . . . . . . . . .

  Figure 2.7Flowblocks[START_REF] Zuckerman | Extending tangible interfaces for education: digital montessori-inspired manipulatives[END_REF] are Montessoriinspired building blocks letting children experiment with abstract concepts of dynamic behavior. . . . . . . Figure2.8Perlman's TORTIS Slot Machine[START_REF] Perlman | Using computer technology to provide a creative learning environment for preschool children[END_REF] (left) allow to program a robot using physical cards. LEGO Mindstorms [LEGO, 2017] (right) combine LEGO bricks with a programmable brick, sensors and actuators to create interactive assemblies. . . . . . . . . . . . Figure 2.9 Pico [Patten and Ishii, 2007] is an interactive tabletop surface capable of tracking and moving small objects to solve spatial layout problems, here laying out cellular phone towers. . . . . . . . . . . . . . . . . . . . . . Figure 2.10 Tangible Query Interfaces [Ullmer et al., 2003] allow to interact with visualizations using tangible controllers. Figure 2.11 Physicalizations [Jansen, 2014a] are objects representing data using physical properties such as shape or material. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.12 The reacTable [Jordà et al., 2007] is a tangible music interface using tokens to create electronic music. . . . . Figure 2.13 Using particle jamming [Follmer et al., 2012], Tunable Clay can modify the stiffness of its material support 3D modeling. . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.14 Lumen [Poupyrev et al., 2004] is a low resolution shape display that can communicate 2D graphical images and augmented with dynamic physical shapes (left). inFORM [Follmer et al., 2013] is a high resolution shape display and can create complex shapes and senses user actions (right). . . . . . . . . . . . . . . . . . Figure 2.15 [ART+COM, 2016]'s kinetic sculptures use actuated suspended objects to represent shapes. Here Kinetic Rain (top) morphs into a plane, The Shapes of Things to Come (bottom) represent a car. . . . . . . . . . . . . . . . Figure 2.16 Madgets [Weiss et al., 2010a] are actuated physical widgets on multitouch tabletop. . . . . . . . . . . . . . . Figure 2.17 Tangible Bots [Pedersen and Hornbaek, 2011] (left) provide physically actuated controls to interact with multitouch tabletop surface, here used for musical creation . Thumbles [Patten Studio, 2014] (right) are robots that can be used as handles to interact with digital media, but also provide dynamic tactile feedback. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.18In the movie X-Men, a surface capable of changing its shape dynamically (Image from[START_REF] Dragicevic | List of physical visualizations[END_REF]). . . . . . . . . . . . . . . . . . . . . . . . . Figure2.19BitDrones[START_REF] Rubens | Bitdrones: Towards levitating programmable matter using interactive 3d quadcopter displays[END_REF] are flying interactive devices supporting direct manipulation, allowing room-scale tangible interaction . . . . . . . . . . . . . . .

Figure 2. 20

 20 Figure 2.20 Partial taxonomy of TUIs. . . . . . . . . . . . . . . . . . Figure 2.21The continua between compositeness and monolithicity(top), and between homogeneity and heterogeneity in TUIs (bottom). . . . . . . . . . . . . . . . . . . . . Figure3.1 Chips (left) and pucks (right) as used in our study. . . Figure 3.2 The experimental Setup. Colors indicate areas -the surface was entirely white except for 3 whose boundaries were marked. . . . . . . . . . . . . . . . . . Figure 3.3 A block consists in four consecutive, alternating between placement and sorting (Colors are assigned randomly). . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.4 Visual targets for a spread placement task (left) and a packed placement task (right). . . . . . . . . . . . . . . . . Figure 3.5 Mean task completion times per condition (left), and within-subject ratios between chips and pucks, where values > 1 indicate an advantage for pucks (right). Error bars are 95% CIs. . . . . . . . . . . . . . . . . . . . Figure 3.6 Mean task accuracies per condition (left), and withinsubject ratios between chips and pucks, where values > 1 indicate an advantage for pucks (right). Error bars are 95% CIs. . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.7 Questionnaire responses regarding subjective experience. Error bars are 95% bootstrap CIs. . . . . . . . . . Figure 3.8 Frequencies of occurrence of manipulation gestures and manipulation errors, for Chips and for Pucks. . . . Figure 3.9 Four chips lost during a bulldozer push (top), while no loss happens with the pucks (bottom). . . . . . . . . Figure 4.1 SmartTokens are small-sized tokens supporting touch and motion sensing, and wireless communication. . . . Figure 4.2 SmartToken hardware design . . . . . . . . . . . . . . . Figure 4.3 Four different grips with a SmartToken. . . . . . . . . . Figure 4.4 SmartToken wireless sensor network architecture, where each SmartToken communicates manipulation events to a coordinator. . . . . . . . . . . . . . . . . . . . Figure 4.5 State-machine describing input events a SmartToken can emit. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.6 Token dispenser (left) and Laura's notification setup (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.7A short strip illustrating the mail notification system . Figure5.1Zooids can be held as tokens, manipulated collectively or individually, behave as physical pixels, act as handles and controllers, and can move dynamically under machine control. They are building blocks for a new class of user interface we call Swarm User Interfaces. Figure 5.2 Freehand swarm drawing (1-3) and curve manipulation (4). . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 5. 2

 2 Figure 2.20 Partial taxonomy of TUIs. . . . . . . . . . . . . . . . . . Figure 2.21The continua between compositeness and monolithicity(top), and between homogeneity and heterogeneity in TUIs (bottom). . . . . . . . . . . . . . . . . . . . . Figure3.1 Chips (left) and pucks (right) as used in our study. . . Figure 3.2 The experimental Setup. Colors indicate areas -the surface was entirely white except for 3 whose boundaries were marked. . . . . . . . . . . . . . . . . . Figure 3.3 A block consists in four consecutive, alternating between placement and sorting (Colors are assigned randomly). . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.4 Visual targets for a spread placement task (left) and a packed placement task (right). . . . . . . . . . . . . . . . . Figure 3.5 Mean task completion times per condition (left), and within-subject ratios between chips and pucks, where values > 1 indicate an advantage for pucks (right). Error bars are 95% CIs. . . . . . . . . . . . . . . . . . . . Figure 3.6 Mean task accuracies per condition (left), and withinsubject ratios between chips and pucks, where values > 1 indicate an advantage for pucks (right). Error bars are 95% CIs. . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.7 Questionnaire responses regarding subjective experience. Error bars are 95% bootstrap CIs. . . . . . . . . . Figure 3.8 Frequencies of occurrence of manipulation gestures and manipulation errors, for Chips and for Pucks. . . . Figure 3.9 Four chips lost during a bulldozer push (top), while no loss happens with the pucks (bottom). . . . . . . . . Figure 4.1 SmartTokens are small-sized tokens supporting touch and motion sensing, and wireless communication. . . . Figure 4.2 SmartToken hardware design . . . . . . . . . . . . . . . Figure 4.3 Four different grips with a SmartToken. . . . . . . . . . Figure 4.4 SmartToken wireless sensor network architecture, where each SmartToken communicates manipulation events to a coordinator. . . . . . . . . . . . . . . . . . . . Figure 4.5 State-machine describing input events a SmartToken can emit. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.6 Token dispenser (left) and Laura's notification setup (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.7A short strip illustrating the mail notification system . Figure5.1Zooids can be held as tokens, manipulated collectively or individually, behave as physical pixels, act as handles and controllers, and can move dynamically under machine control. They are building blocks for a new class of user interface we call Swarm User Interfaces. Figure 5.2 Freehand swarm drawing (1-3) and curve manipulation (4). . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure

  Figure 5.3 Circle swarm drawing, where Zooids are automatically inserted (2) or discarded (3) depending on the circle's diameter. . . . . . . . . . . . . . . . . . . . . . . . Figure 5.4 Exploded view of a Zooid. . . . . . . . . . . . . . . . . . Figure 5.5 Software Architecture. . . . . . . . . . . . . . . . . . . . Figure 5.6 Structure of local PID position control . . . . . . . . . . Figure 5.7 Design Space explored with Zooids. . . . . . . . . . . . Figure 5.8The continuum between "things" and "stuff". . . . . . Figure5.9Alien from the game Space Invaders from Taito(1978) and main character from the game Mario Bros byNintendo (1983). . . . . . . . . . . . . . . . . . . . . . . . . . Figure 5.10 A circle obtained by assembling 16 elements using (1) Bresenham's algorithm and (2) free object positioning. Figure 6.1 Multi-variate scatterplots implemented with Zooids allow to interact physically with data. . . . . . . . . . . Figure 6.2 A reproduction of Jacques Bertin physical matrices by Perin et al. [2015] (left). Physicalizations using colored wooden tiles by Huron et al. [2014c] (right). . . . . . . . Figure 6.3 BMW Kinetic sculpture [ART+COM, 2016] . . . . . . . Figure 6.4 EMERGE [Taher et al., 2015](left) and in-FORM [Follmer et al., 2013] (right) allows to create physicalisations, here 3D barchart. . . . . . . . . Figure 6.5 Multiple juxtaposed scatterplot to represent and explore multivariate data. . . . . . . . . . . . . . . . . . . . Figure 6.6 A line chart physicalization of the usage of the resources of a computer. . . . . . . . . . . . . . . . . . . . Figure 6.7 A line chart physicalization of the temperature of Paris over a year. . . . . . . . . . . . . . . . . . . . . . . Figure 6.8 Examples of existing physicalizations (described in Section 6.2) classified according to our two dimensions, alongside our implementation using Zooids. . . Figure 7.1 Kinetica [Rzeszotarski and Kittur, 2014] allows users to manipulate directly multivariate data on multitouch devices, such as filtering using with a virtual blade. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 7.2 Zooids could be used as pedagical interactive tools to support teachers to explain and explore the functioning of the Solar System. . . . . . . . . . . . . . . . . . .

Figure 1

 1 Figure 1.1: A comparison of possible interactions with "Pictures under Glass" [Victor, 2011] with typical manipulations we do everyday.

Figure 1 . 6 :

 16 Figure 1.6: Bit Planner [Vitamins, 2014] is a calendar using LEGO bricks to represent the schedule in a company. Each line of the calendar is a person (left), each brick represents half a day working on a project indicated by the color.

Figure 1 . 7 :

 17 Figure 1.7: PixelBot [Alonso-Mora et al., 2011] are robots capable of moving in synchrony and changing color. Here a collection of PixelBots displays a fish jumping out of the water.

  user interfaces: history and overview

Figure 2

 2 Figure 2.1: Aish [1979]'s architectural CAD system uses physical bricks to build 3D models. Physical assemblies (left) allow to create equivalent digital 3D models (right).

Figure 2 . 2 :

 22 Figure 2.2: Bishop's Marble Answering Machine represents voice messages with physical marbles. (Illustration by Matt Jones from Dragicevic and Jansen [2012].)

Figure 2 . 3 :

 23 Figure 2.3: Graspable User Interfaces [Fitzmaurice, 1996] use physical handles to control directly digital content, here to draw geometric figures.

Figure 2 . 4 :

 24 Figure 2.4: Ullmer and Ishii [2000]'s MVC interaction model for GUIs (left) and the MCRpd model for TUIs (right).

Figure 2 . 5 :

 25 Figure 2.5: The Suanpan is the Chinese abacus and allows to perform mathematical operations (Image from Wikipedia [2016]).

Figure 2 . 6 :

 26 Figure 2.6: Fishkin [2004]'s 2D taxonomy classifies TUIs according to metaphor and embodiment.

  (see Figure 2.7). Using these to teach modeling and abstract concepts of dynamic behavior, Zuckerman et al. could show that both systems were accessible, engaging and stimulating to learn.

Figure 2 . 7 :

 27 Figure 2.7: Flowblocks [Zuckerman et al., 2005] are Montessori-inspired building blocks letting children experiment with abstract concepts of dynamic behavior.

  Nowadays, LEGO Mindstorms[LEGO, 2017] is probably the most successful platform to teach children and teenagers how to program. Based on the well known LEGO construction kit, Mindstorms includes an "intelligent" brick than can read sensors and control actuators, to create controllable LEGO programmable. The system resulted from a long collaboration between LEGO and MIT MediaLab's Lifelong Kindergarten Group. The first commercial version of was released in 1998 and brought computation to LEGO assemblies. LEGO Mindstorms have evolved through multiple versions, they now provide powerful computing units offering numerous opportunities (see Figure2.8 (right)), and also support visual programming to remain accessible to children of different ages.

Figure 2 . 8 :

 28 Figure 2.8: Perlman's TORTIS Slot Machine [Perlman, 1976] (left) allow to program a robot using physical cards. LEGO Mindstorms [LEGO, 2017] (right) combine LEGO bricks with a programmable brick, sensors and actuators to create interactive assemblies.

Figure 2 .

 2 Figure2.11: Physicalizations[Jansen, 2014a] are objects representing data using physical properties such as shape or material.

Figure 2 .

 2 Figure 2.15: [ART+COM, 2016]'s kinetic sculptures use actuated suspended objects to represent shapes. Here Kinetic Rain (top) morphs into a plane, The Shapes of Things to Come (bottom) represent a car.

Figure 2

 2 Figure 2.16: Madgets [Weiss et al., 2010a] are actuated physical widgets on multitouch tabletop.

Figure 2 .

 2 Figure 2.17: Tangible Bots [Pedersen and Hornbaek, 2011] (left) provide physically actuated controls to interact with multitouch tabletop surface, here used for musical creation . Thumbles [Patten Studio, 2014] (right) are robots that can be used as handles to interact with digital media, but also provide dynamic tactile feedback.

Figure 2 .

 2 Figure 2.19: BitDrones [Rubens et al., 2015] are flying interactive devices supporting direct manipulation, allowing room-scale tangible interaction .

Figure 2 .

 2 Figure2.21: The continua between compositeness and monolithicity (top), and between homogeneity and heterogeneity in TUIs (bottom).

Figure 3 . 1 :

 31 Figure 3.1: Chips (left) and pucks (right) as used in our study.

Figure 3 . 2 :

 32 Figure 3.2: The experimental Setup. Colors indicate areas -the surface was entirely white except for 3 whose boundaries were marked.

Figure 3

 3 Figure 3.3: A block consists in four consecutive, alternating between placement and sorting (Colors are assigned randomly).

Figure 3 . 4 :

 34 Figure 3.4: Visual targets for a spread placement task (left) and a packed placement task (right).

Figure 3 . 5 :

 35 Figure 3.5: Mean task completion times per condition (left), and within-subject ratios between chips and pucks, where values > 1 indicate an advantage for pucks (right). Error bars are 95% CIs.

Figure 3 . 6 :

 36 Figure 3.6: Mean task accuracies per condition (left), and within-subject ratios between chips and pucks, where values > 1 indicate an advantage for pucks (right). Error bars are 95% CIs.

Figure 3 . 7 :

 37 Figure 3.7: Questionnaire responses regarding subjective experience. Error bars are 95% bootstrap CIs.

Figure 3 . 8 :

 38 Figure 3.8: Frequencies of occurrence of manipulation gestures and manipulation errors, for Chips and for Pucks.

Figure 4 . 1 :

 41 Figure 4.1: SmartTokens are small-sized tokens supporting touch and motion sensing, and wireless communication.

Figure 4 . 2 :

 42 Figure 4.2: SmartToken hardware design

Figure 4 . 3 :

 43 Figure 4.3: Four different grips with a SmartToken.

Figure 4 . 4 :

 44 Figure 4.4: SmartToken wireless sensor network architecture, where each SmartToken communicates manipulation events to a coordinator.

Figure 4 . 5 :

 45 Figure 4.5: State-machine describing input events a SmartToken can emit.

  .6)., one for each notification type, and bought a SmartToken kit with tokens of three different colors (see Figure 4.7 right). Laura has been coding for three hours, during which three red tokens fell down (see 2 -4 in Figure 4.7). She barely noticed them, as he placed the dispenser out of her sight and a piece of foam absorbs most of the noise of falling tokens. Later a grey token falls in the wooden container (see 1 in Figure 4.7), indicating a blog comment and making a distinctive noise. Out

Figure 4 . 6 :

 46 Figure 4.6: Token dispenser (left) and Laura's notification setup (right).

Figure 4

 4 Figure 4.7: A short strip illustrating the mail notification system

Figure 5 . 1 :

 51 Figure 5.1: Zooids can be held as tokens, manipulated collectively or individually, behave as physical pixels, act as handles and controllers, and can move dynamically under machine control. They are building blocks for a new class of user interface we call Swarm User Interfaces.

  UI platform in open-source and open-hardware. In summary, our contributions are: A working definition for Swarm User Interfaces with several implemented examples,

Figure 5 . 2 :

 52 Figure 5.2: Freehand swarm drawing (1-3) and curve manipulation (4).

Figure 5 . 3 :

 53 Figure 5.3: Circle swarm drawing, where Zooids are automatically inserted (2) or discarded (3) depending on the circle's diameter.

Figure 5 . 4 :

 54 Figure 5.4: Exploded view of a Zooid.

Figure 5 . 5 :

 55 Figure 5.5: Software Architecture.

Figure 5 . 6 :

 56 Figure 5.6: Structure of local PID position control

Figure 5 .

 5 Figure5.7 gives an overview of the design space of Swarm UIs. They can be organized into an interaction aspect (interacting with one Zooid, control-

Figure 5 . 9 :

 59 Figure 5.9: Alien from the game Space Invaders from Taito (1978) and main character from the game Mario Bros by Nintendo (1983).

Figure 5 .

 5 Figure 5.10: A circle obtained by assembling 16 elements using (1) Bresenham's algorithm and (2) free object positioning.

Figure 6 . 1 :

 61 Figure 6.1: Multi-variate scatterplots implemented with Zooids allow to interact physically with data.

Figure 6

 6 Figure 6.2: A reproduction of Jacques Bertin physical matrices by Perin et al. [2015] (left). Physicalizations using colored wooden tiles by Huron et al. [2014c] (right).

Figure 6

 6 Figure 6.3: BMW Kinetic sculpture [ART+COM, 2016]

Figure 6 . 5 :

 65 Figure 6.5: Multiple juxtaposed scatterplot to represent and explore multivariate data.

Figure 6

 6 Figure 6.6: A line chart physicalization of the usage of the resources of a computer.

Figure 6

 6 Figure 6.7: A line chart physicalization of the temperature of Paris over a year.

Figure 6

 6 Figure 6.8: Examples of existing physicalizations (described in Section 6.2) classified according to our two dimensions, alongside our implementation using Zooids.

Figure 7 . 1 :

 71 Figure 7.1: Kinetica [Rzeszotarski and Kittur, 2014] allows users to manipulate directly multivariate data on multitouch devices, such as filtering using with a virtual blade.

Figure 7 . 2 :

 72 Figure 7.2: Zooids could be used as pedagical interactive tools to support teachers to explain and explore the functioning of the Solar System.

Titre:

  Interfaces Tangible Polyvalentes à Base de Collections d'Objets Mobiles de Petite Taille Mots clés : Interaction Tangible, Polyvalence, Polyvalence, Collections d'objets, Mobilité Résumé : Dans ce manuscrit, je présente mes travaux visant à rendre les interfaces tangibles plus polyvalentes et plus physiques afin de réduire l'espace entre le réel et le virtuel. Pour ce faire, j'étudie et conçois des dispositifs technologiques permettant d'interagir avec le monde numérique exploitant au mieux le potentiel de nos mains.

  In a third step, I introduce Swarm User Interfaces as a subclass of tangible user interfaces that are composed of collections of many interactive autonomous robots. To illustrate them, I present Zooids, an open-source open-hardware platform for developing tabletop Swarm User Interfaces. I demonstrate their potential and versatility through a set of application scenarios. I then describe their implementation, and clarify design considerations for Swarm User Interfaces. As a fourth step, I focus on the field of Information Visualization to illustrate the versatility of Swarm UIs. I define composite data physicalizations as a subset of physical visualizations, and implement several examples with Zooids. I finally draw conclusions from the presented work, and open perspectives and directions for future work.

  .1 Introducing Tangible User Interfaces . . . . . . . . . . . . . . . . 1.2 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Thesis overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tangible User Interfaces: History and Overview . . . . . . . . . 2.2 TUIs for various application domains . . . . . . . . . . . . . . . The SmartToken Input Model . . . . . . . . . . . . . . . . . . . . 4.5 Use Case: Tangible Notification Machine . . . . . . . . . . . . . 4.6 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . 4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 introducing swarm uis, versatile and composite tuis 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Swarm UI Examples with Zooids . . . . . . . . . . . . . . . . . . 5.4 Zooids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Composite Data Physicalization . . . . . . . . . . . . . . . . . . . 6.3 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Physicalizing using Swarm UIs . . . . . . . . . . . . . . . . . . . 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Limitations of the chosen approach . . . . . . . . . . . . . . . . . 7.3 Perspectives and future work . . . . . . . . . . . . . . . . . . . . 7.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	x	contents
		6 designing dynamic physical visualizations using
		swarm uis
		6.1 7 conclusion
		7.1
	2 background
	2.1 ix

2.3 Bringing actuation to TUIs . . . . . . . . . . . . . . . . . . . . . . 2.4 Discussion and Lessons . . . . . . . . . . . . . . . . . . . . . . . 3 studying manipulations of collections of small generic objects 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Experimental Procedure . . . . . . . . . . . . . . . . . . . . . . . 3.5 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Performance and Subjective Experience . . . . . . . . . . . . . . 3.7 Video analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 General Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Conclusion and Future work . . . . . . . . . . . . . . . . . . . . 4 small generic objects as input devices 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Background and Motivation . . . . . . . . . . . . . . . . . . . . . 4.3 SmartToken Design and Implementation . . . . . . . . . . . . . 4.4 5.5 Swarm UIs: Design Principles and Challenges . . . . . . . . . . 5.6 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . 5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Bishop's Marble Answering Machine represents voice messages with physical marbles. (Illustration by Matt Jones from Dragicevic and Jansen [2012].) . . . . . . . . Figure 2.3 Graspable User Interfaces [Fitzmaurice, 1996] use physical handles to control directly digital content, here to draw geometric figures. . . . . . . . . . . . . . .

	Figure 2.4	Ullmer and Ishii [2000]'s MVC interaction model for
		GUIs (left) and the MCRpd model for TUIs (right). . .
	Figure 2.5	The Suanpan is the Chinese abacus and allows to per-
		form mathematical operations (Image from Wikipedia
		[2016]). . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	Figure 2.6	Fishkin [2004]'s 2D taxonomy classifies TUIs accord-

made of 95,000 LEGO Bricks by the artist Sean Kenney. . . . . . . . . . . . . . . . . . Figure 1.6 Bit Planner [Vitamins, 2014] is a calendar using LEGO bricks to represent the schedule in a company. Each line of the calendar is a person (left), each brick represents half a day working on a project indicated by the color. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 1.7 PixelBot [Alonso-Mora et al., 2011] are robots capable of moving in synchrony and changing color. Here a collection of PixelBots displays a fish jumping out of the water. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.1 Aish [1979]'s architectural CAD system uses physical bricks to build 3D models. Physical assemblies (left) allow to create equivalent digital 3D models (right). . . Figure 2.2 ing to metaphor and embodiment. . . . . . . . . . . . . xi xii List of Figures Figure 2.

  .2 The experimental Setup. Colors indicate areas -the surface was entirely white except for 3 whose boundaries were marked. . . . . . . . . . . . . . . . . . Figure 3.3 A block consists in four consecutive, alternating between placement and sorting (Colors are assigned randomly). . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.4 Visual targets for a spread placement task (left) and a packed placement task (right). . . . . . . . . . . . . . . . . Figure 3.5 Mean task completion times per condition (left), and within-subject ratios between chips and pucks, where values > 1 indicate an advantage for pucks (right).

Error bars are 95% CIs. . . . . . . . . . . . . . . . . . . . Figure 3.6 Mean task accuracies per condition (left), and withinsubject ratios between chips and pucks, where values > 1 indicate an advantage for pucks (right). Error bars are 95% CIs. . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.7 Questionnaire responses regarding subjective experience. Error bars are 95% bootstrap CIs. . . . . . . . . . Figure 3.8

  State-machine describing input events a SmartToken can emit. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.6 Token dispenser (left) and Laura's notification setup (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.7

.1 SmartTokens are small-sized tokens supporting touch and motion sensing, and wireless communication. . . . Figure 4.2 SmartToken hardware design . . . . . . . . . . . . . . . Figure 4.3 Four different grips with a SmartToken. . . . . . . . . . Figure 4.4 SmartToken wireless sensor network architecture, where each SmartToken communicates manipulation events to a coordinator. . . . . . . . . . . . . . . . . . . . Figure 4.5

  5.3 Circle swarm drawing, where Zooids are automatically inserted (2) or discarded (3) depending on the circle's diameter. . . . . . . . . . . . . . . . . . . . . . . . Figure 5.4 Exploded view of a Zooid. . . . . . . . . . . . . . . . . . Figure 5.5 Software Architecture. . . . . . . . . . . . . . . . . . . . Figure 5.6 Structure of local PID position control . . . . . . . . . .

	Figure 5.7	Design Space explored with Zooids. . . . . . . . . . . .
	Figure 5.8

  . . . . . . . . . . . . . . . . . . . . . . . . . .

	Figure 5.10	A circle obtained by assembling 16 elements using (1)
		Bresenham's algorithm and (2) free object positioning.
	Figure 6.1	Multi-variate scatterplots implemented with Zooids
		allow to interact physically with data. . . . . . . . . . .
	Figure 6.2

A reproduction of Jacques Bertin physical matrices by

[START_REF] Perin | Diy bertin matrix[END_REF]

(left). Physicalizations using colored wooden tiles by

[START_REF] Samuel Huron | Constructing Visual Representations: Investigating the Use of Tangible Tokens[END_REF]

(right). . . . . . . . Figure 6.3 BMW Kinetic sculpture [ART+COM, 2016] . . . . . . . Figure 6.4 EMERGE [Taher et al., 2015](left) and in-FORM [Follmer et al., 2013] (right) allows to create physicalisations, here 3D barchart. . . . . . . . . Figure 6.5 Multiple juxtaposed scatterplot to represent and explore multivariate data. . . . . . . . . . . . . . . . . . . . Figure 6.6 A line chart physicalization of the usage of the resources of a computer. . . . . . . . . . . . . . . . . . . . Figure 6.7 A line chart physicalization of the temperature of Paris over a year. . . . . . . . . . . . . . . . . . . . . . . Figure 6.8 Examples of existing physicalizations (described in Section 6.2) classified according to our two dimensions, alongside our implementation using Zooids. . . Figure 7.1 Kinetica [Rzeszotarski and Kittur, 2014] allows users to manipulate directly multivariate data on multitouch devices, such as filtering using with a virtual blade. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 7.2 Zooids could be used as pedagical interactive tools to support teachers to explain and explore the functioning of the Solar System. . . . . . . . . . . . . . . . . . .

https://opensource.org/

Positioning my approach within existing TUIs

From the previous survey, I identify different physical properties of TUIs in Figure 2.20. I oppose attributes, for the scope of this work, defined as follows.

Purely physical: A TUI using no other input or output modalities than physical objects.

Hybrid: A TUI using supplementary input and/or output modalities to physical objects.

Monolithic:

A TUI composed of a single element.

Composite: A TUI composed of at least two independent elements.

Homogeneous:

A TUI composed of identical elements.

Heterogeneous: A TUI composed of elements of different kinds.

Dynamic: A TUI supporting self actuation of its composing elements.

Static:

A TUI not supporting self actuation of its composing elements. This partial taxonomy of TUIs represented by Figure 2.20 is simplified. The dimensions are orthogonal, representing the full taxonomy would thus result in a combinatorial explosion. I am only using this partial tree to situate my approach. Physical properties are opposed in this binary tree for clarity