Jeffrey Valino Koh

Kasun Karunanayaka

Ryohei Nakatsu

Jeffrey Tzu Kwan

Valino Koh

Linetic: Technical, Usability and Aesthetic Implications of a Ferrofluid-Based Organic User Interface

Keywords: Electromagnetic, haptic, TUI, OUI, morphable, shape changing

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Recent progress in human-computer interaction has paved the way for a new body of research known as Organic User Interfaces (OUIs) [2] [10]. This field focuses on the need to further explore possibilities for interactive user interfaces using the advances made in electronics and material sciences [START_REF] Ishii | Tangible bits: designing the seamless interface between people, bits, and atoms[END_REF]. OUIs are defined by three factors: the input interface and output display should be one and the same; the form of the object should change continuously and correlate directly with the function it embodies, and finally the function performed by the object depends on how the physical shape of said object is changed [START_REF] Vertegaal | Introduction (special section on organic user interfaces)[END_REF].

OUIs open a path to morph the shape and form of the actual computer interface itself. The interface can now be a piece of fabric, a plastic card, liquid, sand, clay or any other material. OUIs enable users to interact with the interface by manipulating the natural qualities of these materials such as bending, stretching, pulling, stroking, etc. This new paradigm moves away from the traditional approach of metaphors and physical objects as defined by Tangible User Interfaces (TUIs), and explores next generation interfaces focused on the analog, continuous and transitional nature of physical reality and human experiences, as Schwesig points out [START_REF] Schwesig | What makes an interface feel organic?[END_REF].

With advances in technology pushing the boundaries in regards to the materials used to create ubiquitous interactive systems [START_REF] Weiser | The computer for the 21st century, Humancomputer interaction: toward the year[END_REF], it is now possible to expand the computer into the everyday environment through softer, and flexible formats [START_REF] Bailly | Gesture-aware remote controls: guidelines and interaction technique[END_REF]. Using these materials and technologies, this paper presents the implementation of an innovative OUI system based on liquid. This interface explores the potential of liquids as an interface and display device, where the manipulation of liquid becomes both the input and output. Linetic (Link to youtube videohttp://www.youtube.com/watch?v=V5EMyRsLLeQ) can provide the user with a natural and fluid experience where three-dimensional, tangible interaction takes place. Building on the idea of a ferrofluid display created by Sachiko Kodama [START_REF] Kodama | Dynamic ferrofluid sculpture[END_REF], Linetic provides an input/output solution based on ferrofluid. Ferrofluid is essentially a liquid that reacts to magnetic fields. The system is composed of a pool of ferromagnetic liquid combined with a sensing and actuation mechanism. The sensing uses an array of Hall Effect sensors that measure the density of a magnetic field, while actuation is produced by an array of electromagnets. Sound generation using a MAX/MSP patch running on a connected server augments the output experience.

Wearing a set of magnetic rings, the user can interact with the ferrofluid. The magnetic ring position is detected by the array of Hall Effect sensors, which in turn actuates the electromagnets and the sound server. The magnetic field of the active electromagnets produces the morphing of the ferrofluid to create transitional physical buttons in conjunction with the gesture, which then generates a sound. At the same time the pulse of the matching polarity electromagnets produces a force feedback vibration on the rings, giving the user haptic feedback.

Through natural movements of the hand, the interface is able to morph from a twodimensional surface to a three-dimensional form fluidly and dynamically. Using the shape changing quality of ferrofluids, we were able to study how liquids could be-come a novel form of OUI. In sum Linetic provide a tangible, multi-touch interface with haptic feedback that produces a real 3D morphing surface.

Background

Linetic builds on research by both scientists and artists in creating a malleable, threedimensional, multi-touch interface. Hiroshi Ishii's seminal research in TUIs offers insight for natural user interfaces in which this project builds upon [START_REF] Ishii | Tangible bits: designing the seamless interface between people, bits, and atoms[END_REF]. Ishii later went on to categorize generations of TUIs, leading to the development of guidelines for the next iteration of user interfaces, known as Organic User Interfaces [START_REF] Holman | Organic user interfaces: designing computers in any way, shape, or form[END_REF] [10] [START_REF] Wakita | Programmable blobs[END_REF].

Research into the field of OUIs is still in its infancy. In 2008 ACM published a groundbreaking collection of essays that collected the works of OUI's pioneering researchers in a special issue of Communications of the ACM. In this special issue, several key authors outlined various aspects and characteristics that make an OUI:

Input Equals Output: In the graphical user interface (GUI) there is a clear division of input and output. The mouse and keyboard input actions from the user. Based on those actions, output is generated graphically on the screen. A key feature of OUIs is that a piece of Organic light-emitting diode paper, or any potentially non-planar object for that matter, is meant to input actions from the user and also outputs them onto the same object.

Function Equals Form: The form of an object clearly determines its ability to be used as an input. The statement Function Equals Form emphasizes this dependency on one another. Holman and Vertegaal [START_REF] Holman | Organic user interfaces: designing computers in any way, shape, or form[END_REF] argue that these two are in fact inseparable and that it is a mistake to try to deny this in any way.

Form Follows Flow: This principle states that it is of utmost necessity for OUIs to negotiate user actions based on context, e.g. the ubiquitous 'clamshell' phone, where incoming calls alter the phone's function when opening the phone during an incoming call.

In regards to the material we choose to use for our input and output, the ferromagnetic art installations by Sachiko Kodama provide an aesthetic viewpoint of how the power of fluid and transient shapes can capture the imagination of viewers [START_REF] Kodama | Dynamic ferrofluid sculpture[END_REF]. In her projects ferromagnetic fluids are used to create organic shapes that change structure dynamically. Adjusting the power of the electromagnets actuates the ferromagnetic fluid in varying ways. The magnetic field produced by the electromagnets controls the movement of the ferrofluid, producing a visual output. Linetic builds on this work, providing the user with a means to directly interact with the ferromagnetic fluid and the formation of shape changing buttons, while inheriting the aesthetic and kinetic qualities that Kodama's pioneering works offer.

Other ferrofluid interfaces include SnOil, which is a controllable display that allows the user to interact with ferromagnetic fluids by tilting the display [START_REF] Frey | Snoil. a physical display based on ferrofluid[END_REF]. The display uses a grid of electromagnets to actuate ferromagnetic fluid in selected areas, and a built-in tilt sensor is used to activate the electromagnets. Actuation of the ferrofluid is binary, off or on, depending on the respective electromagnet state. Linetic expand upon SnOil by offering a novel means of interaction with ferrofluid by directly inte-racting with the material and provides an analog-like and linearly variable state of the actuated fluid, beyond simply a binary on/off state.

MudTub by Tom Gerhardt [START_REF] Gerhardt | Mud tub[END_REF] is an intriguing, quasi-OUI that exploits the material characteristics of mud as an input material, but fails to follow the contemporary definition of OUIs as outlined by Holman and Vertegaal [START_REF] Holman | Organic user interfaces: designing computers in any way, shape, or form[END_REF], in the sense that the interface medium and output display are not entirely the same. MudTub uses projection to augment the output and provide input. While offering a morphable computer interaction material in the form of mud is novel, our implementation offers more fluid-like properties, a higher degree of controllability, adheres to the tenants of OUIs as outlined by Holman and Vertegaal, and is not nearly as messy.

MudPad [START_REF] Jansen | Mudpad : Fluid haptics for multitouch surfaces[END_REF] is a haptic multi-touch input device that is malleable and can be controlled for localized haptic feedback. It acts similar to Linetic but the materials and methodology of actuation is different. The interaction material is also contained within a bladder and users do not directly interact with the material. Linetic offers the user direct interaction with the display and input material, making it a more compelling example of an OUI.

Programmable Blobs [START_REF] Weiser | The computer for the 21st century, Humancomputer interaction: toward the year[END_REF] is a compelling example of a user interface that uses malleable, programmable matter. It differs from Linetic in that it still uses a quasisolid material compared to the much more fluid attributes of ferrofluid in the Linetic system. Latency during transformation of the Programmable Blobs is also much slower when compared to the speed of Linetic.

All the examples discussed above provide an invaluable and important basis for the research conducted for Linetic. Linetic attempts to expand the field of TUIs beyond solid and static-shaped object manipulation, bridging the gap between physical user actuation and animated representation into a unified input/output device.

System Description

A system diagram is shown in Figure 2 and the first version of the Linetic system is shown in Figure 1. By moving fingers over the ferrofluid buttons, users can manipulate shapes in the liquid in order to interact with the system. The system identifies a magnetic flux change when a finger moves perpendicular and vertically in relation to the surface. The flux change is detected by the Hall Effect sensors embedded beneath the surface of the ferrofluid, which sense the magnetic field produced by the magnets worn on the fingers. In turn, an electromagnet also contained beneath the surface of the ferrofluid is activated and produces a field in which the ferrofluid reacts to, producing a ferrofluid button. The distance between the finger and the ferrofluid button is mapped to the pressure and intensity of the click. Haptic feedback is felt by the user through the natural vibrations of the magnet worn on the fingertips. In the following section we will describe both the hardware (sensing and actuation) and software design of the system in detail. In essence, the Linetic system consists of a set of controller circuits to control the flow of the current, an array of Hall Effect sensors to detect user interaction, an array of electromagnets to actuate the ferrofluid, a ferrofluid pool, a MAX/MSP sub-system for sound generation and a software application programming interface (API).

Hardware Design

In the following section, we will outline the various hardware sub-systems of the Linetic system. Hardware sub-systems include the sensing method, the actuation accessory, the electromagnetic array and circuit.

Fig.3.

Interacting with version two of the Linetic system using the wearable accessory.

Sensing. The hardware for sensing is performed by magnetic Hall Effect sensors manufactured by Hamlin Electronics. Hall Effect sensors measure the density of a magnetic field. The sensor produces an analog output when it is under the influence of any magnetic field. The sensor output is connected to the analog-to-digital conver-ter of the micro controller used. The sensor is placed directly on top of each electromagnet while the user's finger carries a powerful neodymium magnet with like poles of each magnet facing one another. When the electromagnet is turned on, the sensor output becomes fully saturated. If a neodymium magnet of the same pole is brought close to the surface, the sensor output drops and this is detected as the presence of the user's finger. Voltage drop of the sensor is dependent on the strength of the external magnetic field. Therefore, a sensor voltage drop is much higher when the distance between the finger and the ferrofluid display is reduced. This measurement is used as an input to control the height of the spikes, which are a product of the system.

Actuation Accessory. The actuation of the system, as seen in Figure 3, takes place when the user moves their fingers across the ferrofluid surface while wearing a magnetic finger accessory. To design this accessory, we looked at various accessories musicians wear on their fingers, such as guitar picks, to give us ideas on how to best design an accessory that acted as a natural extension of the finger. Achieving this kind of interaction was important for us so we decided to design a ring that allows the users to wear and play with the system. We also developed a wand that can be waved over the surface in order to provide an alternative method of interaction with the system but in this manuscript we will focus on the wearable configuration. We designed a ring that could be worn on the index finger of each hand. Figure 4 shows some of the design iterations. The rings contain a magnet that allows users to actuate the ferrofluid through natural gestures and finger movements. The ring has a cylindrical base made of plastic with a metal, coned shaped tip containing a magnet. When the users play with the system, they can sense a subtle haptic feedback through magnetic repulsion. This enables users to judge the distance from the liquid surface without touching it. This is important since when the user gets too close to the fluid, the liquid will become attracted to the metal cone. To avoid this, the haptic feedback enables the user to use the fluid interface without wetting or staining the fingers. Furthermore, electromagnet polarity is configured in such a way as to repel the magnet on the user's finger, once again subtly directing the user to not place her hands too close to the liquid.

Actuation. The screen consists of ferrofluid placed in a shallow acrylic container. This container is placed on top of an array of electromagnets. These electromagnets, when turned on, actuate the fluid, causing it to form bumps and/or spiky buttons depending on the strength of the magnetic field produced. The system is powered using an array of ATX power supplies. These power supplies are then fed to a current limiting circuit. The output current from the cur-rent limiter circuits is distributed to the magnet driver board consisting of full-bridge drivers. By feeding the microcontroller's pulse-width modulation (PWM) signals to the magnet drivers, the electromagnetic field produced by the electromagnet array is controlled. This electromagnetic field creates spikes in various shapes and sizes on the surface of the ferrofluid display.

Software Design

In the next section, we will outline the various software sub-systems of the Linetic system. Software sub-systems include the Linetic OUI Framework, sound generation and the circuit firmware. Software sub-systems include the Linetic OUI Framework, sound generation and the circuit firmware.

Linetic OUI Framework. One of the objectives of the research is to develop a software framework for the project. It consists of a system background service and a Java API extension. The framework hides the complexity of the overall hardware and lowlevel software code from the developer and provides a convenient way to plug different sub-systems into the Linetic system. Therefore the Linetic system could be configured as a tangible input device, a tangible display, or perform both operations to the external system simultaneously. Various kinds of external systems could communicate with the Linetic system through the Linetic OUI framework API.

Sound Generation. In this application we attached Linetic to an instrumental music generation program developed in the MAX/MSP environment. The Max/MSP patch that is used detects incoming signals from the framework and activates appropriate midi piano keys to create sound.

Circuit Firmware. The circuit firmware is written in C and programmed into the microcontroller within the circuit. It communicates with the PC through USB interface and receives information regarding ferrofluid actuation patterns. According to the given information, it dynamically actuates the ferrofluid and creates dynamic patterns in the ferrofluid display by changing the current of the electromagnet grid underneath the ferrofluid container.

Experiments & Results

The previous work of authors includes a series of experiments to measure spike height versus current, distance of two adjacent spikes, transient state of the system and the static linearity of the system [START_REF] Bailly | Gesture-aware remote controls: guidelines and interaction technique[END_REF]. These experiments demonstrated the linearity of the control mechanisms, which are used in the Linetic system. Since magnetic systems are highly nonlinear, these experiments represent a significant step in controlling ferrofluids. The following experiments explore the control mechanisms of the system. The point of this experiment is to understand the Hall Effect sensor readings in relation to the vertical distance of the actuator (in this case, the finger-worn actuation ring) from the surface. This experiment has been conducted using a Hall Effect sensor and an electromagnet, which generates an average flux density on the surface from 450 to 1950 Gauss for the range of 6V to 24V with 1.9A to 7.5A of electrical current.

Experiment 1: Hall Effect Sensor Reading Versus Vertical Distance

In the experiment we kept the power of the electromagnet at a constant voltage of 10V and a driven current of 2.44A, with the sensor on the vertical axis on top of the electromagnet. The sensor reading is measured versus the distance to the electromagnet. The value of the sensor output voltage taken is the mean value in one second. This plot shows (Fig. 5) that the sensor is most sensitive with respect to the vertical distance from 0cm to 3cm. When the distance is greater than 3cm, the change in output is much lower. At larger distances, for example the values of 6cm and 7cm, the difference in voltage is only 0.011 volts. However such a small voltage difference is not detected by the microcontroller used for this iteration of the system. Therefore, the sensor cannot detect motion from 6cm to 7cm, but it can detect longer distance motion, e.g. moving from 6cm to 8cm.

Experiment 2: Hall Effect Sensor Reading Versus Horizontal Distance

This experiment is similar to this first experiment, but looks at horizontal distance as opposed to vertical distance. Once more keeping the power of the electromagnet constant, the sensor is placed on the vertical axis of the electromagnet at 2cm, since at this distance the sensor is most sensitive, registering the largest change in values with respect to the distance moved. An oscilloscope was used to measure the voltage output of the sensor. Experiment 1 shows that the resolution of the system cannot distinguish any smaller changes. Within 0.02 volts, the magnetic field at 3.5cm and beyond is too small to cause a change in the microprocessor. This experiment shows that the magnetic field from the horizontal area of the magnet is too small to be detected at the optimal vertical distance. This is advantageous for the system since the tracking system gives an almost perfect horizontal reading without external noise from the neighboring magnets.

Experiment 3: Characterization of Magnetic Hall Effect Sensor Readings Under the Influence of Multiple Magnetic Fields

In this experiment the readings of the Hall Effect sensor are measured to determine the influence of the magnetic fields generated by the electromagnets and neodymium magnets. These readings will provide the appropriate parameters for the microcontroller firmware.

The electromagnet is connected to a power supply, supplying 10V and up to 2A of current. Its output magnitude is controlled by a Pulse-Width Modulated (PWM) input to the driver circuit. It is positioned such that the field of the North Pole is directed vertically upwards and the South Pole vertically downwards. The sensor is supplied the rated of 5V and is positioned such that it is in level with the top of the electromagnet and directly next to it. The reading of the steady-state output voltage of the sensor is recorded using the oscilloscope, while varying the height and direction of the neodymium magnet and the PWM input to the electromagnet. First the default sensor value is taken. Without the neodymium magnet or electromagnet influence, the reading is 2.50V. Next, with the neodymium magnet pole at South Pole (facing down), the PWM values and distances are measured. Here (Fig. 7) the strength of the electromagnet's field serves to decrease the reading of the sensor, whereas the position of the neodymium magnet field serves to increase the reading of the sensor. This results in a case where the value of the sensor is unable to detect the presence of the neodymium magnet due to the electromagnet's field. From the data we gathered, this occurs in the case when the distance of the neodymium magnet is 7.0cm. If the electromagnet is off, the reading is 2.53V, but if the electromagnet is turned on, the reading falls below the 2.50V neutral value. To circumvent this problem, we could use like poles instead of unlike poles, i.e. the North side of the neodymium magnet facing the North side of the electromagnet. This causes both magnets to boost each other's readings, which can be compensated for by changing the sensitivity in the software. This approach has the peripheral advantages of preventing the two magnets from attracting each other and preventing the neodymium magnet from picking up the ferrofluid contained on the surface of the system. It also provides a means for haptic feedback while using the system.

User Interaction Methodology

In regards to the user interaction methodology, we attempted to simulate some interactions that reflect both the aesthetics and playfulness of interacting with fluid. Although the dramatic effects of water as a tactile surface has been previously explored [START_REF] Pier | Using water as interface media in vr applications[END_REF], no precedent in regards to the interaction with ferrofluid, which has distinct fluid characteristics, has been recorded. Due to this, an adaptation of previous methods as well as trial and error was employed. Due to the messy nature of ferrofluid as well as the nature of sensing inherent to the system, we decided to employ a finger accessory that allowed users to interact with the fluid without actually touching it. This reduced the methods of interaction to that if simple mid-air gestures, some of which are employed in [START_REF] Bailly | Gesture-aware remote controls: guidelines and interaction technique[END_REF]. These gestures included waving and tapping. Like-polarity between the surface and fingertip magnets allowed a force resistance that made tapping a particularly compelling interaction.

Later on, we decided to remove the haptic subsystem in order to concentrate on the visual and audio effects of the system. This led us to the third iteration of the Linetic system.

Aesthetic Interactions with Linetic

Although efficiency and speed are two important characteristics for most technical devices, research suggests that these attributes are not sufficient to fulfill all of a user's needs [START_REF] Dunne | Hertzian Tales. Electronic Products, Aesthetic Experience, and Critical Design, volume 2nd[END_REF] [START_REF] Korsmeyer | affective computing[END_REF]. Aside from the technical characteristics of the device, a user needs a series of other characteristics that enrich the user experience of an artifact, device or technology. People search for these characteristics that allow exploration and a holistic experience of an artifact instead of just basic features. Through Linetic, we hope to tap into these deeper desires and human emotions in order to be able to produce a more enriching, aesthetic experience of the system. The aesthetic experience refers to the pragmatic aesthetics that Dewey [START_REF] Dewey | Art as experience[END_REF] and Shustermann [START_REF] Shusterman | Pragmatist aesthetics; living beauty, rethinking art[END_REF] describe, as well as the guidelines that Petersen and others have developed for Aesthetic Interactions [START_REF] Petersen | Aesthetic interaction[END_REF] from the pragmatist aesthetics viewpoint.

Fels [START_REF] Fels | Intimacy and Embodiment: Implications for art and technology[END_REF] states that for an individual to have an aesthetic experience there needs to be an intimacy that is built between the person and the object. Fels believes that when people are able to manipulate an object skillfully or intimately, this produces an aesthetic experience. He describes four types of relationships that one is able to have with an object and these are as follows: the person communicates with the object; the person embodies the object; object communicates with person; object embodies the person. Through Linetic we are able to achieve the first three of these experiences: person communicates with object; object communicates with person; the person embodies the object.

First, the user is able to communicate with the system in a direct and natural way through touch. Interacting with the liquid, the user can start a dialogue in which the system responds to the user. The still, dark pool of ferrofluid, with its reflective surface like a black mirror, has the ability to draw people to it in the same way artist Richard Wilson [26] was able to capture the imagination of his audience with his installation 20:50 -a room filled with thick sump oil. According to Khaslavsky et al. [START_REF] Khaslavsky | Understanding the seductive experience[END_REF], the first step to seduce your audience is to entice them in order to grab their attention. It is the very liquid quality of ferrofluid that makes it a material that probes curiosity and encourages interaction. This curiosity naturally leads to one wanting to touch the liquid and explore it.

The Linetic system possesses the playful characteristics needed for learning by playing, which is a learning mechanism for most people. The system lends itself to playful exploration and thus a relationship with the system can begin. Unlike a solid tangible interface, liquid has the ability to take the shape of its container and allows for the free maneuvering of itself. As people move their fingers above the liquid they begin to learn quickly how to sculpt and manipulate the liquid, creating three dimensional, liquid sculptures combined with sound that respond to the user's tactile exploration. This seemingly magical quality of being able to sculpt ferrofluid has been the main spark of inspiration that motivates us to exploit this interaction.

Aesthetic interaction suggests that the use of the system gives it meaning and creates an aesthetic experience [START_REF] Pier | Using water as interface media in vr applications[END_REF], the appropriation of Linetic by the user and the freedom for improvisation makes this a platform for creative expression. Here the user is able to intelligently construct music through the simple and ordinary actions of playing with liquid, making the natural into a magical experience, thus capturing their imagination.

The third way in which a person can relate to an object is when the object communicates to the person; there is no interaction and it is much like how we enjoy a painting. Even at this level Linetic has the ability to engage the audience. The liquid, which seems to have a life of its own, can mesmerize even non-interacting audiences and this quality was evident in the works of Sachiko Kodama. With Linetic, the audience can reach out to the liquid and with the movement of their fingers, can engage with this liquid as a piano player would with piano keys. This allows the user to en-gage in a more intimate way with the interface as Fels defines intimacy, and allows the user to use this interface as an extension of them.

One of the questions we wanted to address with this project was how we can keep audiences engaged in the experience, and help them take away a meaningful experience. Music or art that has stood the test of time are examples of works where the creator is able to captivate the audience even after decades from when the work was produced [START_REF] Locher | A framework for aesthetic experience[END_REF]. Many interactive systems today have not yet been able to capture users in the same way. People easily adapt to new technologies and once they understand how to use it, it can or frequently looses its novelty. Liquid is a material that people from different disciplines continue to study to understand its powerful effect on our body, mind and beyond. Liquid as an interface for computing opens the possibility of a whole new way of how we understand and engage with the computing world. The exact ways in which we might use liquids in future interactive systems is still to be explored and Linetic is a beginning of this exploration.

Characterizing Linetic as Disposed to Analogness or Digitalness

Using the method offered by Koh et al. [START_REF] Koh | Uncovering analogness and digitalness in interactive media[END_REF]. we attempted to situate Linetic in the Analogness-Digitalness Continuum (ADC), by comparing the system with the characteristics for interactive systems offered in the research. Table X shows a breakdown of how Linetic scored in regards to its Analogness & Digitalness.

Characteristics of Analogness & Digitalness Affirmative

The Interface is Intuitive In Analysis of how Linetic scored, we found that our current version of the interaction required some training before being able to use the system and thus did not meet the requirements of an initiative interface.

However Linetic did meet the requirements for a continuous interface where media and content are a singular embodiment, as the morphable ferrofluid used in our interface is both a visual component of information feedback to the user (in addition to sound and formally, haptics), as well as the interface where the user interacts with the system.

Curiously depending on the configuration of Linetic, the interaction characteristics could become discreet. This is mainly due to the possibility to use Linetic as an interface device to an external system.

In conclusion, we find Linetic to lean towards the Analogness side of the continuum, with hybrid-like qualities depending on the configuration of the system.

Future Work

Future work for this project involves efforts in several directions: exploration of new materials, new array configurations, and improvements in the methodology of gestural interactions and most importantly, the development of relevant applications in order to explore the full potential of human-computer interaction using liquids as a medium. The search for new materials for different effects and increased controllability involves experimentation with materials such as electrorheological (ER) and magnetorheological (MR) fluids. These materials will provide another avenue for OUIs and the Linetic project.

Refinement of the electromagnet array is also important so as to create a more finegrained and higher-resolution display. The use of other magnet arrangements will also enable us to control electromagnetic fields better, so as to create more complex and intricate shapes, which will enable us to explore more real-world applications.

An effort to create a concise library of gestures that are more intuitive for use with the system is also desirable in order to disseminate a methodology in which other researchers can base new research on. Once a concise gestural library is created the project will focus on developing and executing more user studies in order to fine-tune the usability of the system. Finally an integrated development environment (IDE) for direct programming of Linetic is desired so that the technology can be shared internationally, which can foster application development in the academic and commercial markets.

Conclusion

In this manuscript we describe new iterations of the Linetic system that uses the Hall Effect as a novel form of interaction and actuation. By using Hall Effect sensing in combination with a new accessory worn on the fingertips, users avoid messiness and staining, which was a previous downfall of the original system.

We also outline in this paper, three new experiments that enabled us to develop the new iterations of the Linetic system. These include findings regarding the relationship between perpendicular distances of the Hall Effect sensor and the magnetic field generated by the electromagnetic array embedded in the surface of the system, the characterization of the magnetic Hall Effect sensor readings under the influence of multiple magnetic fields, and the relationship of distance from the sensor versus PWM.

A preliminary user study was also performed in order to understand the usability of the system, as well as to inform the authors on the possible challenges of evaluating such a system for use as an interface and display device. We also discuss the addition of haptic feedback facilitated by the vibration of the magnets placed on the fingertips, providing an additional modality of feedback, along with the previously developed modalities of sound and visual output. This new actuation accessory provides a means for users to interact with the Linetic system without the need to actually touch the ferrofluid, and still provides instantaneous tactile feedback.

Finally we also discuss the aesthetic implications in regards to the user's experience when using the Linetic system. The aesthetic implications of using liquid as a means for human-computer interaction are an important topic in regards to our research. We felt a need to outline these implications in order to accurately prepare for future liquid-based interactive systems.

Fig. 1 .

 1 Fig. 1. Version one of the Linetic system.

Fig. 2 .

 2 Fig.2. A System diagram of the Linetic system.

Fig. 4 .

 4 Fig.4. Design iterations for the wearable component. The final design is still being developed.

Fig. 5 .

 5 Fig.5. Sensor output versus vertical distance.

Fig. 6 .

 6 Fig.6. Hall Effect sensor output versus horizontal displacement.

Fig. 7 .

 7 Fig.7. Sensor reading values obtained for different distances versus PWM.

Fig. 8

 8 Fig.8 Version Three of the Linetic system without the splash cover.

Table 1 . A table showing how Linetic scored in terms of its disposition to Analogness or Digitalness.

 1

	Interaction is Continuous	✓
	Content and Media are Singular in Embodiment	✓
	Content and Media are Dichotomistic in Embodiment
	Interaction is Discrete	✓
	The Interface is Mediated
	Hybrid Candidate?	✓

Acknowledgement

This research is supported by the Singapore National Research Foundation under its International Research Center Keio-NUS CUTE Center @ Singapore Funding Initiative and administered by the IDM Program Office.


Wei Deng
email: dengwei@whu.edu.cn

Lingda Wu

Ronghuan Yu

Jiazhe Lai

On-line Sketch Recognition Using Direction Feature

Keywords: Sketched symbol recognition, NicIcon database, multi-stroke shapes

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Sketching is a natural form of human communication. Sketch-based interaction is a fast and efficient means of capturing information by automatically interpreting handdrawn sketches and can be an important part of the early design process, where it can help people explore rough ideas and solutions in an informal environment. Sketch recognition refers to the recognition of predefined symbols or free-form drawings(e.g., an unconstrained circuit drawing); in the latter case, the recognition task is generally preceded by segmentation in order to locate individual symbols. This paper focuses on the recognition of hand-drawn isolated symbols. With the growing popularity of devices with touch screens, there is increasing interest in building sketch-based user interfaces. However, many challenges remain in terms of intra-class compactness and interclass separation due to the variability of sketching. Because it is likely that different people will have different drawing styles, such as the stroke-order, -number, and nonuniform scaling , as well as complex local shifts. Moreover, the style of the same individual at different times may differ. A good recognition algorithm should place few drawing constraints on users. Related research is that of handwriting recognition, such as handwritten digit and Chinese character recognition with many effective algorithms.

This paper presents a new approach to sketched symbol recognition based on the direction feature. It is insulated from stroke order and number. We also present two feature representations, which obtain good experimental results in the NicIcon database.

Related Work

One common approach to sketch recognition focuses on building structural shape descriptions. Its basis step is stroke segmentation and primitive recognition using temporal and spatial features. A sketched symbol can be represented as a tree or graph and the similarity between two sketches can be calculated by structural matching.

Hammond and Davis [START_REF] Hammond | LADDER, a sketching language for user interface developers[END_REF] developed a hierarchical language to describe how diagrams are drawn, displayed, and edited. They used this language to perform automatic symbol recognition. The attributed relational graph (ARG) is an excellent statistical model to describe both geometry and topology of a symbol [START_REF] Lee | An Efficient Graph-Based Recognition for Hand-Drawn Symbols[END_REF], and is insensitive to orientation, scaling, and drawing order. The advantage of structural methods is distinguishing similar shapes. Therefore, high accuracy in stroke segmentation and primitive recognition is necessary. An alternative approach looks at the visual appearance of shapes and symbols [START_REF] Ouyang | A visual approach to sketch symbol recognition[END_REF] instead of the relation between geometric primitives. Ouyang and Davis [START_REF] Ouyang | A visual approach to sketch symbol recognition[END_REF] proposed a visual approach to sketched symbol recognition, which is similar in spirit to our proposed method. It used a set of visual features that captured on-line stroke properties like orientation and endpoint location. Almazan et al. [START_REF] Almazan | A non-rigid appearance model for shape description and recognition[END_REF] and Escalera et al. [START_REF] Escalera | Blurred shape model for binary and grey-level symbol recognition[END_REF] described a framework to learn a model of shape variability based on the Active Appearance Model (AAM), and proposed two types of modified Blurred Shape Model (BSM) descriptors(DBSM and nrBSM) as basic shape features to learn the model. Eitz et al. [START_REF] Eitz | How do humans sketch object?[END_REF] described a large scale exploration of human sketches. They analyzed the distribution of non-expert sketches of everyday objects and developed a bag-of-features sketch representation. Willems et al. [START_REF] Willems | Iconic and multi-stroke gesture recognition[END_REF] explored a large number of on-line features, which were sorted in three feature sets due to different levels of details. Recently Delaye and Anquetil [START_REF] Delaye | HBF49 feature set: A first unified baseline for online symbol recognition[END_REF] presented a set of 49 features, called HBF49, for the representation of hand-drawn symbols for use as a reference for evaluation of symbol recognition systems.

This paper presents an on-line sketched symbol recognition method using the direction feature. The key contributions of our method are:

1. A framework to recognize sketched symbols using the direction features. It is insensitive to the variation in the order and number of strokes. 2. Two feature representations. They are statistical vectors rather than structural representations of geometric primitives. This allows it to be robust to different drawing styles. 3. Obtaining high recognition accuracy in the NicIcon database using Support Vector Machine (SVM) classifier in real time.

Our Approach

We designed our approach primarily using the direction properties of symbols. An overview of the recognition process is shown in Fig. 1. First, preprocess the input trajectory, including resampling. Then the direction of every point is decomposed to four directions, horizontal, vertical, diagonal, and anti-diagonal. Feature representation is performed using the grid and direction feature. Finally, an SVM classifier is trained to perform classification.

3.1

Preprocessing.

The preprocessing of the input sketch directly facilitates the pattern description and affects the quality of the description. We choose to normalize patterns in scale and translation, and to apply a trajectory resampling strategy. These operations are simple to perform and guarantee a better stability of extracted features, for any type of input pattern. Since on-line strokes are typically sampled at a constant temporal frequency, the distance between neighboring points varies based on the pen speed. This produces more sampled points where the pen is typically slower. In order to make feature extraction more reliable, we resample each stroke at a constant spatial distance. In our experiments the sampling interval is set to one. Next we remove differences of scale and translation. The coordinates of stroke points are simply shifted and scaled such that all points are enclosed in a standard box. In experiments we set x,y Î [0,100]. It means translating maximal dimension of a symbol to 100 with aspect ratio preserved [START_REF] Liu | Handwritten digit recognition: investigation of normalization and feature extraction techniques[END_REF]. We do not translate the on-line trajectory to image (such as in [START_REF] Ouyang | A visual approach to sketch symbol recognition[END_REF]). Therefore direction features are accurate with the absence of jagged edge.

Decomposition of direction

The so-called direction feature is widely used in character recognition. Similar to Ouyang and Davis [START_REF] Ouyang | A visual approach to sketch symbol recognition[END_REF], we decompose direction vectors of resampling points into four directions, at 0, 45, 90 and 135 degrees. If the direction of a point lies between two standard directions, it is decomposed into two components in the two standard directions, as shown in Fig. 2. One major advantage of this representation is the independence of stroke direction. A stroke drawn left to right is the same as one drawn right to left. This is essential for free hand-drawn symbols.

Grid generation

Grid is used to blur the features, for directly overcoming the local non-uniform variability. We learned that blurring should be considered as a means of obtaining latent dimensions(subspace) rather than as a means of reducing computational cost, though the effects might seem similar [START_REF] Fujisawa | Forty years of research in character and document recognition-an industrial perspective[END_REF]. Moreover, the mesh size of 8x8 used in statistical approaches was determined by the optimum blurring parameter in light of the Shannon sampling theorem [START_REF] Fujisawa | Forty years of research in character and document recognition-an industrial perspective[END_REF]. We refer to the grid generation by Almazan et al. [START_REF] Almazan | A non-rigid appearance model for shape description and recognition[END_REF]. First, we calculate the geometric centroid of all resampling points. Then the region is split into four sub-regions. Therefore, the points are split into four sets. Next, in each sub-region the iterative procedure works until we get a grid of size 8x8, as shown in Fig. 3.

Feature representation

The representation schemes of the input sketch and the model database are important since the classification method depends strongly on them. We present two feature representations. The size of the grid is 8x8, and the sub-direction feature is S={x i , y i , dir 1i , dir 2i , dir 3i , dir 4i }, i=1,2,…,n, where n is number of resampling points; {x i , y i } is coordinate of point P i ; dir 1i , dir 2i , dir 3i and dir 4i describe sub-direction strength of P i in above four directions, respectively. One simple feature representation is to cumulate sub-direction strength in each grid cell. That is

{ } j V v  , 1 2 3 4 { , , , } i j i j i j i j j i i i i p m p m p m p m v dir dir dir dir         
where i=1,2,…,n , j=1,2,…,8 2 ; and m j is the j th cell in the grid, P i is the i th point in sampling points, v j is direction feature of m j . So we can form a 8x8x4 dimensional vector, which can be used in a statistical classifier. We denoted this representation as 'Dir' , compared with below 'DirBSM'. Another feature representation is presented, called Directional Blurred Shape Model (DirBSM), is presented here. BSM [START_REF] Almazan | A non-rigid appearance model for shape description and recognition[END_REF][START_REF] Escalera | Blurred shape model for binary and grey-level symbol recognition[END_REF] is a shape descriptor , whose main idea is to describe a given shape by a probability density function encoding the probability of pixel densities of a certain number of shape sub-regions. Given an image of a particular symbol, each pixel contributes to compute the BSM descriptor. This is done by dividing the given image in a kxk grid (cells). Then, each cell receives votes not only from the shape pixel located inside its corresponding cell, but also from those located in the adjacent cells. Thereby, every pixel contributes to the density measure of its sub-region cell, and its neighboring ones. This contribution is weighted according to the distance between the point and the centroid of the cell receiving the vote.

However, in our approach the shape is not an image but a set of shape points with four direction attributes. We have modified the BSM descriptor to DirBSM. For each direction attribute, we calculate a BSM vector, as illustrated in Fig. 4. Then four vectors can form a final shape descriptor, which is the feature representation of a symbol.

Input: A grid of k k × cells and { , , }

i i i S x y dir = , where i dir  { 1 2 3 4 , , , i i i i dir dir dir dir } , i ∀ = 1,2,…,n
Output: the BSM of one sub-direction 1. Denote grid as , with j c the centroid for grid cell j r .

2. Denote N(r) as the neighbor cells of cell r, Let p r be the cell which contains the point p.

Initialize the probability vector v as v( j )=0, 2 1, 2,..., j k = .

3.For each point p(x i , y i ) D=0 For each ( )

p h r N r ∈ , 2 ( {1, 2,..., }) h k ∈ 2 ( , ) || || h h h d d p r p c = = - 1/ h D D d = +
End for Update the probability vector v as ( ) ( ) /( )

h h i h v r v r dir Dd   End for 4.Normalize 2 1 ( ) ( )/ ( ) k j v h v h v j = = ∑ , h ∀ = 1,2,…, 2 k

Fig. 4. Getting BSM vector of one direction feature

To modify the feature distribution for improving the classification performance, all the measurements in the feature vectors are transformed by variable transformation y=x p [START_REF] Liu | Handwritten digit recognition: investigation of normalization and feature extraction techniques[END_REF]. In our experiments we tested different values of p to obtain optimal parameter.

Experiments

In this section, we show the performance of the proposed method for sketch recognition tasks using above two feature representations.

NicIcon database

The NicIcon database [START_REF] Niels | The NicIcon database of handwritten icons[END_REF] is composed of 26,163 sketched symbols of 14 classes in the domain of crisis management and incident response systems. An example is shown in Fig. 5. The database comes from 34 different writers and it is commonly used for on-line symbol recognition, but off-line data is also available. The database is already divided into three subsets (TRAINING, TEST and EVALUATION sets) for both writer dependent and independent settings. Approximately, 36%, 24% and 40% of symbols are contained in the training, test and evaluation sets, respectively. We have selected the on-line data that is writer-independent, because it is the most similar to human-computer interaction. In order to get optimal meta-parameter p, we train a classifier using the TRAINING set and test its performance using the TEST set. Fig. 6 shows the accuracy with parameter p. The optimal p is p=0.3. So all the measurements (both in traing and testing) in the feature vectors are transformed by variable transformation y=x 0.3 [START_REF] Liu | Handwritten digit recognition: investigation of normalization and feature extraction techniques[END_REF].

Then we use both the TRAINING and the TEST sets for training the classifier and the EVALUATION set for measuring the recognition performance (T+V/E). The accuracy rates are 95.57% (DirBSM) and 97.95%(Dir) , respectively.

Results and discussion

In Fig. 6, when p is small, the accuracy of Dir is better than that of DirBSM. The reason for this may be that the BSM descriptor blurs the shape and cannot distinguish shapes in details.

As Table 1 indicates, on this dataset we achieved an best accuracy rate of 97.95%., compared to other results in the literatures [START_REF] Almazan | A non-rigid appearance model for shape description and recognition[END_REF][START_REF] Willems | Iconic and multi-stroke gesture recognition[END_REF][START_REF] Delaye | HBF49 feature set: A first unified baseline for online symbol recognition[END_REF]. For improvements in the recognition system, we analyzed misrecognition factors in the experiments. The top four false-positive symbols are accident, bomb, paramedics and electricity. It means there are other symbols that were misrecognized as them. And the top four false-negative ones, which were misrecognized as other symbols, are fire, casualty, police and fire brigade. Factor-1 is the large distortion in sketching. Some of these sketches cannot even be recognized by people, as shown in Fig. 7. Factor-2 is the presence of extra strokes or missing ones. Factor-3 is the similarity in direction features of sketches, mainly between the symbols gas and flood. Some distorted symbols are similar to different patterns, when they are represented by direction features. These may be resolved in terms of introducing other useful features.

Runtime Performance

Finally, we evaluate the runtime performance of our approach. We use Matlab to program this method on a 3.10*2 GHz machine (Inter Core™ i5-2400). The average runtime of recognizing one symbol is 97.6ms, including extracting feature and classifying with SVM-RBF. This method should therefore realistically be usable in real-time human-computer interaction.

Conclusions

This paper presents an on-line sketch recognition method based on the direction feature. It is a statistical method, which uses the visual feature of shape points instead of regarding the shape as an image or temporal sequences of geometric primitives. We also present two feature representations to train an SVM classifier. Experiments show that in the NicIcon database fine results are achieved, including high accuracy rate and real-time effects. This method is insensitive to stroke order and number. In future, we will do more experiments using larger databases with more classes, and introduce more feature to enhance accuracy rate.

Fig. 1 .

 1 Fig. 1. System overview.

Fig. 2 .Fig. 3 .

 23 Fig. 2. Decomposition of direction

Fig. 5 .

 5 Fig. 5. Samples of 14 different classes of NicIcon database

Fig. 6 .

 6 Fig. 6. Getting optimal p using training set and test set

  (a), (b) and (c) come for reference[START_REF] Almazan | A non-rigid appearance model for shape description and recognition[END_REF]. And the experiments in (b) and (c) used off-line data in NicIcon database.

Fig. 7 .

 7 Fig. 7. Misrecognized samples. (a) too large distortion; (b) and (c) distortion of triangle becoming a rectangle;(d) and (e) extra stroke; (f) distortion of rhombus becoming a trapezium

Table 1 .

 1 Accuracy compared with other methods

	Methods	Accuracy
	(a) combination of features[7]	92.63% *
	(b) NRAM+nrBSM(SVM)	94.29% *
	(c) NRAM+nrBSM(NN)	95.18% *
	(d) HBF49[8](SVM)	97.44%
	our approach(SVM)	95.57 (DirBSM) 97.95 (Dir)
	*The values of accuracy rate in