Filipe Quintal
email: filipe.quintal@m-iti.org

Lucas Pereira
email: lucas.pereira@m-iti.org

Nuno Nunes

Valentina Nisi
email: valentina.nisi@gmail.com

Mary Barreto
email: mary.barreto@m-iti.org

WATTSBurning: design and evaluation of an innovative eco-feedback system

Keywords: Sustainability, Aesthetics, Eco-feedback, User Interfaces, Prototyping

This paper reports a 15 weeks study of artistic eco-feedback deployed in six houses with an innovative sensing infrastructure and visualization strategy. The paper builds on previous work that showed a significant decrease in user awareness after a short period with a relapse in consumption. In this study we aimed to investigate if new forms of feedback could overcome this issue, maintaining the users awareness for longer periods of time. The study presented here aims at understanding if people are more aware of their energy consumption after the installation of a new, art inspired eco-feedback. The research question was then: does artistic eco-feedback provide an increased awareness over normal informative feedback? And does that awareness last longer? To answer this questions participants were interviewed and their consumption patterns analyzed. The main contribution of the paper is to advance our knowledge about the effectiveness of eco-feedback and provide guidelines for implementation of novel eco-feedback visualizations that overcome the relapse behavior pattern.

Introduction

Individual household consumption accounts for a significant part of the total worldwide energy consumption. For example domestic electricity is responsible for two thirds of the electricity used in the United States, 36% of the greenhouse gasses, and 12% of the fresh water consumption [START_REF] Wilson | Buildings and the environment: the numbers[END_REF]. In the European Union final energy consumption for households is about 31% of the total energy consumed, only second to transportation, which accounts for 36%. Between 1985 and 1998, the actual amount of energy consumed per household remained nearly constant, but the growing number of households increased energy use by 4%. This effect will have a dramatic global impact as developing countries contribute to the increase of households with energy access. Also while household electrical appliances are becoming more efficient, there are more of them and they are being used more often. Reports from the EU show that consumption by all-electrical appliances and lighting represents about 55% of the electricity used by households. These appliances include the six large consumers of electricity (refrigerators, freezers, washing machines, dish-washers, TVs and dryers), and many other small appliances. [2].

Studies have shown that individual activities can control up to 50% of residential electricity consumption, depending on the physical structure of the building and what set of appliances users can control [START_REF] Schipper | Linking lifestyle and energy used: a matter of time?[END_REF]. Moreover, most people are in fact concerned about the consequences of their actions. However, they are also unaware of the impact of their daily activities and more importantly how they can change their behavior to reduce consumption. This gap between users concerns and their actual knowledge of energy consumption habits, motivated companies and researchers to develop technologies that present users with information about their consumption. This type of technology is commonly called eco-feedback and is defined as technology that provides feedback on individual or group behaviors with a goal of reducing environmental impact [START_REF] Froehlich | The Design of Eco-Feedback Technology[END_REF].

The work presented here is part of a broader sustainability interdisciplinary research project which involves the deployment of a combination of sensing and ecofeedback technologies to motivate and trigger people to think, act, reflect and consume sustainably.

Related work

The advancement and availability of sensing systems for environmentally related activities (e.g., human activity inference [START_REF] Froehlich | The Design of Eco-Feedback Technology[END_REF]) and interactive displays to feedback this data (e.g., mobile phones) provides a rich space of prospects for new types of ecofeedback solutions [START_REF] Mankoff | Leveraging Social Networks To Motivate Individuals to Reduce their Ecological Footprints[END_REF]. Currently there is an increasing number of commercial applications that provide real time energy monitoring. These solutions range from low-cost single-outlet (Kill-a-Watt 1 and Watts Up2 ) to medium cost whole house power consumption (CurrentCost3 , TED 4 , Efergy 5 , Owl6 , etc.) to higher cost ambient feedback solutions (Wattson 7 , Energy Orb8 ). Through these solutions, feedback is often presented as raw energy use (e.g., Watts), personal cost (e.g., money), or environmental impact (e.g., CO2 emissions). Furthermore, eco-feedback is also an increasingly important research arena, confirmed by the growing number of articles presented at top international venues like CHI, INTERACT, DIS and Ubicomp, which nowadays devote specific sessions to sustainability. As a consequence, literature is abundant in design strategies and guidelines to implement such systems [START_REF] Mills | Review and comparison of web-and disk-based tools for residential energy analysis[END_REF][START_REF] Yolande | Designing eco-feedback systems for everyday life[END_REF][START_REF] Mills | Review and comparison of web-and disk-based tools for residential energy analysis[END_REF][START_REF] Consolvo | Design Requirements for Technologies that Encourage Physical Activity[END_REF].

One of the main challenges of designing eco-feedback systems is to present to users, how individuals or group activities can impact the environment. Commercially available feedback systems, have tried to present this information as kilograms of CO2 emissions. Pierce et al. [START_REF] Pierce | Energy Aware Dweling: a critical survey for interaction design for eco-visualizations[END_REF] surveyed several publications exploring the impact of eco-feedback technologies in energy consumption and on consumers' behaviors. The authors found out that the use of eco-feedback technology resulted on savings between five and twelve percent of daily energy consumption. They also concluded that when savings didn't happen, the eco-feedback was displayed too infrequently (monthly) and hence was disconnected from the consumption behavior. Egan [START_REF] Egan | How customers interpret and use comparative displays of their home energy use[END_REF] confirmed that receiving timely feedback is key to motivating behavior change. Moreover, Fischer reports that eco-feedback is more efficient when given frequently, clearly presented, using computerized tools and allowing historic or normative comparisons [START_REF] Dunlap | Lay Perceptions of Global Risk[END_REF].

Normally eco-feedback is associated with behavior change. There is evidence that the single goal of saving money (less electricity you consume the less you pay), is not enough to motivate conspicuous behavior change -between 5 and 15% reduction on average [START_REF] Fischer | Feedback on household electricity consumption: a tool for saving energy[END_REF]. Consolvo et al. [14] implemented a system that promotes a more physically active lifestyle. Here the authors propose eight qualities that a system should have in order to be well accepted by users: Abstract & Reflective, Unobtrusive, Public, Aesthetic, Positive, Controllable, Trending / Historical and Comprehensive. These studies have encouraged research into non-traditional feedback systems, turning to digital art and disruptive design for help. In the digital art domain, efforts were made to raise awareness regarding their behaviors in relation to sustainability.

For example, digital artist Tiffany Holmes, visualizes energy consumption through an art installation situated in a public space [15]. Holmes uses digital art to display hidden data of real time usage of key resources (such as electric appliances) and providing an aesthetically striking visualization in the public space of the building hall. Dwellers were able to relate to the visualization as a community, negative feedback was avoided and people living in the building were empowered to act upon their consumption by getting to know data that wouldn't be available to them otherwise. Another example of digital art employed to raise awareness about energy consumption is the Helsinki based project presented in [START_REF]Nuage Vert Nuage Vert Project[END_REF]. The project consists of a public installation, in which a green cloud proportional to the city energy consumption is laser projected onto the smoke generated by the chimney of a coal energy power plant in the city. During the seven days of the installation, the green projected cloud would grow and shrink in direct proportion to the city energy consumption. In the final day of the installation, residents were asked to unplug the devices to reduce consumption, in order to dramatically increase the size of the green cloud. This resulted in a reduction of the peak demand in 800 kVA (approximately the same power generated by a windmill running for one hour).

Despite these developments the effectiveness eco-feedback is known to have problems. Peschiera, reports that that after a certain period of usage of the feedback devices, users consumption relapse to values prior to the study [START_REF] Peschiera | Response-relapse patterns of building ccupant electricity consumption following exposure to personal, contextualized and occupant peer network utilization data[END_REF]. Holmes reported this phenomenon in [15] where it was possible to see that users gradually returning to their previous behaviors if feedback was less frequent or no longer present. Our own research [START_REF] Nunes | Deploying and evaluating the effectiveness of energy eco-feedback through a low-cost NILM solution[END_REF] confirmed this: "We would check our consumption more often initially until we got a rough idea or perception of what our consumption was but after that it would less frequent ", furthermore we observed that this lack of attention starts to happen after the fourth week of usage.

Previous work

Prior to the development and test of the WATTSBurning system, the research team was responsible for 2 long-term eco-feedback deployments. Those deployments involved 30 houses and apartments located in an urban area in southern Europe. To better understand our starting ground, we briefly discuss the results from those deployments that lead us to the implementation of the WATTSBurning system. A detailed explanation can be found in the corresponding reference.

First Deployment.

In the first deployment the research team designed an eco-feedback interface based on an evaluation of commercially available systems. The system presented real-rime and historical consumption using bar charts and involved three modes of operation: idle, attention and detail. These modes were triggered by a camera (using face and motion detection algorithms), which were also used to log user activity. The initial system was deployed in 21 houses for a period of 9 weeks. The data showed a 9% average aggregated decrease of consumption. A deeper look into the data disclosed that families that used the eco-feedback system more often had higher decreased consumption. Nevertheless there was a steep decrease in the users' interested in the eco-feedback, particularly after the first 4 weeks (users interest was measured by the amount of access to the different application features) [START_REF] Nunes | Deploying and evaluating the effectiveness of energy eco-feedback through a low-cost NILM solution[END_REF].

Second Deployment.

Based on the findings of the first study, the feedback system and sensing infrastructure was completely redesigned and revised. The new eco-feedback interface was developed with the help of a designer following a set of "guidelines" extracted from the research literature. We conducted a 52 weeks long-term study using the revised system deployed in 13 houses starting from the first sample [START_REF] Pereira | Understanding the Limitations of Eco-feedback: A One Year Long-term Study[END_REF].

Results from the second study revealed that initially people interacted with the system more than with the previous one. However, the frequency of the interaction started to decrease after two weeks, and again the steepest decrease happened after four weeks confirming our initial findings [START_REF] Nunes | Deploying and evaluating the effectiveness of energy eco-feedback through a low-cost NILM solution[END_REF]. Detailed analysis of interaction data revealed that after four weeks, some users even stopped checking the system altogether. A few others kept using it but less frequently (only once or twice a day). During this second deployment users' consumption remained virtually unchanged from the first deployment, suggesting that savings come from the initial understanding of consumption patterns.

Design Rationale

The experience gained from the studies described above suggested that a different approach was required to overcome the decreased interest of users in energy data. Using inspiration from artist we brainstormed ideas and decided to change strategy and test a new paradigm of eco-feedback. We postulated a more inspiring and emotional visualization towards a less information driven eco-feedback system. After brainstorm sessions with artist designers and engineers it was decided to implement the feedback based on mapping the energy consumption of the household with elements of the local landscape, with the goal of leveraging the emotional connection of families with the local natural patrimonies, which is home of a UNESCO heritage forest. It is important to note that most of the inhabitants already feel a strong connection with the local forest. This was visible after the tragic mudslides in 2010 and the forest fires in 2011. A thorough description of this process can be found in [START_REF] Nisi | Beyond eco-feedback: using art and emotional attachment to express energy consumption[END_REF].

Pilot

To test the concepts described above a novel interface was developed for the ecofeedback system. It consisted of a video based animation of a well-known forest site.

Based on the consumption level several elements of that landscape would change. The electricity real time consumption was mapped as the movement of the clouds in the background, and by adding and removing animals in the landscape. More consumption meant that the clouds would pass by faster, and more appliances turned on or off meant more animals in the forest. This was a neutral feedback since more appliances being turned on or off does not necessary means more consumption, and the movement of the clouds does not have any direct negative meaning. Figure 1 shows these how these two feedback modes where displayed in the application. This new eco-feedback version was tested in eight households during one month. Consumption and interaction data was saved and four of the eight families were interviewed. Quantitative data revealed no significant differences in the energy consumption of the families neither prior or during the study. However, families using this new version of the system had an increased number of interactions when compared with families that used a "traditional" eco-feedback device during the same period. Nevertheless there was still a decrease in the interaction along the period of the study, consistent with the response-relapse behavioral pattern reported above. However, this was not a linear decrease as there were several peaks in interaction during the month.

Qualitative analysis of the interviews revealed that neutral feedback was not well understood by users. Users that were exposed to previous versions of the feedback also missed out on the quantitative data provided previously:

"No I didn't relate it to my consumption levels. I wasn't even near to realize that (laughs). I was looking for data, logical data about it (...) I could see some extra elements, I thought you were decorating the landscape but I didn't understand why were they being placed there" Family 4 mother.

"This one is simpler, there's just the image and the consumption on the right. In the other one I could see the consumption in terms of the whole day and this was more elucidative. I find this one more interesting but we need some kind of heads ups about how can we see the consumption throughout the whole month, the whole week" Familiy 3 husband.

On the other hand after the artistically inspired mappings of the energy consumption to the landscape were explained to the users (during the final interview) they found this new feedback paradigm interesting, and suggested merging the landscape visualization with the quantitative data display:

"I think both are valid. Maybe there could be a symbiosis between the two. This one is more pleasant the other one is just data. What matters to me is to have the data, it's probably the most important for me. However, I like the way this one is presented, I feel it's more interactive than the other one" Family 3 husband.

Three of the four families interviewed suggested that the concrete data provided by the first two versions should be merged with the aesthetic pleasure of the natural landscape shared this view.

3.2

Refinements of the feedback system.

The experience from running the pilot of the WATTSBurning, made it clear that users found the feedback interesting, but they still wanted the concrete data about the consumption. Additionally the mapping between the forest and consumption was not fully understood and some users needed further explanation of how it was represented.

In the WATTSBurning system we also wanted to address some limitations found in the two previous deployments. In the initial deployments, the feedback was given via the display of a small netbook installed behind the main door of the house (where the main fuse box is located in most of the houses) [START_REF] Pereira | The Design of a Hardware-software Platform for Long-term Energy Eco-feedback Research[END_REF], this type of stationary feedback is clearly limited since it is not accessible to all the household members at the time of decision (e.g. turning off a high consumption appliance). Also the fact the system was connected to the main fuse box made some families worry about the safety of the device, and sometimes would not allow children to use it. Furthermore some additional requirements emerge form the state of the art. The feedback should be accessible to all family members since the family dynamics and communication have an impact in the decision [START_REF] Barreto | Social Translucence as a theoretical framework for sustainable HCI[END_REF]. The system should allow the comparison between different periods (hour, day, weeks and months) so that people can explore and better understand their consumption patterns. The system should also provide simple tips about energy conservation and best practices promoting sustainable behavior change. The hardware itself should also be aesthetically pleasant, as Petersen [START_REF] Petersen | Aesthetic interaction: a pragmatist's aesthetics of interactive systems[END_REF], argues that pragmatist aesthetics is a promising approach for designing interactive systems as it promotes aesthetics of use, rather than aesthetics of appearance. Aesthetics play an increasingly important role in interaction design, in particular when designing for homes and everyday lives rather than for the workplace.

Collectively our prior experience and these requirements meant that the new version had to undergo major updates both on the software and in the hardware side. The eco-feedback device should ideally be portable so it can be accessed anywhere in the house (or outside). The sensing framework should provide data remotely to the ecofeedback device so that the sensing is removed from the house, addressing users safety concerns. The visual eco-feedback should be aesthetically pleasant while still providing concrete consumption information.

System design

In order to provide accurate and meaningful eco-feedback, the system needs to measure the energy and resources consumption effectively. Measuring energy/resources consumption is in itself a challenging research problem. Researchers are striving to measure energy consumption in more cost-effective, accurate and less intrusive ways. One of the most promising research approaches is non-intrusive load monitoring (NILM), which reads data from a single point and tries to monitor and desegregate the consumption per appliance. The main assumption of most of the exiting NILM approaches is that every change in the total load consumption of a household happens as a response to an appliance changing its state. Therefore specific appliances can be isolated and their individual consumption calculated using complex signal processing and statistical learning techniques [START_REF] Nunes | Deploying and evaluating the effectiveness of energy eco-feedback through a low-cost NILM solution[END_REF]. Low cost and non-intrusiveness are the main advantages of the NIML approach thus it was chosen as the building block of the sensing infrastructure that supports this research [START_REF] Pereira | The Design of a Hardware-software Platform for Long-term Energy Eco-feedback Research[END_REF].

Implementation

In this section we explain how the requirements described in the sections above were implemented. Firstly we explain how the sensing framework evolved to the current version. Then we explain the implementation of the front end eco-feedback visualization of the system.

Sensing platform/framework. Our sensing and eco-feedback platform is based on a custom made NILM system [START_REF] Nunes | Deploying and evaluating the effectiveness of energy eco-feedback through a low-cost NILM solution[END_REF][START_REF] Pereira | The Design of a Hardware-software Platform for Long-term Energy Eco-feedback Research[END_REF]. Our research was target at real world deployment of low-cost eco-feedback systems that are capable of sensing and disaggregating energy data in households. Our technical requirements were to find a low-cost solution that could run different NILM algorithms and eco-feedback visualizations while also collecting relevant user data.

Originally our system was based on a netbook that acted as both as the sensing infrastructure (reading current and voltage from the microphone jack) and the visualization (via the built-in screen) and human sensing (via the built-in camera) system. Our ini-tial prototype provided an easy way to deploy the system and evolve the visualizations during the first deployments. However, it was limited in terms of location (next to the mains at the entrance) and required installation inside the homes and unreliable Internet access. A second version of our system was implemented using a more capable DAQ (Data Acquisition Board) at the entrance of apartment buildings, hence measuring the energy consumption of multiple houses from one single location. This new version was less intrusive and enabled data to be pushed via web-services to different visualization platforms inside the home (web, tablets, etc.). Figure 2 shows the last implementation of our multi-apartment NILM sensing system. On the left hand-side one can see the three voltage sensors and the DAQ board, on the right it is possible to see current clamps installed on the current conductors of the apartments monitored. All of these signals are acquired and processed by a single computer that also runs the NILM algorithms and provides the data for eco-feedback devices via restful web services. The system also stores all the data in a database and provides access for consolidation in a datawarehouse.

WATTSBurning eco Feedback visualization

Following the requirements described before we implemented a new hardware and software platform but also modified the eco-feedback component, which now could be deployed on different portable devices. The new visualization addressed our experience in pilot (section 3.1) producing an aesthetically pleasant landscape that mapped the household consumption to elements of the natural forest. We also provided a way for users to access a second layer of more detailed consumption information. The eco-feedback visualization was implemented on a 7'' android tablet using the android native SDK (Figure 3). The tablet specifications allowed us to implement rich visualization, which can be accessed in different places of the house, or even outside. We believe that the device itself (a tablet) was still viewed as a novelty, and that helped our application to fit in the household as well as encouraging interaction.

The application receives real-time consumption and historical data from the sensing framework. The real time data is received using sockets and a custom made communication protocol. Historical data is gathered by using the web-services described in the previous sections. The historical data is also stored in the tablet so the users can still check their past consumption without an Internet connection. The transitions between the different modes of operation are stored locally enabling the analysis of the usage patterns by the research team. For example if a user picks up the tablet, presses the back button to go to the home screen and selects the daily consumption, the application will store 3 interactions with 3 different ids reffering to the 3 different views accessed by the user. Our eco-feedback system involves two main modes of operation. When it is not used for two minutes it goes into the Energy Awareness mode that shows the consumption mapped as a digital illustration of the local endemic forest. Once the user interacts with the tablet, by pressing the back soft key, the system goes to Detailed Consumption mode and shows daily, weekly and monthly information about the home energy use. It is important to point that Energy Awareness was the default mode of the system, the tablet never went to an idle mode like it happens by default in some android applications.

Energy awareness mode.

To select the landscape and the metaphors used in this mode we performed a think aloud with two families (these families were later recruited as participants for this study). In these sessions the family members were presented with several paper prototypes displaying the consumption in different landscapes and represented with different items in the forest as seen in Figure 4. We decided to use a landscape of a well know forest site that was easily recognizable by the local community. The digitally modified pictures of the forest represented the comparison between the real time consumption and an average baseline consumption level. After the think aloud session we rejected some concepts like displaying the historical consumption as items in the landscape (e.g. moss growing on a tree) or the real time consumption mapped in the movment of an animals, these concepts were not clearly understood by the users.

In total there were five levels of consumption represented in the forest (as shown in Figure 5). These five levels represent when an household consumption is slightly above/bellow, well above/bellow or belongs to the average baseline. The baseline was composed of an average of the consumption on that period, for example the real time consumption on a Monday at 12:20 was compared against an average of all the Mondays during the period between 12:00 and 13:00. Aditional ilustrations of the forest are used to ensure a smooth transistion between the states, however the animation only stops in the five levels aforementioned.

Detailed consumption mode.

This mode is triggered when the user presses the tablet back button. As a consequence the system presents a tabbed menu with four options: "Home", "Day", "Week" and "Month". The "Home" tab shows a summary of the overall consumption as well as the current real-time consumption (Figure 6 Left). The summary contains aggregated consumption of the current day/week and month, and comparisons between homologous periods. Also in this tab the user is presented with a "tip of the day" with general sustainable actions. The "Day" "Week" and "Month" tabs (Figure 6 Right) present a chart displaying the consumption over that period and the total aggregated consumption. It also informs the user of where the peak consumption happened and how it compares to the average of that period (for example in the "Week" tab the system shows how the consumption in the current week compares with an average of the previous weeks). By default the information presented here refers to the current day/week/month but the user can select preceding periods.

Collected Data

Our sensing framework samples the current and voltage waveforms with at a high sampling rate (about 3.2 kHz) but due to database size constraints only average power consumption is stored in a database (in 30 seconds sampling intervals). Additionally the application stores locally every transition between the different modes of operation of the eco-feedback system which is then periodically uploaded to one of our servers, hence allowing us to keep track of how the system is being used without having to wait for the end of the deployment. The data analysed corresponded to two weeks of baseline consumption data collected when no eco-feedback was available and 15 weeks of consumption data after the eco-feedback was installed, as well as all the user interactions with the feedback during that time.

EVALUATION

To test our system we recruited users from an apartment complex in an urban area in southern Europe. All the apartments were relatively new and they had similar appliances (provided by the construction company), consequently all of the families were relatively new to their houses. The sample is composed of six households: five couples with one or more children, and one young couple with no children. Two of the families had to leave the study earlier, but in both cases this was related to the fact that the families had to leave their apartments and not because they wanted to abandon the eco-feedback study. The families were told to use the tablet as they wanted (for example for browsing the web). But only the interactions in the WATTSBurning system were recorded, as described in the previous section.

Method

In order to assess the changes in consumption we collected baseline consumption data before the eco-feedback system was installed. This data was collected between the 13 th and the 27 th of August. After that each family received a tablet with the WATTSBurning application installed and given a short explanation of how to access the consumption data. After 22 days we visited the families and performed a short semi-structured interview. The families continued to use the system until the end of the year. Here we analyze the consumption and interaction data until the 17 th of December for a total of 17 weeks. We wanted to avoid the Christmas period, since the results in that period would most probably be biased.

Qualitative Assessment

When the families were interviewed after the first three weeks, we wanted to understand how the system was received, and if the Energy Awareness mode was being clearly understood and creating an increased awareness about energy consumption. It was also important to verify if any behavior change was triggered by the presence of the eco-feedback system. Five of the six families were interviewed, they will be referred as F1, F2, F3, F4, F5. We asked all the family members to be present in the interviews, in order to gather every member' opinion about the system. However, for F2 and F3 only the husbands were present during the interview.

The system was well received by all the families, and none of them had any major problems using it. "I think it was simple, even the wireless connection was stable" F4 All the families agreed on the fact that the system increased their awareness about electricity consumption.

"It's raising my awareness, I don't think I've changed my consumption patterns yet, but I'm more aware now". F1 "… if we have more devices turned on we can see right away there is an increase" F2 "It provides us with immediate feedback, such as daily and weekly consumption, and we can see what we do and how we behave in our daily routines to reduce our energy consumption" F3 "I got more alert, so necessarily I will try to consume less" F5

Another observation transversal to all of the families was that every family learned something about the consumption of certain devices. " In the weekend for example, we were using the oven and I noticed it consumed a lot, then I turned on other devices out of curiosity" F1 "…the electric stove for example, I learned from the system that it consumed a lot" F2 "Especially the oven, it increases to 2000/2500W… I see there that it goes to 200 ºC but I wasn't expecting so many watts… I was even surprised with the TV's" F3. "I've learned that the oven consumes a lot, also the fridge" F5 It was also important to verify if the energy aware mode of the application was well understood. All families found that it was easy to relate their current consumption with the animation of the forest drying up and eventually being set on fire. It was also mentioned that the picture showing the state of the forest in the feedback interface worked as an alert of what was going on in the house, and it made them more aware of their consumption at specific moments.

"Yes but…. For example when the dryer is working I see the forest catches fire, but I was expecting that, but I knew I couldn't keep that consumption for a long time, maybe I got more conscious" F1 "It gets dryer until it catches fires, when I have a lot of things turned on (…) also when there were only a few things on it showed rainbows and butterflies" F2 "(…) there's an association, the bigger the consumption the more destruction is visible in the forest, when we see it goes from green to yellow to red,… it's scary," F3.

All the families opted to place the system in a central place in the house, where it could be visible to all the family members. However in the five families that were interviewed it was mostly the adult male and young children who used it more often. "… I was curious in the beginning to see the forest on fire, but it was mostly my kid"F2 "It was mostly me and my son, my wife and my daughter didn't pay much attention to it, it was me and my younger son"F3, "It was mostly me, she (girlfriend) wouldn't use the system".

Although the system was usually located in a central visible place in the house, it was common for participants to move it around, mostly because some of them wanted to check the consumption of a device in real time.

"It was mostly there, but it was in the kitchen for some time" F2 "It was there close to the sofa, because it had better reception for the internet (…) but I would also take it to the balcony while sitting there"F3 "I used it in different places in the house, I thought of it as one of those portable weather forecast displays that can be placed anywhere". F5

Quantitative Assessment

From the qualitative assessment users reported a noticeable increase in their energy awareness. However, it is important to confirm if the reported awareness translates into an actual behavior change, confirmed by the quantitative data acquired from our sensing platform. In the following sections we analyze and discuss this issue.

Energy consumption

In order to compare energy consumption we use the week as the standard period of time. This is the unit that best spans the routines of a family impacting their energy consumption. For instance, some families organize the major cleaning in a single day of the week, others the ironing. The week also comprises the working days and the weekend that usually correspond to very different routines. Therefore we compare the consumption data in the 15 weeks of eco-feedback deployment with the average of the two weeks of baseline data.

If we consider the 6 households as a whole, after 15 weeks the average weekly consumption dropped on average 2% from the initial baseline (n =6, SD=7.99%). However, the standard deviation suggests that the savings were not constant across the houses. As a consequence we analyzed each house individually. Three families (F1, F2, F5) reduced their consumption by 5%, 7% and 13% and their consumption was below the baseline for long periods -12 weeks for F1, 11 weeks for F2 and 5 weeks for F5. The other three families consumed on average more than the baseline , respectively 10% (F3), 1.5% (F4) and 2.5% (F6). These 3 houses were below the baseline average during 3, 6 and 6 weeks respectively. Figure 7 shows how each family's consumption changed during the study.

Interactions with the system

The analysis of the logged user interactions (the saved interactions are defined in section 3.6) showed that families used the system in completely different ways. Two of families (F3,F6) used the system more on the Mondays ,F1 and F4 used the system more on Wednesdays, F2 used the system mostly on Fridays and F1 on Tuesdays. This diversity was also noticeable when looking at how families used the system throughout the day. However, in al the families the view that was most triggered was the summary, followed by the animation view.

During the 15 weeks of the study a total of 1577 interactions with our system were logged among all of the households. That value represents an average of 266 interactions per house and an average of 16.3 interactions (N=1597, SD=29) per day and 4.2 interactions per house and per day (N=6, SD=6.06). Our system was mostly used in the afternoon (between 13:00 and 19:00), almost a third of the interactions happened during this period. It was also noticeable that all the families had a lot of interactions on the first four to five weeks of the study. After that the number of interactions dropped significantly (by more than two a thirds). Again the pattern was not similar in all houses, which justified the high standard deviation. Table 1 shows a summary of the number of interaction with the system, in the first four weeks of the study and rest of the period and in total. We choose to isolate the first four weeks because our previous research suggests that that after this period there is a steep drop in the usage of the feedback devices [START_REF] Nunes | Deploying and evaluating the effectiveness of energy eco-feedback through a low-cost NILM solution[END_REF][START_REF] Pereira | Understanding the Limitations of Eco-feedback: A One Year Long-term Study[END_REF].

To better understand the usage patters we analyze each household individually. The number of interactions on F1 only decreased by less than half, they kept using the system with an average of 14 interactions per day after the initial four-week period. Similarly with F2 the decrease in interaction was not considerable, but it was noticea-ble that they stopped using the system on a daily basis. Families 3, 4 and 5 had a lot of interactions in the first week, after that the interaction values dropped to less than one interaction per day (0.6 ,0.4 and 0.2 per day respectively). F6 simply didn't interact directly with the system, even though this family agreed to participate in the study, they never showed much interest regarding the system nor they were ever available for interviewing.

We also analyzed the correlation of the user interactions with the energy consumption. The three households that on average didn't reduce their consumption are among the households that interacted less with the system (F4, F5 and F6). This finding is also consistent with our previous research. 0.6 0 0.4 It is important to note that the system only logged direct interaction with the interface. It would be very hard to count the number of time the users looked at the tablet with the Energy Aware mode on. Furthermore users confirmed that a significant amount of interactions were done with the tablet in the Energy Aware mode.

"It worked as an alert for me (...) it was easier to see from a distance (…) when we started to consume more it would get darker and catch fire" F5

Discussion

The presence of eco-feedback increased users knowledge about the devices that they had at home. In fact all of the participants learned something about a particular appliance, this indicates a rise in the awareness about energy consumption, despite the level of usage or the change in overall consumption. However, the decreasing interest for eco-feedback after several weeks is an important factor leading to the relapse effect. Our novel eco-feedback system tried to overcome this issue by trying to map energy consumption to elements of the natural landscape. From the interviews with users we can conclude that the mapping the consumption with images of the local forest landscape was clear and well understood by participants. Although users didn't mention an emotional connection with the illustrated forest landscape, the energy awareness mode, where the forest dries up as more energy is consumed, did work as an alert for when consumption was higher than normal. Some users even found the forest on fire for a long time a disturbing factor that would motivate them to investigate which appliances were responsible for the higher level of consumption. Also, displaying the eco-feedback in a portable device allowed users to move the device through the household and explore the consumption of different appliances.

In terms of usage the system had a lot of interactions during the first four to five weeks, after that period the number of interactions decreased. However, unlike in our prior research the reduction was not linear and here were several peaks until the count finally settled at a low value. We believe this was a consequence of placement of the eco-feedback but also the presence of the energy awareness mode, which reminded people of the long-term consequences of their daily actions. The results in terms of user-interaction are an improvement over previous studies with classic quantitative forms of eco-feedback, since most families kept using the system after the four/five week period although less frequently and with different patterns. This difference in how families use the system is inline with the "one size does not fit all" [START_REF] He | One size does not fit all: applying the Transtheoretical Model to Energy Feedback Technology Design[END_REF] argument for eco-feedback systems. If we relate the quantitative interaction data with the consumption information it is noticeable that the houses that saved more energy are the ones that used the system more.

Conclusions and Future work

In this paper we describe the studies and refinements leading to the design, prototyping and testing of WATTSBurning an eco-feedback device designed to foster awareness of energy consumption in households. Our research aimed at overcoming the know problem of people relapsing to previous behavior after several weeks of exposure to eco-feedback. Through an iterative design, testing and refinement process we improved our eco-feedback system introducing a new artistic metaphor that combines energy consumption levels to artistic representations of the local forest landscape. Our goal was to verify if this novel metaphor would improve on significant reduction of interaction with eco-feedback after four weeks of deployment. After initial prototyping that removed some ambiguity in the mappings of consumption to natural elements, the WATTSBurning system was successful in improving the levels of user attention and usage. Most families kept using the system even after four to five weeks, although to a lesser extent. The placement of the eco-feedback device and the presence of an energy awareness mode showing the landscape changing was an important motivator to retain user attention and awareness over time. These findings provide a good motivation to explore new forms of eco-feedback that go beyond traditional quantitative information. Clearly without more research eco-feedback technology will not confirm the promising results coming out of short-term three-week studies as the ones published in many HCI venues. The households in our study did manage to reduce their consumption, but they showed very different consumption and eco-feedback usage patterns. Overall households that used the system more saved more energy but there is still a lot to learn about the long-term consequences of ecofeedback. We believe our attempts with non-conventional forms of eco-feedback and, in particular, when exploring emotional and aesthetical aspects is a promising path to explore further.

Our deployment is still ongoing, and the plan is to further investigate the qualitative data. Relevant results might be achieved when comparing across houses or even between different periods in the same house. Furthermore our sensing framework is storing all the power events extracted from the NILM, which can also be useful to research into users consumption and behavior patterns. Even thought the feedback system was already removed the sensing framework is still gathering consumption data and it will be interesting to observe if there is any change in the consumption after the feedback is removed.

Fig 1 .

 1 Fig 1. Screenshot of the application used in the pilot. At left there's the main view of the forest, the consumption is mapped as the movement of the clouds in the background. At right there is the landscape with elements added based on the appliances used.

Fig. 2 .

 2 Fig. 2. Shows the system as it was deployed in one of the buildings of our study. At left there is the DAQ Board. At right there is the current clamp sensing each house.

Fig. 3 .

 3 Fig. 3. System installed in one of the households.

Fig. 4 .

 4 Fig. 4. Different landscape possibilities used during the think aloud process

Fig. 5 .

 5 Fig. 5. Different views of the landscape according to the consumption. Ranging from low consumption (Image 1) to high consumption (image 5).

Fig. 6 .

 6 Fig. 6. Left: Home screen of the system. Right: Tab with the consumption of the current week.

Fig. 7 :

 7 Fig. 7: Consumption in the 6 households during the study. The baseline period is highlighted in red. The horizontal axis represents the week in the study, and the vertical represents the average consumption in kWh.

Table 1 .

 1 Summary of the average of interactions with the system by day, in different periods of the study.

	Family		Average interaction by day
		First 4 weeks	Rest of the study	Total
	F1	23.2	13.9	16.2
	F2	13.3	1.2	4.2
	F3	5.3	0.4	1.6
	F4	2.1	0.6	0.9
	F5	3.8	0.2	1.6
	F6

http://www.p3international.com/products/special/p4400/p4400-ce.html

https://www.wattsupmeters.com/secure/products.php

http://www.currentcost.com

http://www.theenergydetective.com

http://www.efergy.com

http://www.theowl.com

http://www.diykyoto.com/uk/

http://inhabitat.com/the-energy-orb-monitor-your-electricity-bill/

Acknowledgments

We would like to acknowledge designers Joana and Sofia Gomes for their contribution building the illustrations of the forest used the in WATTSBurning application. We would also like to thank all the families that took part of this study. The research presented here was funded by the SINAIS project (CMU -Portugal) CMU-PT/HuMach/0004/2008.


Sharon Lynn Chu

Francis Quek
email: quek@vt.edu

Yao Wang

Rex Hartson
email: rex.hartson@gmail.com

Finding-NEVO: Toward Radical Design in HCI

Keywords: Radical design, design method, innovation, HCI

de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

HCI is an interdisciplinary field with a foot each in the doors of science and design [START_REF] Fallman | Design-oriented Human-Computer Interaction[END_REF][START_REF] Fallman | Why Research-Oriented Design Isn't Design-Oriented Research: On the Tensions Between Design and Research in an Implicit Design Discipline[END_REF][START_REF] Carroll | Artifact as Theory-Nexus: Hermeneutics meets Theory-Based Design[END_REF], causing an inherent tension since "[s]cience and design have different principal objectives" [START_REF] Carroll | Artifact as Theory-Nexus: Hermeneutics meets Theory-Based Design[END_REF]. While design values innovation and is generative in nature [START_REF] Gaver | What Should We Expect From Research Through Design?[END_REF], science builds on prior advances in verifiable steps. Approaches like theory-based design (TbD), design-oriented research (DoR), and research through design (RtD) have been proposed to resolve this conflict. In DoR [START_REF] Fallman | Why Research-Oriented Design Isn't Design-Oriented Research: On the Tensions Between Design and Research in an Implicit Design Discipline[END_REF], design and technology implementations serve to test and validate the research concepts. Similarly, TbD [START_REF] Carroll | Artifact as Theory-Nexus: Hermeneutics meets Theory-Based Design[END_REF] and RtD [START_REF] Zimmerman | An Analysis and Critique of Research through Design: Towards a Formalization of a Research Approach[END_REF] see the value of design in the light of artifacts embodying some form of knowledge. Conversely the design of commercial products focuses on design practice, or solving problems and real-world obstacles, that Fallman calls research-oriented design (RoD) [START_REF] Fallman | Why Research-Oriented Design Isn't Design-Oriented Research: On the Tensions Between Design and Research in an Implicit Design Discipline[END_REF]. Our research looks at process in the context of the DoR tradition in HCI.

To properly answer a research question, DoR prototypes have to embody the ideas to be tested. Yet, there is little concrete guidance on how to design such research prototypes. Our goal is to gain an understanding of prominent design processes in HCI through interviews with HCI researchers and domain experts, and to suggest a plausible process model for the design of prototypes in research.

Design of Research Prototypes

We employed a general model (Figure 1), as a probe for interviews, of how research prototypes are designed. This model is similar to Gaver's [START_REF] Gaver | What Should We Expect From Research Through Design?[END_REF] idea of designs occupying points in the design space or creating a design space around themselves. The HCI researcher advances a concept from some conceptual rationale (label A), and produces a seed idea (B). The conceptual rationale may be a theoretical construct, a new technology, a research intuition, or a user need, and the seed idea is an instantiation of how the theoretical construct may be embodied in a system. This seed idea is situated in the space of possible designs of the domain in question (C), which may be populated by a set of existing designs (D). If the conceptual rationale is novel, and the design space is not densely populated, the new seed idea would likely be far from existing designs. We call such a design a potentially radical design (E) because of this difference from existing solutions. The seed idea does not fully specify a testable system. Many pathways of design choices or decisions have to be made to develop a system (F). Some of these design choices flow directly from the seed idea, and some may be necessary to realize the system but do not find guidance from the seed idea. All of these design choices have potential impacts on the testing of the conceptual rationale of the research. The term NEVO (Non-Embarrassing Version One) designates the first testable prototype that faithfully embodies a concept.

There is an inherent tension, that we seek to address, when realizing the design pathways (F). This tension can be characterized by different types of 'creeps' (G) that threaten fidelity to the original conceptual rationale. Prior experience with existing systems by designers, developers, and study participants introduces forces to move the design to resemble existing solutions. We call this experience creep. Also, our tendency to resort to the most convenient way for implementation and design using tools and frameworks at hand can introduce convenience creep.

Contextualizing our Work

Design of Research Prototypes: There are few well-defined, systematic process methods to develop good research prototypes. We briefly describe two frameworks that are particularly relevant to the problem of finding NEVO. Carroll's & Kellogg's [START_REF] Carroll | Artifact as Theory-Nexus: Hermeneutics meets Theory-Based Design[END_REF] theory-based design (TbD) proposes that successful HCI designs embody psychological claims in contexts of use, and advances that claims coalescing together in the implementation of a system projects "a model world", "a believable illusion" to the user. The careful recording of this 'design rationale' can inform the design of future artifacts. Keyson and Alonso's [START_REF] Keyson | Empirical research through design[END_REF] Empirical Research through Design method embeds interaction design hypotheses into working prototypes to contribute to design knowledge by creating experimental variability. The method lists several broad guidelines (e.g., "prototype variability has to be carefully defined so as not to confound the research question at hand"), and specifies the use of design iteration and techniques, but presents no clear process model.

Process for Radical Design: Work on radical innovations tends to come overwhelmingly from the management sciences. Many either emphasize the importance of the individual with passion and vision, the "human side" (e.g. teamwork, networks, roles [START_REF] O'connor | The Human Side of Radical Innovation[END_REF]) and organizational mechanisms (e.g. rewards, management policies), or advocate the approach of concept refinement through the involvement of users (e.g. [START_REF] Lettl | Users' Contributions to Radical Innovation: Evidence from Four Cases in the Field of Medical Equipment Technology[END_REF][START_REF] Heiskanen | User involvement in radical innovation: are consumers conservative[END_REF]). Management science is concerned with the potential of a product to be adopted by users, thus defining radical products "in the sense that they imply changes in consumers' everyday lives" [START_REF] Heiskanen | User involvement in radical innovation: are consumers conservative[END_REF]. In contrast, our concept of radicalness in HCI research is that it is valuable because it is an extension to knowledge in and of itself.

Existing Designs

Space of Possible Designs for a Domain

Closely aligned with Figure 1, a 'hill-climbing' model was proposed by Norman [START_REF] Norman | Incremental and Radical Innovation: Design Research Versus Technology and Meaning Change[END_REF]. He characterizes the design space for a particular domain as being occupied by multi-dimensional 'design hills'. A particular design sits on the slope of a hill, and ascending the hill constitutes a design improvement. Human-centered design (HCD) approaches allow designers to climb a particular hill by an iterative incremental process of design improvements driven by lessons learned from user studies. Norman states that while HCD enables a design to ascend its hill, it cannot move the design to another hill with a higher peak. Such a jump to a different hill would constitute a radical design shift. HCD is thus "only suited for incremental innovation", to improve existing products according to the user's contextual needs. Norman & Verganti [START_REF] Norman | Incremental and Radical Innovation: Design Research Versus Technology and Meaning Change[END_REF] argue that potentially radical seed ideas have to be driven by technology or meaning change. For radical innovation, they suggest that HCD must admit the "simultaneous development of multiple ideas and prototypes" (essentially many seeds).

Study Methodology & Data Analysis

We carried out hour-long semi-structured interviews with two sets of HCI people: 1. Meta-Interviewees (MIs) and 2. System Builders (SBs). The MIs are leading researchers from prominent HCI research labs throughout the world who have done substantial conceptual or theoretical work in design methodology. The SBs, researchers who develop technology using the DoR approach, were selected by combing through CHI papers published in the last decade. The selection, done by three members of our team, identified papers that 1. Adopt an approach where a system is built to investigate a research question; 2. Have a recognizable seed idea and an explicit or implied design process; and, 3. Present some form of user testing. The selection process yielded 97 papers across HCI domains (e.g. accessibility, web search, ubiquitous computing, embodiment, social interactions, 3D interaction).

An email request for interview was sent to the first authors of all the papers. If we did not receive a reply, a request was sent to the second authors of the papers. Three local researchers (2 SBs and 1 MI) were used as pilot interviewees. Twenty-two HCI researchers, 20 from university research labs and two from industry were interviewed as SBs. Eight leading HCI researchers were interviewed as MIs. Five of the eight MIs also did the SB interview. Our interviewees were from the US, Canada, the UK, Japan, Singapore, South Korea, and Australia. Eighteen participants were interviewed through video teleconferencing, two by telephone, and two in face-to-face sessions. Oral consent for audio recording was first obtained from each interviewee.

The SB interview guide comprised 32 questions (multiple-choice survey and rating, and open-ended questions) on four main themes: 1. Description of the research undertaken specifically in the CHI paper; 2. Methods used to design the system; 3. Testing of the system; 4. Design in HCI in general. The MIs' interview was divided into two sections: 1. The first six questions probed on research of the MI in general, and on her perspective on design in HCI. 2. We explained the purpose of our study using Figure 1. Nine questions guided the discussion and prompted the interviewees about design processes to move from an idea to a research prototype and the role of user testing in radical design. We asked the permission of some MIs to quote their responses non-anonymously, and they were given the opportunity to review the paper.

All interviews were transcribed and coded for themes of importance in several iterative passes. The research process used by each interviewee was identified. A preliminary process model was constructed through a first high-level analysis. We refined our model in subsequent iterations through deeper-level analyses and discussion of the insights gleaned. Survey data collected in the interviews were collated on a spreadsheet and summarized. These helped us to get a sense of how each component of our model was manifested in each project. We present a few relevant study findings below in a synthesized account of the qualitative and survey data.

Findings

The basis used by most of our SB interviewees to make decisions on what to include in a design was their own intuition and experience with their original conceptual rationale, or by generating a list of design requirements based on studies of users. Figure 2 shows the different categories of how researchers made design choices.

However, none of our SBs could provide a reliable method of how to interpret study results (e.g. unexpected study results could be because the original design idea is unsuitable for the particular user group or the system was improperly implemented). All interviewees who provided an answer said that an iterative design process was needed, and some said that using low-fidelity prototypes for user studies could enable the researcher to make an informed guess "based on the reaction we got from people". Elements of intuition and experience were also put forward by some MIs as key to making judgments about study results.

All MIs agreed that there is currently no process to guide researchers on how to embody a seed idea in a design, resulting in many ineffective designs. The MIs' perspectives can be summarized into six 'high-level strategies', all pointing to the uncertainty inherent in the ad-hoc way in which HCI research is conducted: 1. One needs to try "existing solutions first so that one doesn't innovate unnecessarily. When you fail with existing solutions, that's when you innovate and fly." (Scott Klemmer, Stanford U.) 2. One should "stick" to one's idea and persist: "You simply have to face your own ideas and go forth and most of the time you will fail. But the few times you succeed make up for the failures." (Norman, Norman Nielsen Group) 3. One should gen-Figure 2. Basis for design decisions erate many designs and test often: "The principle is to try to do this (minimal prototyping) with little effort as possible. So you can try a lot of designs" (Jacob, Tufts U.). 4. One should make use of intuition, skills and experience. But for Norman, "this is where the huge experiences and skills and intuitions of the designers come in. Is that a method? No." David Frohlich (U. of Surrey) contrasted "inspiration for design" and "requirements for design". In the former, one may "generate ten times more designs" than the latter which employs UCD to produce just one or two from the requirements. Frohlich states that intuition is the only way to bridge these two approaches. 5. One should be pragmatic and be flexible to adopt any method that makes sense without limiting oneself to a "fixed methodology". 6. One should persist based on faith. Norman suggested that the only evidence that the researcher needs to move forward with her idea is to see (from test results) whether at least "one or two peo-ple…really believe deeply in what you are doing".

Proposed Model

Through iterative modeling of current methods we extracted in our interview data, we derived a model (Figure 3) of how design decisions (Label F in Figure 1) may be made to better arrive at NEVO. As in Figure 1, the radical design process begins with a conceptual rationale in the form of an integrated Research Aim, Motivation or Rationale from which a seed idea (A) is generated. A set of idea-defining characteristics is produced (B) to guide the design process to further specify the original concept. Exhaustive articulation of these characteristics, however, is not always possible. More importantly, over-specification of idea-defining characteristics may even have the undesirable side effect of overconstraining creativity in the design process. The model addresses this by recognizing the need for an informal Team Understanding and Consensus (B). Together, the Idea-Defining Characteristics and the Team Understanding and Consensus embody the seed idea, which then guides the Gatekeeper pro-Figure 3. Finding-NEVO model cess. The Gatekeeper process (D) is used to vet individual design ideas in advance of development. These design ideas (C) may be the product of methods such as brainstorming and ideation, with respect to a particular design issue (which may be identified through methods like ethnographic studies, design principles, experience or simple intuition).

Our key contribution is in the insertion of a formalized 'evaluation' at block D into the accretion process of the seed idea into a full design. We do not prescribe hard and fast way to implement the gatekeeper. This could be consensus among a group of stakeholders, the judgment of a single individual, or discussion in the research team. Each design idea may be determined by the Gatekeeper process to be: positive (it is in line with, supports or even extends the specified seed idea) or negative (it violates the premises of the seed idea). In the event of a negative judgment, either an acceptable compromise is found (e.g., owing to implementation and other pragmatic reasons such as time limitations, a modified version of the design idea may be judged the best alternative), or if the violation is so severe as to contradict the original research question and no compromise can rectify this, then the design idea is rejected.

Additionally, the gatekeeper may decide that the design idea is orthogonal to the seed idea. Here, a second choice needs to be made. In DoR, an unnecessary design component can introduce confounds into the study results. If the design idea is a necessary aspect of the system, the process proceeds to development, if not, it is rejected. The prototyping process (E) is typically done together with iterative user testing. As discussed earlier, this is an incremental process, that in Norman's parlance, climbs the design hill on which the seed idea is planted [START_REF] Norman | Incremental and Radical Innovation: Design Research Versus Technology and Meaning Change[END_REF]. The result of this process is a system (NEVO) that can be used to answer the original research questions in formal user studies (F).

Discussion

Our model is distinct from previously proposed models in a number of ways. TbD argues for the use of psychological theories to inform the design of artifacts; the main aim is to produce new artifacts. In contrast, the purpose of Finding-NEVO is to reveal knowledge and understanding; it uses design in the service of research. Moreover, while the focus of TbD is on repeatability (i.e., being able to analyze, understand and maybe reproduce artifacts), ours is on validity (i.e., how to stay true to one's idea when testing it through the use of artifacts). TbD then can be considered to be a design method (others being ideation, contextual design, etc.) that can be used in Box C in our model to generate design ideas.

Much of our model is grounded in similar assertions as those of Norman's 'hillclimbing' model. However, we believe that even if the seed idea is potentially radical (through meaning change, technology push or random tinkering) and sits at the base of a higher 'hill', it may or may not develop to the top of the 'hill' using incremental HCD methods because of creeps (convenience, experience, feature and user input creeps described before). More importantly, for these same reasons, the design may jump back to some other 'hill' of a more mature but ultimately limited design trajectory. Our approach ensures that a potentially radical seed idea is allowed to blossom.

To be clear, our model is not inconsistent with Norman & Verganti's position. We simply suggest an additional gatekeeper process that may make radical designs more likely without necessarily posing HCD as irrelevant to radical innovation.

Interestingly, the five MIs who were also interviewed as SBs gave differing perspectives on HCD depending on their roles. As MIs, they were candid the inadequacies of HCD in bringing about radical designs needed for DoR, but as SBs, they employed and justified their approaches within an HCD framework. We posit two possible reasons. First, the field is in need of new models and methodological warrants to break out of the HCD paradigm. Second, as SBs/researchers they needed to publish their work, and employing HCD may make their work more acceptable to reviewers. Both interpretations suggest that the field of HCI needs to re-evaluate its dependence on the HCD paradigm. This paper contributes by adding to this discourse.

Beside the contributions above, this paper enriches our understanding of HCI as a scientific domain, and contributes to research and design methodologies in HCI:

Contributing to HCI as a field: HCI is a relatively young field that is still being defined. Grudin asked the question "Is HCI homeless?" [START_REF] Grudin | Is HCI Homeless? In Search of Inter-Disciplinary Status[END_REF]. Much discussion has occurred on how to come to terms with the interdisciplinary nature of HCI (e.g. [START_REF] Carroll | HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science[END_REF]). Our model provides a way for different disciplines to be integrated in one HCI research project. The conceptual rationale and seed idea (Boxes A and B) can come from any domain (e.g. social sciences/humanities). Methods from the creative disciplines (e.g. art/design) are employed in Box C, design thinking, to generate ideas for a prototype. If technical implementation is desired, engineering knowledge is required at Box E for prototype development, testing and integration. And finally research methods from the sciences are needed to carry the project through at Box F. HCI has also found it challenging to bridge many 'gaps' [START_REF] Lazar | Inherent Conflicts in HCI[END_REF]. As we mentioned in the introduction, the conflict between the needs of science and those of design is very evident in HCI, and yet HCI is touted as being "the science of design" [START_REF] Carroll | Human-computer Interaction: Psychology as a Science of Design[END_REF]. We believe this paper facilitates a better reconciliation of the two.

Contributing to HCI methodology: Finding-NEVO is a formalized process model based on current methodological practices of researchers and refined to enable HCI research to be done more transparently. The inclusion of a gatekeeping procedure not only serves to infuse systematicity into the research process, but also provides a roadmap for new researchers to the field to know how to proceed. Even among our interviewees, many found it difficult to articulate clearly the steps that they took in their research project. Some stated that "it was messy". Moreover, our approach can provide a common mental model of research and common terminology among project team members with differing backgrounds and perspectives. Broadly, the core contributions of Finding-NEVO is not only in the methodological systematicity it allows, but also in the common basis of understanding that it can provide to the field.

Gaver [START_REF] Gaver | What Should We Expect From Research Through Design?[END_REF] lamented that the RtD approach may be seen as unscientific since the criterion of falsifiability cannot be applied: the "synthetic nature of design is incompatible with the controlled experiments useful for theory testing". The danger that threatens the scientific validity of the approach is precisely that design is generative [START_REF] Gaver | What Should We Expect From Research Through Design?[END_REF]. However, setting up a constant gatekeeper in the system development process enables one to stay 'on track' with regards to the initial conceptual rationale. I.e., two re-searchers starting out with the same rationale may not end up with the same systems, but following our model, there is greater chance for the systems to be truthful embodiments of the original idea, leading to similar results in final scientific user tests.

Conclusion

The Finding-NEVO model for DoR articulates a methodology to select design as that yield prototypes that are faithful to a conceptual rationale and seed idea. To the degree that the rationale and seed are novel, the method is likely to produce radical designs that may impact design for the marketplace. Our model is constructed through an analysis of current practice to understand both the applicability and failings of current methodology. In doing so, we contribute both theoretically and methodologically to HCI. Future work includes the validation of the Finding-NEVO model across various HCI projects through longitudinal adoption.

Figure 1 .

 1 Figure 1. Going from concept to testable prototype


Madihah Sheikh

Abdul Aziz

Gitte Lindgaard
email: glindgaard@swin.edu.au

Tw Allan Whitfield
email: awhitfield@swin.edu.au

The Design and Usability Testing of DACADEa tool supporting systematic data collection and analysis for design students

Keywords: Software Engineering (Usability Testing), Human Factors in Software Design (User Interfaces), user-centered design, human-centered design

, stating that designers must understand technology, business and psychology to support design decisions. For designers to acquire the necessary statistical skills, design curricula should incorporate statistical courses teaching systematic data collection and data analysis. This paper presents the development and formative usability tests of the prototypes of a software tool called DACADE intended to support design students collecting and analyzing data systematically early in the design phase. It uses a 2D map and a Napping® technique to support effective and efficient communication between designers and target audiences in the design decision process by providing visual data and descriptive statistics without needing statistical knowledge.

Introduction

With the spread of the Internet, large-scale data collection, analysis, and product/ service testing are feasible and cost-effective. Specialist companies provide statistical packages, such as SPSS to analyse test results as well as offering data entry services and even data analysis. This complements the trend of data mining in both the retail and the social media advertising domains. Inevitably, however, the analyses tend to be biased towards quantitative rather than qualitative outcomes; statistical procedures employed vary from simple Cluster Analysis to Multidimensional Scaling and Multi-ple Regression, even Structural Equation Modelling. Design company clients will increasingly receive market research reports incorporating such analyses.

In our study, we first interviewed 20 design students and 10 design lecturers at one University, followed by a survey of 51 universities with design courses in Australia, North America, Europe, United Kingdom, and Asia. Results revealed that none of the curricula included statistics, thus supporting Norman's claim that design students do not understand statistical concepts. It is therefore legitimate to question future designers' ability to understand statistical information as well as their ability to engage actively and meaningfully in this emerging world of statistical sophistication. Norman contends that they cannot [START_REF] Norman | Why Design Education Must Change[END_REF]. The Data Collection and Analysis for Designers (DACADE) software tool described here aims to help future designers collect and analyze product-related data from prospective consumers.

Existing visual statistical tools

Research shows that designers prefer visual information [START_REF] Hashim | The Multi Dimensional Scaling: An Interactive Method for Establishing Perceptions of the Appearance of Product[END_REF]. Following l i inen et al., [START_REF] Kalviainen | Visual Research: Means of producing shared meanings, Joining Forces[END_REF], this study thus focuses on tools allowing visual data collection and analysis. Several visual research tools that automate data collection and analysis are available in the literature, including Computer Aided Kansei Engineering (CAKE) with XML [START_REF] Chuang | Computer Aided Kansei Engineering with XML Technology[END_REF], the Web-based 2 Dimensional (2D) analytical tool [START_REF] Lin | Analyzing target user group's preferences and product form design specification through web-based 2-dimensional design decision tool[END_REF] allowing respondents to position images of products on a 2D map using Semantic Differentials (SD), 3), and others. Tools based on SDs scales require designers to pre-assign sets of bipolar adjectives to product images (e.g. Bad -Good, Ugly -Pretty). l i inen et al., [START_REF] Kalviainen | Visual Research: Means of producing shared meanings, Joining Forces[END_REF] proposed that product images with pre-assigned sets of bipolar adjectives could constrain the respondents' possible suggestions and that respondents should be free to evaluate products as they see fit [START_REF] Kalviainen | Visual Research: Means of producing shared meanings, Joining Forces[END_REF]. Some of the available tools assume that designers know statistics: none of these have been tailor-made for future designers.

1.2

Napping® and 2D map DACADE includes two techniques for collecting and analyzing opinion data. One allows evaluators to position product images on a blank screen in any way they like. Products positioned close to others are perceived to be similar; those positioned far apart are seen to be different. Once all images have been placed, evaluators enter adjectives to represent the products/groupings meaningfully. Based on these evaluatorselected terms, designers can then select suitable bipolar adjective pairs for further study. This technique is called projective mapping or Napping®. It is used widely in sensory analyses in the food industry [START_REF] Pagès | Collection and analysis of perceived product inter-distances using multiple factor analysis: Application to the study of 10 white wines from the Loire Valley[END_REF]. Based on the sets of original adjectivepairs, other respondents' product perceptions can be tested in later design stages.

Involving a 2D map that enables the direct reading of consumers' perception, the second technique relies on designer-selected adjective-sets from which the designer generates a perceptual map of consumer perception. Perceptual maps are popular in marketing for studying the perception of products ranging from consumer products (e.g. toothpaste, cars) to activities (vacation spots, movies) [START_REF] Rice | An Introduction to Brand/Perceptual Mapping[END_REF]. DACADE uses perceptual maps because they are easy to interpret, and they provide visual output as designers prefer. Marketing researchers and practitioners tend to rely on statistical software packages such as SPSS to run highly advanced statistical analyses that may be too complex for statistically naïve design students. Instead, DACADE uses nonparametric Guttman-Lingoes Series MDS [START_REF] Guttman | The basis for Scalogram Analysis[END_REF][START_REF] Guttman | A general nonmetric technique for finding the smallest coordinate space for a configuration of points[END_REF] that rely on visual and spatial information for generating perceptual maps.

2

Initial design and first formative DACADE usability test DACADE was designed using User-Centred Design (UCD) [START_REF] Rogers | Interaction Design beyond human-computer interaction[END_REF] and the ISO-9241/11 [START_REF]Ergonomic Requirements for Office Work with Visual Display Terminals (VDT) s-Part II Guidance on Usability[END_REF] definition of usability, i.e. effectiveness, efficiency, and user satisfaction. Five graduate design students evaluated the initial evaluation low-fidelity paper prototype in individual sessions, audio recorded with permission. A separate prototype was produced for each of four tasks, namely (1) designing a new study, ( 2) editing an existing study, (3) collecting data (i.e. running the study) using both abovementioned techniques, and (4) conducting simple descriptive statistical analyses. The analysis component calculates the median, mode, and mean of the research data as well as providing a visual output of the distribution of objects tested. Written task instructions and relevant task scenarios were given; answers were written on a blank sheet of paper. On each screen, participants placed a pen on the object they wished to select. Task requirements were generated separately for each task and given to participants one at a time. All tasks were completed in the same order [START_REF] Brooke | SUS-A quick and dirty usability scale[END_REF][START_REF] Chuang | Computer Aided Kansei Engineering with XML Technology[END_REF][START_REF] Effendi | The Product Effect: Do Designed Products Convey Their Characteristics To Their Owners? Degree of Doctor of Philosophy[END_REF][START_REF] Guttman | The basis for Scalogram Analysis[END_REF]. The stimuli comprised some 10 pictures of cars borrowed from Effendi [START_REF] Effendi | The Product Effect: Do Designed Products Convey Their Characteristics To Their Owners? Degree of Doctor of Philosophy[END_REF]. Upon completing a task, participants filled in the System Usability Scale (SUS) [START_REF] Brooke | SUS-A quick and dirty usability scale[END_REF] to assess user satisfaction. Notes were taken throughout the sessions; audio records were transcribed verbatim.

Design and test of the DACADE tutorial

The tutorial designed next, covers the four DACADE components as well as a basic explanation and illustration of descriptive statistics. The tutorial was first tested with four new participants who did not complete the tasks in the formal tool. In order to illustrate the calculation of the mean, mode, and median, a picture of bananas varying in size was produced as shown in Fig. 1. Surprisingly, the students perceived this as having sexual connotations, which was both inappropriate and embarrassing for the researcher. The image was therefore changed to a set of books as shown in Fig. 2. This appeared to convey the intended meaning. Several exercises were included in the tutorial. One required participants to calculate the mean, mode, and median of a set of numbers differing from those in the illustration, as shown in Fig. 3. Unfortunately, the median had inadvertently been placed in the center of the distribution in Fig. 2. Rather than calculating the median of the unsorted data set as required, one participant simply counted the number of books from each end of the distribution, selecting the central number ("13") without considering the values of the numbers in the data set. Evidently, the concept was still not conveyed clearly enough. The image was therefore changed again to avoid the median being in the central position.

Second formative usability (and comprehension) test

As there were no additional usability issues or apparent comprehensibility issues, the next test, involving the revised tutorial and the revised DACADE tool, included a new sample of six participants together with the original instructions and tasks. Participants worked through the tutorial and exercises first. They were then given the cover story and the task instructions and asked to work with the tool. The results showed a distinct improvement to DACADE: only five (9.43%) of the 53 issues were usabilityrelated. However, some 19 (35.85%) issues concerned a lack of comprehension. Another 17 (32.08%) were suggestions, eight (15.09%) were participants' uncategorized comments, and four (7.55%) were the researcher's observations. Among the comprehension issues, one showed that participants had difficulties understanding the purpose and meaning of the 2D map, shown in Fig. 4. In both the tutorial and the tool, references were made to a fictitious study of water bottles. In one tutorial exercise, participants were asked to identify the most interesting and modern water bottle, the most traditional and boring bottle, and so forth. We explained that, for both sets of adjectives (boring-interesting and traditional-modern), 'more is better', with the most negative term placed on the bottom and to the left, with the most positive term placed at the top and to the right.

Fig. 4: A perceptual map showing the placement of eighteen water bottles

We also explained the meaning of clusters (mode), and so on. Still, when asked to interpret the contents of the map in Fig. 4, said to represent the placement of bottles by another participant, one participant, noting certain similarities between bottles, said: "Its like more of a sample, I would say they should be in the same place, because they are all the same. Similar bottles should be in the same place, like now they are kind of everywhere. They are all the same". Similarly, when asked to identify the best bottle, one participant said: "The answer is R, because it is made of metal, so you can bring it on to outdoors or, go [on an] expedition, that's my judgment". Another participant said: "The best bottle, it depends on environmentally, or the needs, like a flask or something, for me I'll just go for B". Another participant, who was unsure of the position on a 2D map, asked, "Does it matter if it is located further up on the quadrant?" This participant further commented that the perceptual map, "…itself looks very statistical and technical; you need to make it more fun and interesting. I want to be able to enjoy the graphics. It really depends on how you design this later" At this stage, we are at a loss as to how to convey the meaning of visually presented data more clearly, but this problem must be solved before redesigning the prototype.

Other misunderstandings were due the prototype being presented on paper. For example, as shown in Figure 5, two participants did not recognize that the meaning of a greyed-out button was the same as on a computer.

Passive Buttons or Grey out button s

Figure 8: An inte r face of a so ware tool with an example of a grey out button Figure 8 shows the last sc r een of creating a new study with some buttons available .

e button with a red circle , i s shaded with grey, t o show that it is pass i ve, and cannot be clicked. Fig. 5: A screen with a greyed-out button Participants were also confused about adding a new map on a paper screen. That function enables designers to add one or more 2D maps to be tested in their study. However, instead of adding a new map, they just changed the adjectives on the existing map. This will hopefully be overcome once we use high-fidelity prototypes.

One participant wanted a definition of the term 'adjecti es', "The word adjective might confuse people. I think 90% of people don't read instruction manuals; there should be some rough instructions. You can add a link or a question mark symbol so that people can click on them to get some hints. Or roll over to get some tips". Tooltips will be provided in the implemented version of DACADE. Another suggestion was to present the tool online, to enable the designers to involve more people in their studies. While large samples make sense, one of the aims of DACADE is to educate design students and help them to interact and communicate with prospective consumers face-to-face. In future, DACADE might be extended once designers begin to use it, but for the moment, this suggestion will not be considered. Failing to understand the concepts of groups of participants and total number of participants, some participants suggested that DACADE should allow an unlimited number of participants, which it does, but also that the number of participants in a group should not be constrained. That is, they wanted the concept of participant-groups removed. This is infeasible because the tool is specifically designed to guide them in conducting systematic research. This includes the ability to compare between-group responses, e.g. seniors versus teenagers, or allowing the same number of males and females.

Participants made several good suggestions, for example, adding clear instruction to consumers to position images on a 2D map or a blank screen because consumers will not be reading the tutorial. For the task of creating a new study, it is clear that additional instructions are needed. However, in the light of the comment that "…90% of people don't read instructions", it is difficult to balance the amount of instructional text and diagrams. One possible solution is to animate the tutorial, thereby changing it to a 'show and tell' show, with oice-overs and animated diagrams only. This may be done later. Note, however, that if students are unfamiliar with terms such as 'adjecti es', e en a oice-over will not necessarily help them. Another suggestion was to enable simultaneous display of all images to be placed on the map instead of presenting them one by one. This will be done to give consumers a general idea of the entire range of product images to be positioned from the start of the task. In previous research [START_REF] Chuang | Computer Aided Kansei Engineering with XML Technology[END_REF][START_REF] Lin | Analyzing target user group's preferences and product form design specification through web-based 2-dimensional design decision tool[END_REF], images were presented one at a time or in pairs.

One important was that participants looked bored while working through the tutorial. They complained that it was too long and had too much text, again suggesting that animation may be best for creating a playful, yet useful and comprehensive tutorial while retaining its brevity. Finally, it was clear that some participants failed to read the tasks thoroughly; they needed several reminders to refer to the task instructions on how to proceed.

Second formative usability (and comprehension) test

Usability goals were set for both tests in terms of the number of questions, hints, and errors allowed during the usability tests. Table 1 shows the distributions of these for each of the four tasks and the results for both usability tests of the DACADE tool. The between-task variation in these numbers in Test 1 was based on our best guess of the difficulty associated with each task. Clearly, the task of 'Running one or more analyses' or RA was most difficult, as it was the only task that failed in Test 1. In Test 2, the goals were based on the outcome of Test 1. Note that the goals were more stringent in Test 2 than in Test 1 because we assumed that the tutorial added in Test 2 would enable participants to understand the descriptive statistics and interpret the perceptual maps. The fact that all four tasks passed in Test 2 could lead one to misinterpret the tool's apparent effecti eness. In cases in which user comprehension may be as important as, if not more important than, usability, comprehension should be added in the task protocol as a separate variable in the performance assessment.

Conclusion and Next Steps

This paper described the design of a visual statistical technique to help future designers understand consumers' product perceptions. We showed that it was relatively easy to identify and eliminate usability problems, but that finding effective ways to convey even simple statistical knowledge is fraught with unforeseen difficulties. The issues from the second usability test are now being addressed, but rather than producing a slightly another prototype, DACADE will be implemented, and professional designers will be consulted on its look and feel. It is anticipated that the results of a user acceptance study will be available by the time INTERACT'2013 takes place.

Fig. 1 .Fig. 2 .Fig. 3 .

 123 Fig. 1. The original illustration of the mean, median and mode

Table 1 :

 1 The usability goals set for both tests according to tasks

	Tasks			Test1				Test2
		NS	ES	CD	RA	NS	ES	CD	RA
	Questions	3	2	1	3	2	0	0	2
	Hints	2	1	1	1	1	1	0	1
	Errors	2	2	1	2	0	0	0	0
	Results	P	P	P	F	P	P	P	P

*Note: NS=New Study, ES=Edit Study, CD=Collecting Data, RA=Running analyses, P=Pass, F=Fail

Please note that the LNCS Editorial assumes that all authors have used the western naming convention, with given names preceding surnames. This determines the structure of the names in the running heads and the author index.

Acknowledgements. This PhD research is funded by the Swinburne University of Technology Australia and International Islamic University Malaysia.