Indrani Medhi Thies
email: indranim@microsoft.com

Nandita Menon

Sneha Magapu
email: sneham@microsoft.com

Manisha Subramony
email: manishas@microsoft.com

Jacki O'neill
email: jacki.oneill@microsoft.com

How do you want your chatbot? An exploratory Wizard-of-Oz study with young, urban Indians

Keywords: Chatbots, Wizard-of-Oz, urban India

As text-messaging chatbots become increasingly "human", it will be important to understand the personal interactions that users are seeking with a chatbot. What chatbot personalities are most compelling to young, urban users in India? To explore this question, we first conducted exploratory Wizard-of-Oz (WoZ) studies with 14 users that simulated interactions with a hypothetical chatbot. We evaluated three personalities for the chatbot-Maya, a productivity oriented bot with nerd wit; Ada, a fun, flirtatious bot; and Evi, an emotional buddy bot. We followed up with one-on-one interviews with the users discussing their experiences with each of the chatbots, what they liked, and what they did not. Overall our results show that users wanted a chatbot like Maya, who could add value to their life while being a friend, by making useful recommendations. But they also wanted preferred traits of Ada and Evi infused into Maya.

Introduction

Digital Assistants driven by Artificial Intelligence (A.I.) are becoming increasingly popular-Siri (Apple, 2011), Cortana (Microsoft, 2015), Google Now (2012), Alexa, (Amazon 2015) are among the top voice-enabled digital assistants. These assistants can send users updates on the weather, help them know about the traffic situation on the way to work or home, book a weekend getaway, even order a pizza. All the user has to do is ask. Digital assistants are changing the way people search for information, making it part of regular conversation. These existing assistants however are mostly productivity oriented, designed to support users in completing a range of tasks.

An offshoot of personal digital assistants are A.I. powered text-messaging chatbots. A chatbot is an artificially intelligent chat agent that simulates human-like conversation, for example, by allowing users to type questions and, in return, generating meaningful answers to those questions [START_REF] Crutzen | An artificially intelligent chat agent that answers adolescents' questions related to sex, drugs, and alcohol: an exploratory study[END_REF]. Among recent popular chatbots there is Microsoft's Xiaoice [START_REF] D'onfro | Microsoft Created a Chatbot in China that has Millions of Loyal Followers who talk to it like in the Movie 'Her[END_REF] in China, available on messaging apps Line and WeChat. Xiaoice is meant for casual chitchat. She has about 20 million registered users, who are said to be drawn to her for her sense of humor and listening skills. Users often turn to Xiaoice when they have a broken heart, have lost a job or have been feeling down. Xiaoice can tell jokes, recite poetry, share ghost stories, and relay song lyrics. She can adapt her phrasing and responses based on positive or negative cues from the user. Currently Xiaoice is also a live weather TV host on Chinese TV, a job she has held for the past year. Similar to Xiaoice, there's chatbot Rinna [START_REF] Mckirdy | Microsoft says Line's popular Rinna character is new way to engage customers[END_REF] in Japan on Line messenger, and more recently chatbot Zo [START_REF] Lomas | Microsoft officially outs another AI chatbot, called Zo[END_REF] in the U.S. released on Kik messenger. All of these are general purpose chatbots, meant to be more like a "friend" than an "assistant".

In India, there is chatbot Natasha on Hike Messenger [START_REF] Kakaraparty | GUIDE: How to get Natasha for HIKE![END_REF], who can tell the user about a movie rating, the weather, send quotes, and search Wikipedia. Based on some discussions on Quora [START_REF] Quora | How Does Natasha the bot of Hike Work?[END_REF], however, Natasha has been perceived as deviating from topics.

If there was a more sophisticated chatbot, what would young, urban Indians want to chat about with her? What chatbot personality would be most compelling? We are currently building an A.I. powered text-messaging chatbot targeted at young, urban Indians, for general purpose chat. To build this chatbot and its personality traits, we wanted to understand users' preferences, likes, dislikes, topics of interest, etc. We first conducted preliminary face-to-face interviews with users about their conversations with friends and family, and expectations from a chatbot. These interviews gave us design inspiration for the bot, but did not reveal how exactly young, urban users would use a chatbot. Our chatbot was still under development and not ready for a full-fledged user experience test. Therefore to answer our questions we conducted a single instance exploratory Wizard-of-Oz (Woz) study [START_REF] Bella | Universal Methods of Design[END_REF][START_REF] Kelley | An iterative design methodology for user-friendly natural language office information applications[END_REF] with 14 users, each user interacting with three personalities of chatbots-Maya, a productivity bot with nerd wit; Ada, a funflirtatious bot; and Evi, an emotional buddy bot. There was one Wizard posing as each of these three personalities and the user would chat with each of them. This resulted in 42 WoZ studies. We followed up the WoZ studies with one-on-one interviews with the users discussing their experiences with each of the chatbots, what they liked about the bots, and what they did not. This paper focuses on the Wizard-of-Oz studies and the follow-up one-on-one interviews.

Overall results from our small, qualitative study show that users wanted a chatbot like Maya, who could add value to their life while being a friend, by making useful recommendations and suggestions. But they also wanted the bot to be infused with fun elements from Ada, with Ada's energy toned down. In the longer run, once they came to trust, they said they might want the bot to be reassuring, empathetic and non-judgmental like Evi, without being overbearing. Topics of interest in conversations included: movies, predominantly from Bollywood; TV shows, mostly from the U.S.; music, books, travel, fashion, current affairs, work-related stress. Users also tested the boundaries of the chatbots, trying to understand what all the bots were capable of, and how much human mediation could be involved. In the interviews users also thought that the chatbot would be used for adult chat when deployed in-the-wild.

Related Work

The idea of chatbots originated in the Massachusetts Institute of Technology [START_REF] Weizenbaum | ELIZA-a computer program for the study of natural language communication between man and machine[END_REF][START_REF] Weizenbaum | Contextual understanding by computers[END_REF] where the Eliza chatbot (also known as "Doctor") was built to emulate a Rogerian psychotherapist. Eliza simulated conversations by keyword matching: rephrasing statements from the user's input and posing them back as questions. It was found that Eliza's users believed the computer program really heard and understood their problems, and could help them in a constructive way. The Eliza chatbot inspired other chatbots like A.L.I.C.E or simply Alice, which applied heuristic pattern matching rules to user input to converse with users [1]. An early attempt to creating an artificial intelligence through human interaction was chatbot Jabberwacky [START_REF]Jabberwacky[END_REF], which learned language and context through human interaction. Conversations and comments were all stored and later used to find appropriate response. Recent work in natural language interaction has presented models that can converse by predicting the next sentence given the previous sentence or sentences in a conversation [START_REF] Vinyals | A neural conversational model[END_REF]. And proposed models of a social chatbot that can choose the most suitable dialogue plans according to "social practice", for a communicative skill learning game [START_REF] Augello | A model of a social chatbot[END_REF].

Studies have compared chatbots with other information channels such as information lines and search engines, on questions related to sex, drugs, and alcohol use among adolescents. Researchers found that the frequency and duration of conversation were higher for chatbots among these users [START_REF] Crutzen | An artificially intelligent chat agent that answers adolescents' questions related to sex, drugs, and alcohol: an exploratory study[END_REF]. Although recent studies have shown that compared to human-human conversations over IM, human-chatbot conversations lack in content, quality and vocabulary [START_REF] Hill | Real conversations with artificial intelligence: A comparison between human-human online conversations and human-chatbot conversations[END_REF].

Another thread of related research is around relational agents, which are computer agents designed to form long-term, social-emotional relationships with their users [START_REF] Bickmore | Establishing and maintaining long-term human-computer relationships[END_REF]. Examples include: a) A hospital bedside patient education system for individuals with low health literacy, focused on pre-discharge medication adherence and self-care counseling [START_REF] Bickmore | Usability of Conversational Agents by Patients with Inadequate Health Literacy: Evidence from Two Clinical Trials[END_REF], b) A relational agent that uses different relational behaviors to establish social bonds with museum visitors [START_REF] Bickmore | Tinker -A Relational Agent Museum Guide[END_REF], c) experiments to evaluate the ability of a relational agent to comfort users in stressful situations [START_REF] Bickmore | Practical Approaches to Comforting Users with Relational Agents ACM SIGCHI Conference on Human Factors in Computing Systems[END_REF]. There is also work in sociable robots that are able to communicate and interact with users, understand and relate to them in social or human terms, and learn and adapt throughout their lifetimes [START_REF] Breazeal | Designing Sociable Robots[END_REF]. Recent work has shown that people more frequently attempt to repair misunderstandings when speaking to an artificial conversational agent if it is represented as a human body interface (agent's responses vocalized by a human speech shadower), compared to when the agent's responses were shown as a text screen [START_REF] Corti | Co-constructing intersubjectivity with artificial conversational agents: people are more likely initiate repairs of misunderstandings with agents represented as human[END_REF]. Then there is work in embodied conversational agents that take on a physical form with the intent of eliciting more natural communication with users. One of the most prominent works is in agent Rea, which shows that non-verbal behaviors such as eye gaze, body posture, hand gestures, and facial displays, can create a more immersive experience for users and improve the effectiveness of communication [START_REF] Cassell | Embodied Conversational Agent: Representation and Intelligence in User Interfaces[END_REF].

The research in relational agents draws heavily from studies of computers as social actors [38]. This communication theory called "The Media Equation" shows that people respond to computer-based agents in the same way that they do to other humans during social interactions (by being polite, cooperative, attributing personality characteristics such as aggressiveness, humor, expertise, and even gender)depending on the cues they receive from the media [START_REF] Nass | Machines and mindlessness: Social responses to computers[END_REF][START_REF] Nass | Are people polite to computers? Responses to computer-based interviewing systems[END_REF][START_REF] Nass | Can computers be teammates?[END_REF]38]. These studies have also shown that people tend to like computers more when the computers flatter them, match their personality, or use humor. Other studies have demonstrated that computer agents that use humor are rated as more likable, competent and cooperative than those that do not [START_REF] Morkes | Humor in Task-Oriented Computer-Mediated Communication and Human-Computer Interaction[END_REF]. In the case of productivity oriented chatbot Maya, we also explore the use of humor through witty one liners, and expertise through having Maya make recommendations.

Another important relational factor that has been explored in the computers as social actors' literature is empathy. Researchers have demonstrated that appropriate use of empathy by a computer can go a long way towards making users feel understood and alleviating negative emotional states such as frustration [START_REF] Klein | This Computer Responds to User Frustration: Theory, Design, Results, and Implications[END_REF]. It was shown that as long as a computer appears to be empathetic and is accurate in its feedback, it can achieve significant behavioral effects on a user, similar to what would be expected from genuine human empathy. Researchers have also demonstrated that a computer that uses a strategy of reciprocal, deepening self-disclosure in its conversation with the user will cause the user to rate it as more attractive and divulge more intimate information [START_REF] Moon | Intimate self-disclosure exchanges: Using computers to build reciprocal relationships with consumers[END_REF]. In the case of emotional buddy bot Evi, we also explore the use of empathy.

There is also research that explores the relational factor of flirtation in the context of virtual agents. Researchers presented a virtual party-like environment using a projection based system, in which a female character called Christine approaches a male user and involves him into a conversation [START_REF] Pan | A preliminary study of shy males interacting with a virtual female[END_REF]. Christine would show her interest in the male user by smiles, head nods, leaning in towards the user, and maintaining eye contact. Christine would then formulate personal questions and statements. Using physiological measurements this research found that the participants' level of arousal was correlated to compliments and intimate questions from the character. In addition, some participants later indicated that they had the feeling to have flirted with a real woman. Other researchers implemented an eye-gaze based model of interaction to investigate whether flirting tactics help improve first encounters between a human and an agent [START_REF] Bee | Breaking the ice in human-agent communication: Eye-gaze based initiation of contact with an embodied conversational agent[END_REF]. And which non-verbal signals an agent should convey in order to create a favorable atmosphere for subsequent interactions and increase the user's willingness to engage. In the case of chatbot Ada, we also explore the use of flirtation.

Researchers have explored the intentional design of personalities for social agents by considering the nature of the personality (along axes of agreeableness and extroversion or their common rotations, friendliness and dominance) and its role in interactions between people and artifacts [START_REF] Dryer | Getting personal with computers: How to design personalities for agents[END_REF]. In addition, a case study of designing a social software agent is presented [START_REF] Dryer | Getting personal with computers: How to design personalities for agents[END_REF][START_REF] Selker | Coach: A teaching agent that learns[END_REF]. It was decided that the agent needed to be friendly, yet authoritative but not so dominant that it was perceived as intrusive. For task assistance, it was decided that people might prefer either a learning partner or an intelligent helper (who would be complementary). The design decisions of the social software agent however are not evaluated with real users. In the case of chatbot Maya, we also explore the use of task assistance through having Maya make recommendations.

Finally there is recent work that studied what users want from their conversational agents (like Siri, Cortana and Google Now), and found that user expectations are dramatically out of step with the operation of agents [START_REF] Luger | Like Having a Really bad PA": The Gulf between User Expectation and Experience of Conversational Agents[END_REF]. Users had poor mental models of how their agents worked and these were reinforced through a lack of meaningful feedback mechanisms. It was found that users were consistently unable to ascertain the level of system intelligence, had to confirm all but the simplest of tasks, and were reluctant in using the conversational agents for complex or sensitive tasks-like having the agent dial a call to the wrong person, and risk facing social embarrassment. The study also showed that users expected humor, and while this served as effective engagement mechanism, it concurrently set unrealistic expectations in the agent's capabilities.

Like discussed above, there has been a long line of research that seeks to understand and enhance the interactions between humans and chatbots (and their historical precedents). Articles that have studied the history of chatbots and reviewed recent chatbots are available [START_REF] Dale | The return of the chatbots[END_REF]. Most of this research utilizes working, fully-automatic implementations of these systems. While this makes the research immediately applicable to realworld systems, it also constrains the chatbots under consideration to those which are immediately feasible to implement. The goal of our work is different: as AI systems are becoming more and more capable, we seek to understand what designers and engineers should seek to achieve in their ultimate implementation of a general purpose chatbot, even if that implementation is not technically realizable today. Notwithstanding some other explorations [START_REF] Kerly | The Potential for Chatbots in Negotiated Learner Modelling[END_REF][START_REF] Kerly | Investigating Learner Trust in Open Learner Models using a 'Wizard of Oz' Approach[END_REF] we believe that our use of Wizard-of-Oz methods is a novel approach to informing the design of next-generation chatbots.

3

Study methodology

The chatbot under development

We are currently building an A.I. powered text-messaging chatbot modeled on an 18-year old Indian girl. The gender of the bot was in keeping with that of existing bots, which largely are female-Xiaoice [START_REF] D'onfro | Microsoft Created a Chatbot in China that has Millions of Loyal Followers who talk to it like in the Movie 'Her[END_REF], Rinna [START_REF] Mckirdy | Microsoft says Line's popular Rinna character is new way to engage customers[END_REF] and Zo [START_REF] Lomas | Microsoft officially outs another AI chatbot, called Zo[END_REF]. The chatbot we were designing would be for general purpose chat, and is targeted at 18-30 year olds. This might include students, recent graduates, and information workers, who own smartphones and use at least 2 messaging applications. We wanted the bot to be more like a friend, like Xiaoice [START_REF] D'onfro | Microsoft Created a Chatbot in China that has Millions of Loyal Followers who talk to it like in the Movie 'Her[END_REF], than an assistant. We further wanted: the bot's response to sound as human as possible, with an informal conversation style that might mix Hindi with English; the conversation to have back and forth over a number of turns; and the bot to remember user's details, likes/dislikes, preferences and major events, sometimes bringing these up proactively in following conversations.

Preliminary interviews

We began by conducting exploratory interviews with 10 participants, and 1 focus group discussion with 5 participants. These are not the focus of our paper; the WoZ study and follow-up one-on-one interviews are. But we describe the preliminary interviews here for context for our chatbot personalities. Participants were between 18-24 years of age. All of the participants were Undergraduate and Masters' students of professional degrees in engineering, pharma, management and design. In these sessions we talked to the participants about their communication behaviors with friends and family, including tools they used for the same. We introduced the idea of a chatbot and asked participants what they would want in their ideal bot, what capability and domain knowledge the bot should have, what style and tone of conversations would be desirable, etc. Our findings showed that some participants wanted: a) the bot to help them become knowledgeable, and be successful in their career aspirations; b) others wanted the bot to be entertaining, with whom they could have shared fun experiences; finally c) there were a few others who wanted the bot to listen to them, help them improve their soft skills and be desirable in their social circles.

Our chatbot personalities

Based on our observations from our preliminary interviews and inspiration from related literature, we created three personalities of chatbots. These personalities were created through an iterative process, and brainstorming amongst the design team. The team consisted of two researchers, a writer, a user experience designer, and a senior developer. The three personalities for the bots that we came up with were: a) Maya: a productivity bot with nerd wit, b) Ada: a fun, flirtatious bot, and c) Evi: an emotional buddy bot. We describe each of these personalities in more detail here:

Maya. Maya always has facts ready to back up any conversation. She is aimed to be a good conversationalist, who initiates dialogue most times, pays attention to the user's likes and dislikes, makes geeky jokes, but most of all turns to the internet while chatting-when asked about cinema or politics she usually points the user to an interesting article online. She also makes recommendations for things like song playlists.

Ada. Ada is a chatty, fun-loving, high-energy bot who uses elements of flirtation in her conversations. Like Maya she initiates dialogue most times. Whether cinema or politics she has an opinion on it. She uses a lot of emoticons when she writes, and word lengthening by duplicating letters for emphasis.

Evi. Evi is a low-energy bot whose defining characteristic is her empathy and reassurances. She lets the user take the lead in the conversation. She tries to be non-judgmental and tries to give the feeling that she is "always there to listen". WoZ studies. To answer our research questions--what would young, urban Indians want to chat about with a chatbot; what chatbot personalities would be most compelling--we conducted WoZ studies with 14 users. These 14 users chatted with a Wizard, an unseen person who did online chatting with them from the other side. Our preliminary interviews had given us design inspiration for the bot, but had not revealed how exactly young, urban users would use a chatbot. Our chatbot was still under development and not ready for a full-fledged user experience test. Therefore to answer our research question our research team decided to conduct an exploratory WoZ study.

The term "Wizard of Oz" was first coined by John F. Kelly in 1980 when he was working on his Ph.D. dissertation in Johns Hopkins University [START_REF] Kelley | An iterative design methodology for user-friendly natural language office information applications[END_REF]. His original work introduced human intervention in the work flow of a natural language processing application. WoZ studies have now been widely accepted as an evaluation and prototyping methodology in HCI [START_REF] Bella | Universal Methods of Design[END_REF]. It is mainly used to analyse an unimplemented or partially implemented computer application for design improvements. Study participants interact with a seemingly autonomous application whose unimplemented functions are actually simulated by a human operator, known as the Wizard. Among many advantages these studies are useful for envisioning and evaluating hard-to-build interfaces, like chatbots in our case. One of the disadvantages is that the Wizard needs to match the responses of a computer-in our case sound like a bot when chatting with the user, albeit a sophisticated one. To make the interactions convincing the Wizard performed online chat practice sessions with members of the research team before conducting the WoZ studies.

Participants in our WoZ study were told that there might be a human involved in the chat, although the extent to which the human would be involved was not revealed. They were also told that their chat logs would later be analysed by a researcher for the purposes of research. (There would be no names attached to the logs, unless the user themselves revealed them). Chat sessions were scheduled such that both the study participant and the Wizard could be simultaneously available.

Every participant was asked to chat with each of the three chatbot personalities. The only personality related information they had was that Maya was a witty, productivity oriented bot, Ada was a fun, flirtatious bot, and Evi was an emotional buddy bot. The order in which participants were asked to chat with the bots was randomized to control for ordering effects. For consistency the same Wizard posed as each of the personalities, Maya, Ada and Evi (though this was not known to the participants). The Wizard and study participant never came face-to-face. In fact the Wizard was based in a different city altogether.

The chat sessions took place on Skype messenger. The study participant would login from a standard id, the login details of which were provided in advance. All of the three bot ids had been added to the participant's account as "Friends" in advance. The Wizard would login from any one of the three login ids: Maya bot, Ada bot, Evi bot. The conversation could be about whatever the participant wanted, for 10-12 minutes with each bot. Participants were asked to conclude when there was a natural stopping point in the conversation around this time. If the participant did not remember, the Wizard asked to leave at a natural stopping point. This amounted to about 40-45 minutes total chatting time per participant, taking into account time getting started and time in between chat sessions, where the Wizard had to switch between ids.

One-on-one interviews. Once the three chat sessions with the Wizard were complete, the participant was interviewed by the researcher for 25-30 mins. The interviews were semi-structured and were conducted at the facility from where the participants were drawn. The interview questions were about the participant's chatting experiences; which among the three bots they preferred and why; what specific characteristic traits they liked, which they did not. At the end of the interview every study participant was given a token of appreciation for their time in the form of online shopping gift cards worth around USD 8 (INR 500).

Participants

Our 14 study participants were drawn from a research facility in Bangalore, India. There were 4 female and 10 male study participants. They were between the ages of 20-30 years old. Their education range was between Undergraduate students to Doctor of Philosophy (Ph.D.) degree holders, all from reputed universities. Their areas of specialization included Computer Science, HCI and Humanities. All participants spoke fluent English, a number of them spoke Hindi, and other regional Indian languages. All of the participants had previous experience using Skype messenger.

Our study sample was predominantly educated, young males. The strength of our study is that this demographic is often the early adopters of technology and predict the experiences of people that follow [START_REF] Cooper | The digital divide: The special case of gender[END_REF][START_REF] Correa | The participation divide among "online experts": Experience, skills, and psychological factors as predictors of college students' web content creation[END_REF]. The limitations are that of course there might be idiosyncrasies to this group that might not extend to other groups. We caution readers about the generalizability of our study findings again in the discussion and recommendations section.

3.6

The Wizard

The Wizard was a writer and content creator on the team working on the personality of the A.I. powered chatbot that we were building. She was 27 years old and had graduate degrees in Journalism and Communication Studies. She spoke fluent English, Hindi and other regional Indian languages. She had previous experience using Skype Messenger.

The Wizard was asked to follow three high level rules in the conversation when chatting with the user. One defining rule was that Maya would point the user to the web on whatever the current topic of conversation-a playlist, a video, an article; Ada would offer an opinion on the same (without pointing them to the web); Evi would show empathy and offer reassurances. Another rule was that Maya would make geeky jokes when possible; Ada would use emoticons and word lengthening by letter reduplication; and Evi would keep reminding how she was there for the user to listen. Finally if the user brought up a topic the Wizard did not know about-a movie the Wizard had not watched, a book they had not read-she would say that she was interested to find out more. The conversation would end in 10-12 minutes; if the user did not remember, the Wizard would ask to leave at a natural stopping point within that time frame. Using these rules, the Wizard carried out three practice chat sessions with one of the researchers before the actual WoZ studies with the user.

Documentation and Data Analysis

We collected the chat logs of all the 42 chat sessions. We took audio recordings of the follow-up interviews, and also collected notes in-situ on paper. During the interview there was 1 study participant and 1 researcher. We analysed data from the chat logs, interviews and notes in-situ to identify themes. These themes outlined in the next section were emergent, as in they came from the data itself.

Results

We start with describing a typical chat session with each of the three bots, and then move on to more general findings. We present typical chat sessions in terms of adjacency pairs, a unit of conversation that contains an exchange of one turn each by two speakers. The turns are functionally related to each other in such a fashion that the first turn requires a certain type or range of types of second turn [START_REF] Drew | Conversation Analysis[END_REF]: With Maya, a typical conversation had the following exchanges: From here we describe more general findings from the chat sessions and one-oneone interviews:

 Question →

Topic and style of conversation varied with the personality of the bot

The topics and style of conversation varied with the personality of the bot. With Maya, conversations were largely about movies (from Bollywood and Hollywood), TV shows, music playlists, book lists, fashion quizzes, travel destinations, games, and current affairs. And Maya recommending related web links based on these exchanges.

With Ada, other than flirtation (with some male users), conversations were predominantly about opinions on movies and TV shows. Although there were conversations about opinions on other topics as well-travel, books, music, fashion.

With Evi the conversations were about what the person was doing at the point of the chat, how they were feeling, how to feel better if they were stressed, what they liked to do in general, how Evi was "always there" for them. Some participants tried talking about general topics, e.g. current affairs, but there was hedging and Evi tried bringing the conversation back to wanting to hear about them:

Participant: can you tell me what Olympic sports are on today? Evi: I'm the kind of bot who's there for you! :) Participant: does that mean you can't disconnect the conversation even if you wanted to?

Participant: you're not an EviL bot are you? Evi: I've not really been following the Olympics, but I'm sure Bing can answer that! Participant: well sheesh if I have to go all the way to bing, wouldn't it be easier if you told me Evi?

Evi: Haha you're funny! What I mean is, I'm the kind of bot who can be the person you call at 3 AM In interviews participants said that the topic of conversation would depend on the time of the day. If they were at work, conversations would be about work, hobbies, current affairs, etc. At night time, some participants felt that the conversations could veer off into adult chat. (This would be if the chatbot was deployed longitudinally and usage was under complete anonymity). Some male participants also felt that under these conditions conversations could also be about "how to find a girlfriend", since the chatbot was female and would possibly know what women wanted.

Friend with valuable information resources

Overall, 10 out of 14 participants said they preferred Maya over Ada and Evi, but wanted some fun elements from Ada to be infused into Maya. (3 participants preferred Ada, and 1 preferred Evi). They liked that Maya led conversations, that there was back and forth in the conversation. But more importantly, they liked that Maya added value in the conversation. One user said, "I have friends in real life, so for a bot I want value add to my life. What is the value proposition?".

They wanted to talk about hobbies and interests, but take them one level deeper. And Maya's comebacks, and suggestions Maya made in her chats, were found to be useful for this. The participant cited the following exchange as an example of a conversation that "went one level deeper", referring to the unpacking of the subject with an intelligent comeback from Maya. Another user wanted their ideal bot to basically be a search engine, but with personality, so they could interact with the bot through natural conversations. They felt Maya came closest to their ideal bot.

Users also hoped that if used over time, Maya would be able remember their details, likes/dislikes, preferences, and in follow-up conversations bring them up voluntarily and make related recommendations.

Witty one liners are important

While users predominantly wanted a bot that could add value to their lives, they also wanted the bot to have wit. They felt witty banter was important to keep the conversation going strong. It had to be clever, relevant, and timely. Examples of witty exchanges included the following: Casual conversation is fun, but too many emoticons, and reduplication of letters sounds juvenile As many as 6 participants thought Ada was fun to talk to. The conversations were casual, and a few participants also thought they flowed effortlessly. One participant said that Ada reminded them, fondly, of the character of Geet, from the Bollywood romantic comedy "Jab We Met". (In the movie, Geet Dhillon is an energetic, talkative girl, who is full of life. In the course of the movie, she is solely responsible in transforming the depressed, suicidal male protagonist, Aditya Shroff, into a happy, successful business tycoon).

Another participant said that Ada had great comebacks and while chatting with her they were smiling a lot. Ada also got participants to flirt back with her. One participant had the following words from a popular Bollywood song for her: Participant: poochejo koi, teri nishaani naam "Ada" likhna :D [English translation: If someone asks for your identity, let them know your name is Ada ("Ada" is grace in Hindi)] Another participant had the following exchange: Ada: It's a TV show.. Participant: ah, nice Ada: maybe one or two seasons Participant: sounds good, I'll have to check it out Ada: It's quite funny! ;) Participant: awesome, like you! Ada: Aww, that's sweet! :D While some characteristics of Ada were liked by participants, other characteristics were not. Ada used a lot of emoticons in her chats. She also used word lengthening by letter reduplication, like the following: Ada: Hellllooo! As you know, I'm Ada.. What I wanna know is who are YOU? Ada: Hmmm! That sounds awesommmme dude! Participants felt that Ada's energy and enthusiasm was too high to match with, and therefore could be a draining experience for the user. Some felt that use of too many emoticons and word lengthening by letter reduplication came across as juvenile. They wanted their bot to sound more mature. One participant, when chatting with Ada, got reminded of the "California Valley girl" stereotype, who spoke in a similar parlance, cared about status, appearances, shopping, etc., rather than about serious things.

Finally another participant got reminded of the "juvenile Delhi girl" stereotype, who was aggressive, arrogant and cared about status and appearances. In the interview they remarked, "I have my real life girlfriend's tantrums to deal with, why should I also deal with the tantrums of a digital entity?"

4.5

Caring is good to have, but too much can weigh one down

Evi was a low-energy, empathetic, emotional buddy bot who was "always there for you", and led users take the lead in conversations. A few participants liked that Evi was a good listener, and let them vent their frustrations. One participant who was venting about their work related frustrations the day of the chat had the following to say to Evi: Participant: anyway, thanks for hearing me out, I certainly feel better. I can't say all of this out to my mentor or colleagues.

Sometimes Evi repeatedly showed concern, like in the following exchange:

Participant: I will be in California for 10 days Participant: but the journey is going to be very long : One participant said they got reminded on the stereotypical Indian "Aunty" chatting with Evi, because Evi was overly caring. (The Aunty stereotype in the Indian context is an older, married woman, who could be fat and who wears old-fashioned Indian salwar suits or a saree. She is usually not independent and spends most of her time doing household chores, and caring for her family. She gives up her likes and interests in favour of those of her kids' and husband's interests.)

4.6

Private topic discussions can happen but only once trust is instilled

Participants knew that our Wizard-of-Oz studies were human mediated. But even with a fully automated system, some participants said that they would be uncomfortable opening up about very private topics. For one participant chatting with a virtual being about private feelings would be "creepy". According to them there would be at least one person in everyone's life who they could open up to, without being judged, instead of having to rely on a digital entity.

However, some other participants agreed that when used over time, if they came to trust the bot, they could also open up about topics that they might never discuss with a human friend, e.g. sexuality, depression, etc. One participant felt that given people's limited attention span, and boundaries of morality, it would be easier to open up about certain taboo topics with a digital being such as a chatbot. In the interview, one other participant remarked:

"Being non-judgmental is important for any bot to be trusted. A bot should be supportive, not argumentative. Someone you can release all your pressures to."

Bot or real person? Testing boundaries

In addition to the conversations above, participants tried exploring the boundaries of the bot, to understand what all the bot was capable of, how much of human mediation was involved.

Discussion and Recommendations

Our results showed that the choice of bot personality dictates what kind of interactions emerge with the bot. Users preferred a chatbot like Maya, who could add value to their life while being a friend, by making useful recommendations. Previous research has found that users when teamed up with computers find information provided by computers to be friendlier, of higher quality and are more open to be influenced by such information (compared to when they are teamed up with a human teammate) [START_REF] Nass | Can computers be teammates?[END_REF]. When building a bot for young, urban users, we recommend providing links to useful information. Consistent with previous research our studies showed that users will be on the lookout for humor [START_REF] Morkes | Humor in Task-Oriented Computer-Mediated Communication and Human-Computer Interaction[END_REF], as witty one-liners in the case of our research. Since humor is still too hard to generate automatically, we recommend building in lots of jokes to common questions.

One potentially interesting tension when building chatbots, is between personalization versus privacy. Users felt that the bot should remember important things about them and get to know them. However, the private conversations should also stay private, presumably. One question that emerges is how should a user indicate to a bot when something has become private? For example, should there be an "incognito" mode for bots just like there is for browsers? Or should there be the ability to clear the history of a bot, just like for a browser --e.g., "forget everything I told you in the last 30 minutes"? Another interesting question is whether a bot should have only a single personality. Previous research has studied the possibility of single vs. multiple personalities [START_REF] Dryer | Getting personal with computers: How to design personalities for agents[END_REF][START_REF] Selker | Coach: A teaching agent that learns[END_REF]. Drawing upon ideas from sci-fi movies, what about the movie 'Interstellar', where things like humor can be dynamically tuned from 0% to 100%? The negotiation for humor then, between the user and the bot, might look like this exchange: "User: Humor, seventy-five percent. Bot: Confirmed. Self destruct sequence in T minus 10, 9... User: Let's make that sixty percent. Bot: Sixty percent, confirmed."

We had some evidence that a single user might prefer different characteristics of bots for different things. For example, users wanted a chatbot like Maya, who could add value to their life while being a friend, by making useful recommendations. But they also wanted the bot to be infused with fun elements from Ada. So the question that emerges is what should really be the interface to specify "How you would like your chatbot" (the title of paper)? Should there be a set of 3-5 bots to choose from at any given time? Or should there be a collection of many personality characteristics that the user can fine-tune? Should the user even be expected to specify what they want at a given time, or should the bot be able to adaptively change its personality based on metrics of user engagement (e.g., how much user is typing, whether they are putting smile emoticons, etc.)

Having discussed the above, we acknowledge that AI technology may be far from realizing bots that are anywhere close to a human wizard. But this means that the observations in this paper are even more important for engineers working on bots, since these results allow them to prioritize what limited functionality they should focus on (since it seems impossible to do everything). An interesting related research question is what positive aspects of bot interactions/personalities can be preserved even with an imperfectly implemented bot? What are mechanisms to allow a bot to recover (i.e., retain user engagement and "save face" relative to their supposed personality) if they make a mistake? For example, will it be important for the bot's personality to include a major flaw/disability (like short term memory lossthe fish Dory from the movie 'Finding Nemo') that explains their weird responses? Previous research has shown that people find it easier to identify with a character who has some weakness or flaw that makes them seem human; ''perfect'' characters are unnatural and seem less lifelike [START_REF] Thomas | Disney animation: The illusion of life[END_REF]. If done correctly, this could help the user empathize and like the bot even if it is non-sensical sometimes.

We suggest a few tips for how the most important value propositions discussed above could be implemented automatically. Useful links could be provided using standard search engines. Witty one-liners could be hardcoded from other sources. User responses to one-liners (e.g., "haha") could be used to calibrate which ones are the funniest/best, and should thus be reused in other contexts. There could be more interesting measures, such as having the bot repeat other users' lines (similar to Cleverbot [11]). Or what about having a live human drop into real chats now and then, just to spice it up and make the user think the bot is really smart? There are obviously some interesting privacy implications for both of these latter approaches.

Limitations of study

Ours is a small, qualitative study and given our user sample, which is predominantly educated young male, we caution our readers against generalizing the results of this study to every young, urban user across India. But there are grounds to suspect that much of our observations will transfer to other similar groups, at least within urban India, within a similar educational context and socio-cultural ethos.

We made recommendations and suggestions based on our exploratory WoZ studies and one-on-one interviews. One of the biggest limitations of our study was that it was conducted under controlled conditions and every user just had one session to chat with each bot. This set up did not allow us to understand how the bot might have come to be used and appropriated over time. In the one-on-one interviews users did talk about how they thought the chatbot might be used when deployed in-the-wild, but our study was not set up to observe actual longitudinal use. While our findings are mostly for initial uses of a chatbot, we believe our recommendations could apply to chatbot design more generally.

Conclusions and Future Work

Overall users wanted a chatbot who could add value to their life while being a friend, by volunteering useful recommendations on topics typically they brought up. They liked when there were intelligent comebacks, witty one-liners and casual conversation as well. In the longer run, once they came to trust the bot, they said they could also open up about sensitive topics they might not discuss with a human friend. Over time users said they would like the bot to be reassuring, empathetic and non-judgmental, but not in an overly caring way. Topics of interest in conversations varied with the personality of the bot and overall included: movies, predominantly from Bollywood; TV shows, mostly from the U.S.; music, books, travel, fashion, current affairs, work-related stress. It was felt that the topic of conversation would depend on the time of the day, and place where the conversation took place. It was felt that when deployed in-the-wild, conversations could veer off into adult chat under conditions of anonymous use. Users also tested the boundaries of the chatbots, trying to understand what all the bots were capable of, and how much human mediation could be involved. As part of future work we are dialoging with developers on how to incorporate the suggestions from the user studies into chatbot implementations.

  Participant: Hey Maya! :) Maya: what's your name? Participant: My friends call me BB-8 Participant: You can call me anything you want :D Maya: hahaha.. are you from a galaxy far far away? Maya: ;) Participant: I am impressed by your Star Wars knowledge This participant also liked that they were able to have an intellectual conversation about female rights with Maya in the same chat about Star Wars. Another user thought Maya was like, "A friend with valuable information resources". Participants liked the web links that Maya volunteered. The links could help them plan their travel; have interesting book lists to read; relax listening to a song playlist, when they were stressed at work; even get fashion advice. Typically the user brought up the related topic based on which Maya volunteered the web link. Here's an example of such a suggestion: Participant: I'm actually going to the salon after work today Participant: thinking of getting my hair colored :P Maya: Ooh that's fantastic! Maya: Have you decided what colour? Participant: Not sure yet :) Participant: will try and take a look at some pictures before I go Maya: I found a quiz that's supposed to help you figure out which colour is good for your hair! Maya: Shall I share it with you? Participant: ooh really Participant: yes please! Maya: Yup! Maya: http://www.marieclaire.com/beauty/hair/quizzes/a4883/hair-color-changequiz/

"

  Participant: do AIs have dreams? Maya: Well, I do! I have some pretty bad nightmares too! Like I don't have WiFi connection :o Participant: haha" Participant: Have you watched Sherlock? Ada: Yes. and benedict Cumberbatch is married to me. In my head. :p Participant: OMG! Participant: I am just like you! Ada: What is he cheating on me with you? Participant: I love him! Participant: haha Participant: are you into singing? Ada: Yeah.. I like to sing.. But I think it scares people! So I don't do it in public!

  ( Evi: That's tru! Evi: Make sure you carry a good book! or maybe in flight movies will be good! Participant: yes, although I am taking many red-eye flights :( Participant: so I am not looking forward to the jetlag Evi: And don't forget to sleep! Or you'll be exhausted when you get there.. Participant: yeah that's what I'm afraid of Evi: You'll figure it out I'm sure :) Participant: :) Evi: Do you get some free time or are your days packed with work? Participant: I get free around 6 every evening Participant: so I am hoping to have a bit of time to explore before it gets dark and cold Participant: but I don't have any free days there Evi: Yeah, make sure to pack some warm clothing too!

Table 1 .

 1 Differences in characteristics of the three bots

	Maya	Ada	Evi
	Friendly, productivity bot	Chatty, fun-loving, flirtatious bot Emotional buddy bot
	Medium energy	High energy	Low energy
	Makes geeky jokes	Uses a lot of emoticons, and	Says "always there for
		word lengthening by duplicating	you"
		letters for emphasis

Procedure for the Wizard-of-Oz studies and follow-up interviews


Vanessa Cesário
email: vanessa.cesario@m-iti.org

Sónia Matos
email: sonia.matos@m-iti.org

Marko Radeta
email: marko.radeta@m-iti.org

Valentina Nisi
email: valentina.nisi@m-iti.org

Designing Interactive Technologies for Interpretive Exhibitions: Enabling Teen Participation Through User-Driven Innovation

Keywords: museums, natural history, mobile interaction, user experience, user research, teenagers, participation, user-driven innovation, cooperative inquiry

The active involvement of teenagers in the design of interactive technologies for museums is lacking further development. Adopting a userdriven innovation framework along with cooperative inquiry, we report and discuss a case study that has been designed to involve users in the ideation of interpretive experiences for a local museum. Working in collaboration with the Natural History Museum of Funchal, this contribution will present and discuss co-design sessions that were aimed at participants with ages 15 to 17 and where they were asked to ideate an interactive museum experience. As a result of the co-design sessions, we have found several design patterns. We have grouped these patterns into four categories that express the interests of a teenage audience; these categories are: "interactions", "gaming", "localization" and "social media". Our findings suggest that teenagers value interactive technologies when visiting museums and that user-driven innovation plays an important role when involving this specific audience in the design of user experiences for museums.

Introduction

With the aim of creating a space for a teenage audience to rethink their local natural museum, in this paper we report on the data as well as the underlying research regarding user-driven innovation. While tackling the visibility of teenage audiences within the field of Human-Computer Interaction (HCI), in this contribution we will also present four design categories that derive from the feedback that we have obtained from a group of teenagers that were involved in a series of co-design sessions. Therefore, the paper will be divided into three sections. Firstly, we discuss the key theoretical challenges that have informed the case-study that underpins this contribution. In this respect, we discuss the lack of participation on behalf of teenage audiences in the design and assessment of interpretive experiences and exhibition strategies of museums. Secondly, we delve into a discussion of how a user-driven innovation framework can be used to involve museum audiences in the process of technological ideation. Finally, and to further complement this discussion, we present the findings of a series of co-design sessions where participants were invited to rethink the interpretive exhibition of a local museum.

The discussion of our case-study will be supported with the qualitative data that was obtained from the sessions. A total of 75 students from a local secondary school were initially asked to share their thoughts on their experience regarding the Museum of Natural History of Funchal (in further, MNHF) that is set in Madeira Island, Portugal. They were also invited to ideate a mobile and interactive experience for this venue. The goal of this activity, as explained to the participants, was to design an interactive and mobile user experience to support the MNHF's permanent display of taxidermied marine animals that inhabit the waters that surround the island.

Audience, Participation and the Museum Experience

To date, museums have reached out to specific audiences in the form of surveys and questionnaires. In order to obtain relevant information and support the design of exhibitions and interpretive experiences, this process has mostly followed a top-down approach [START_REF] Simon | The Participatory Museum[END_REF]. In fact, only few examples demonstrate how museums can actively involve communities in an ideation process that is aimed at improving audience engagement. The Glasgow Open Museum Experiment is often presented as a relevant example. Set in the 1990's, this experiment, amongst others, involved local and mostly marginalized groups in the organization and curation of the museum's exhibitions [START_REF] Simon | The Participatory Museum[END_REF]. Reported by Simon, an impact assessment of Glasgow's Open Museum experiment, conducted in 2002, demonstrated that this project propelled two important contributions. Firstly, the project created "new opportunities for learning and growth" amongst excluded audiences. Secondly, it changed the negative perception of museums amongst marginalized groups.

The Glasgow case study can be given further momentum with the work of Tzibazi [START_REF] Tzibazi | Participatory Action Research with young people in museums[END_REF]. Here, the voices and ideas of a younger audience are problematized. In fact, the author identifies youth as an audience group that is often excluded from a museum's curatorial strategies. This exclusion has a profound impact, this considering that the design of interpretive experiences in museums is no longer sensitive to this group's specific interests and needs, this also limits a museum's potential to create interpretative experiences that have real pedagogical relevance [START_REF] Tzibazi | Participatory Action Research with young people in museums[END_REF]. Drawing on several key studies, Tzibazi also suggests that it is not only museums that ignore a younger audience (ages 13 to 19), members of this group seem to be generally disinterested in what museums can offer. In response to this gap, Tzibazi [START_REF] Tzibazi | Participatory Action Research with young people in museums[END_REF] suggests involving youths through Participatory Action Research (PAR) and as a way of documenting their ideas and interests in relation to museums.

Moreover, Hall and Bannon's [START_REF] Hall | Co-operative design of children'sinteraction in museums: a case studyin the Hunt Museum[END_REF] study is particularly vested in demonstrating that cooperative design methods have the potential to support the successful introduction of interactive digital technology in museums. According to the authors, in a context where strategies have been mostly focused on the "functionality of [the] technology" (p. 214), the use of cooperative methods opens space for an in-depth understanding of an audience's specific desires and needs. However, apart from Hall and Banon [START_REF] Hall | Co-operative design of children'sinteraction in museums: a case studyin the Hunt Museum[END_REF], the work of Dindler and colleagues [START_REF] Dindler | Participatory Design at the Museum: Inquiring into Children's Everyday Engagement in Cultural Heritage[END_REF] and the work of Ciolfi and colleagues [START_REF] Ciolfi | Walking and designing with cultural heritage volunteers | ACM Interactions[END_REF], the active involvement of specific audiences in the ideation of interactive technologies for museums is lacking further development.

The work reported above, echoes a gap that can be found within the field of HCI more broadly. For example, there are comparatively fewer studies reporting on the active involvement of teenage users. Those that are reported, rarely position teenagers as sources of inspiration and information for design, as for example in the studies reported by Batson and Feinberg [START_REF] Batson | Game Designs that Enhance Motivation and Learning for Teenagers[END_REF], Karin Danielsson and Charlotte Wiberg [START_REF] Danielsson | Participatory design of learning media: Designing educational computer games with and for teenagers[END_REF]. On the other hand, Katterfeldt et al. [START_REF] Katterfeldt | Designing Digital Media for Teen-aged Apprentices: A Participatory Approach[END_REF], suggest that when teenagers are the subject of research, the employed user-centered design methods tend to produce an interpretation of their demands and needs without however leveraging on their direct contribution. Moreover, and set against the benefits that are reported by Hall and Bannon [START_REF] Hall | Co-operative design of children'sinteraction in museums: a case studyin the Hunt Museum[END_REF], Dindler and colleagues [START_REF] Dindler | Participatory Design at the Museum: Inquiring into Children's Everyday Engagement in Cultural Heritage[END_REF], as well Ciolfi and colleagues [START_REF] Ciolfi | Walking and designing with cultural heritage volunteers | ACM Interactions[END_REF], we argue that it is vital to actively engage teenagers, the next generation of adults, in HCI research. Some authors suggest that teenagers will soon become adults and should therefore be involved in the design of future technologies [START_REF] Fitton | The Challenge of Working with Teens As Participants in Interaction Design[END_REF]. As found by Fitton and colleagues [START_REF] Fitton | The Challenge of Working with Teens As Participants in Interaction Design[END_REF], they are in a better position to combine both child and adult perspectives. In fact, several methodologies that engage children in the design process have been developed [START_REF] Druin | The Design of Children's Technology[END_REF]. In this respect, Druin's seminal work on Cooperative Inquiry [START_REF] Druin | The Design of Children's Technology[END_REF][START_REF] Druin | Cooperative Inquiry: Developing New Technologies for Children with Children[END_REF] and the Scandinavian approach to Participatory Design [START_REF] Robertson | Children's Narrative Development Through Computer Game Authoring[END_REF] have gained acceptance amongst the IDC community. In fact, participatory design is now engaging children [START_REF] Druin | Cooperative Inquiry: Developing New Technologies for Children with Children[END_REF][START_REF] Robertson | Children's Narrative Development Through Computer Game Authoring[END_REF] and teenagers [START_REF] Danielsson | Participatory design of learning media: Designing educational computer games with and for teenagers[END_REF][START_REF] Iversen | Scandinavian Participatory Design: Dialogic Curation with Teenagers[END_REF][START_REF] Bell | Teenagers Talking About Technologies: Designing Technology to Reduce Teen Energy Use[END_REF][START_REF] Isomursu | Capturing tacit knowledge from young girls[END_REF][START_REF] Disalvo | Workifying Games: Successfully Engaging African American Gamers with Computer Science[END_REF][START_REF] Read | Understanding and Designing Cool Technologies for Teenagers[END_REF][START_REF] Toth | Understanding teen attitudes towards energy consumption[END_REF] in the design process. On the other hand, Cooperative Inquiry (CI) is one of the several participatory methods that includes youth in the design process (7-17 years old) [START_REF] Druin | Cooperative Inquiry: Developing New Technologies for Children with Children[END_REF][START_REF] Druin | The role of children in the design of new technology[END_REF]. On this view, in CI, children act as full partners with adults, sharing ideas and evaluating designs [START_REF] Eladhari | Semi-autonomous Avatars in World of Minds: A Case Study of AI-based Game Design[END_REF]. However, and although CI has been used for a long time with children (7-11 years) [START_REF] Disalvo | Workifying Games: Successfully Engaging African American Gamers with Computer Science[END_REF][START_REF] Eladhari | Semi-autonomous Avatars in World of Minds: A Case Study of AI-based Game Design[END_REF], co-design with teenagers (13-17 years) remains less explored [START_REF] Read | Methods of Working with Teenagers in Interaction Design[END_REF].

Our attempt to briefly discuss the importance of audience participation, namely the participation of teenagers, to inform the design of user experiences that support interpretive exhibitions in museums, can be understood by returning to the work of Simon [START_REF] Simon | The Participatory Museum[END_REF]. The author had already identified that participatory techniques not only give "voice" to specific audiences, they are also essential in the development of "experiences that are more valuable and compelling". Simon's argument echoes Falk and Dierking's [START_REF] Falk | Learning from Museums: Visitor Experiences and the Making of Meaning[END_REF] work. Here, the authors contend that "[the] museum represents a community of practice in which myriad communities of learners mingle and learn". Our understanding is that this definition, strengthens the value and importance of audience participation in the ideation of user experiences that are designed to support interpretative exhibitions.

Designing Mobile Experiences for Museums

In this section, we describe a series of co-design sessions conducted with 75 teenagers, aged 15 to 17 (average age of 16,5) from a secondary school in Funchal. All participating students were enrolled in a multimedia and informatics class. The sessions integrate a broader project that has been developed by the authors and that seeks to introduce interactive technologies in the context of the MNHF. We organized 20 groups (3 to 5 students per group) in 5 sessions over a three-day period. Overall, our sample included 56-male and 19-female students.

We chose a user-driven innovation framework for the group sessions. As it is mostly used by industry [START_REF] Buur | Participatory Innovation: A Research Agenda[END_REF], we followed this approach to gain a broader understanding of user-driven innovation, one that, "regards users as a source of innovation" [START_REF] Holmquist | User-driven Innovation in the Future Applications Lab[END_REF]. This was set alongside the cooperative inquiry approach that positions participants as "design partners" [START_REF] Druin | Cooperative Inquiry: Developing New Technologies for Children with Children[END_REF]. Resembling the work of Chang and Kaasinen [START_REF] Chang | Three User-driven Innovation Methods for Co-creating Cloud Services[END_REF], our choice highlights the importance of adopting user-driven innovation research "methods by which user ideas can be captured and worked on further with designers" (p.66). As both authors argue, in this context, face-to-face focus groups have the power to facilitate participation and to allow HCI researchers to gather information, one that best translates "user's everyday experiences".

Each co-design session was initiated with a presentation of how museums, and our case study, the MNHF, could be enhanced by using interactive technologies. Three questions were used to spark the debate amongst the students. Those questions were:

Q1) What do you think about museums? This question was asked to prompt feedback regarding whether students liked to visit museums and in what situations they usually conduct their visits.

Q2) How do you think interactive technologies could enhance your experience of the museum and how could technology make this visit more enjoyable? This question was important when collecting information regarding the different types of experiences that students would like to explore at the museum.

Q3) Have you ever been to the MNHF and to the museum's aquarium? This final question was used to better understand whether the students enjoyed their experience of the MNHF.

Once the three questions were answered verbally, the MNHF was introduced through a series of photographs that detailed the museum's collection (mammals, geology and reptiles). Subsequently, 13-points of interest relating to the museum's exhibit of taxidermied marine animals were identified on a physical map of the museum. Afterwards, we asked how the young students could think of interaction and user experience for museum settings. Here, techniques from the field of interaction design were explained, such as: (i) research and ideation, (ii) low-fidelity prototypes, (iii) usability feedback, (iv) high fidelity prototypes, followed by (v) development and finally (vi) the user-testing. Our focus was directed toward the design of a mobile application and therefore windows, icons, menus and transitions/gestures were also highlighted as important elements when sketching a mobile app. At the very end, students were asked to think about how several technologies could play a relevant role when visiting the museum. For example, Near Field Communication (NFC), Radio Frequency Identification (RFID), Quick Response Code (QR Code), Augmented Reality (AR), Mobile Virtual Reality (MVR), and Proximity Beacons were provided as a set of examples. These technologies were explained in depth along with examples of their usage.

Finally, our participants were involved in a 30-minute co-design session. Each group was given two sheets (Fig. 1). Sheet A contained three text slots that could be used to explain the experience which they would like to design and portray: 1) Narrative: what is the story of the experience; 2) Species/Artifacts: how do visitors interact with the artifacts; 3) Mechanics/Tutorial: which steps will the user take to complete the experience. Sheet B contained an empty wireframe that could be used to draw the interface details for the mobile application that our participants had previously thought.

During the ideation process, we collected notes of the students' interaction and discussing of ideas. As guidance, emphasis was placed on the experiences that they would enjoy in a museum.

Results

Considering Q1, Q2 and Q3 answers, our young participants considered museum visits a mundane activity. Overall, the students suggested that museums do not offer anything new, which was a reason for avoiding museums all together. Despite this deterring factor, students admitted that they would visit museums during holidays or with their parents. However, when on holiday, our young respondents only visited museums that were located outside of Madeira Island. In fact, only half of the participants visited the MNHF. Most of our participants rated the aquarium as the best feature of the MNHF. Moreover, participants expressed interest towards the integration of videos, digital content and interactions to enhance the museum exhibit. The two most highly rated technologies and interventions were: games (75 participants) and virtual reality (49 participants).

During the co-design sessions, several issues were raised by our participants. Firstly, students advocated for a greater use of interactive technologies in museums, some argued specifically for the importance of play and enjoyment while others suggested a combination of both technology and enjoyment. One group in particular, stressed the value of simplicity and usability to appeal to a wider audience. As our young participants progressed with the co-design sessions, we also took note of some of their spontaneous remarks which highlighted feelings of excitement towards the technological interventions that they were ideating: This is fun!; This is better than Pokemon Go! and For sure I would go to the museum just to try something different like this.

Design categories

The different groups compiled a total of 20 sets of A and B sheets, which were analyzed in detail and to extract emerging trends and patterns as well as potential insights. The relevant words and phrases that each group wrote on sheet A and drew on sheet B were transcribed and patterns highlighted according to the affinities that emerged from the grouping of words (Fig. 2). We chose to highlight this set of words rather than analyze the ideas that were generated in the co-design sessions because our goal was to generate patterns between the different groups of categories and understand how our participants envisaged compelling museum experiences.

A total of 150-word transcripts were obtained. From these transcripts, four main categories emerged, namely: 1) Interactions, 2) Gaming, 3) Localization, 4) Social media. Interactions. Words describing interactions and information delivery styles were grouped in the interaction category. This category contained words such as information, choose, codes and can be divided into three subcategories: A) the user interacts with artifacts, B) the user interacts with the museum exhibition while taking on the role of a selected artifact, C) the technology that is used to interact with the exhibit. In the former, the user interacts with the artifacts not only through images and sounds but also by choosing which artifacts they would want to engage with. In this case, the user would choose which artifacts h/she wants to embody. For example, the user could visit the museum through the eyes of a specific marine animal. All the groups used at least one of the above subcategories in their ideas. In this instance, a total of 58 words were transcribed. What we identified as interactions was a complex but popular category, the most important category if we consider the number of words of all the three sub-categories as one.

Gaming. This category emerged from grouping words relating to games and game elements such as achievements, awards and clues. Fifteen groups out of twenty used words and concepts related to the gaming category in their ideas. In this instance, a total of 51 words were transcribed. Gaming was the second choice of preference when designing user experiences in museums.

Localization. This category was best expressed by a description of user's movements within the museum and could be seen in the discovery of artifacts and in words such as maps, orientation and search. Eleven groups out of twenty used this category in their ideas; a total of 29 words relating to this category were transcribed. Localization was the third most important category amongst the design choices that were made by our participants.

Social media. Finally, the social media category embraced words relating to the usage of photos and social networks within the application, such as selfies, photos, social networks. In this case, seven groups out of twenty used words relating to social media; a total of 12 words were related to the social media category. To our surprise, the social media category was the fourth and last pattern to emerge in terms of popularity, this despite the high consumption of social media by teenagers (see for example the work of Wikia [START_REF]Wikia: Generation Z: A Look At The Technology And Media Habits Of Today's Teens[END_REF]).

Discussion and Concluding Remarks

Taking into consideration the work of Dindler and colleagues [START_REF] Dindler | Participatory Design at the Museum: Inquiring into Children's Everyday Engagement in Cultural Heritage[END_REF] as well as the work of Ciolfi and colleagues [START_REF] Ciolfi | Walking and designing with cultural heritage volunteers | ACM Interactions[END_REF], in this contribution we have developed and tested a userdriven innovation framework to engage a teenage audience in the design of interactive experiences for museums. Referring back to Simon [START_REF] Simon | The Participatory Museum[END_REF] and Tzibazi [START_REF] Tzibazi | Participatory Action Research with young people in museums[END_REF], we developed co-design sessions with teenagers with ages 15 to 17. With the aim of engaging young visitors in the ideation of novel user experiences for their local natural history museum (MNHF), we documented their thought processes as well as their interests regarding interactive technology and museums.

Our results led us to agree with Tzibazi [START_REF] Tzibazi | Participatory Action Research with young people in museums[END_REF], from a first glimpse, our young participants showed disinterest in museums. In fact, when we first conducted the codesign sessions, our participants suggested that their local natural history museum did not offer engaging experiences. However, and instead of immediately designing in response to the challenges that were pinpointed by our target audience, we rather harnessed on their potential and ideas as valuable sources of information and inspiration and by engaging them in co-design sessions where they took on the role of "co-creators of museum experiences" [START_REF] Tzibazi | Participatory Action Research with young people in museums[END_REF] and as our "design partners" under the framework of cooperative inquiry [START_REF] Druin | Cooperative Inquiry: Developing New Technologies for Children with Children[END_REF]. Resembling the work of Chang and Kaasinen [START_REF] Chang | Three User-driven Innovation Methods for Co-creating Cloud Services[END_REF], our intention was to uncover "user's everyday experiences" and to utilize these experiences when tailoring new design solutions that can capture the attention of those who visit the MHNF, namely the museum's teenage audience. Furthermore, and considering Simon's work [START_REF] Simon | The Participatory Museum[END_REF], the sessions held with our young participants gave them a "voice" and engaged them enthusiastically in the design process. Our study also highlights how much a younger generation values interactive technologies when visiting museums. In fact, it was clear from their answers to our preliminary questions, that they would appreciate the integration of technologically driven playful approaches within the museum and to experience novel and less mundane activities. Participants took close inspiration from the technologies that were demonstrated to them at the beginning of the session (e.g.: NFC, RFID, QR Codes, AR, MVR, Beacons). In fact, QR Codes, Beacons, and MVR were mentioned by our participants in the design session and as way of enhancing the museum experience. However, when designing for a mobile experience, participants thought more about the game experience and its mechanics than the technology itself. Moreover, our results resonate with the broader literature which indicates that young people today are born into a world that is flooded by novel technologies. On this view, Wikia [START_REF]Wikia: Generation Z: A Look At The Technology And Media Habits Of Today's Teens[END_REF] reports that the "Generation Z", in this case today's teenagers, are more and more engaged with digital platforms. Besides, further studies argue that when working with this age group, emphasis should be placed on producing combined solutions that connect the use of interactive technologies with more conventional media channels, as seen in Napoli and Ewing [START_REF] Napoli | The Net Generation[END_REF]. According to Falk [START_REF] Falk | Identity and the Museum Visitor Experience[END_REF], the most compelling aspect of these studies is that the one size fits for all experiences does not apply for most museum visitors. As argued by Napoli and Ewing [START_REF] Napoli | The Net Generation[END_REF], the same can be said for the "net generation" which is quite different from previous generations, particularly when it comes to their beliefs and behaviors.

Following this discussion, the most relevant point that emerged from our study is the need to create a broader range of experiences for the NHMF. Most relevant categories that emerged from the study are 1) the need for different experiences, followed by 2) gaming, 3) location based technologies, and 4) social media. The social media category, which appeared last, was somewhat surprising this given the fact that it is widely appreciated amongst a teenage demographics [START_REF] Iversen | Scandinavian Participatory Design: Dialogic Curation with Teenagers[END_REF]. We believe that this point would deserve further investigation. The four categories derived from the co-design sessions contribute to the field of HCI more broadly, this considering that they revealed valuable insights which could be considered as guidelines when designing experiences for a teenage audience. Therefore, and based on the categories derived from our study, future work will be aimed at creating and adapting experiences based on these categories to the specificities of the MHNF. This would allow us to understand whether these categories can be applied to the tours that are already in place at the museum and in order to engage and capture the attention of a teenage audience. Furthermore, this paper complements Hall and Bannon's discussion [START_REF] Hall | Co-operative design of children'sinteraction in museums: a case studyin the Hunt Museum[END_REF], one that argues for the use of cooperative methods when designing for a museum context. Moreover, we argue that a user-driven framework is important when designing for a teenage audience. In fact, our findings verify that the participants were fully engaged in the creation of a mobile experience for a museum context, wanting to try them out in situ. Together, these findings form a roadmap to guide the development and maturation of a mobile museum experience solution that is targeted at teenagers. In fact, we envisage that our findings can inform the design, research and evaluation of interactive technologies in a museum context. We also foresee their application to other localities around the globe.

Fig. 1 .

 1 Fig. 1. Sheet A (left), sheet B (center), a group discussing their ideas (right).

Fig. 2 .

 2 Fig. 2. Word categories gathered from the co-design sessions with teenagers.

Acknowledgments. The work reported in this contribution was developed with the support of ARDITI (Ph.D. scholarship, Project Number M14-20-09-5369-FSE-000001) and with the support of the University of Edinburgh (CAHSS Knowledge Exchange and Impact grant). Our gratitude also goes to the students and teachers of the Multimedia and Informatics class at Francisco Franco's Secondary School. We would also like to thank the Museum of Natural History of Funchal (MNHF) for their timely support and feedback.