Max Willis
email: max.willis@unitn.it

Antonella De Angeli
email: adeangeli@lincoln.ac.uk

Massimo Zancanaro
email: zancana@fbk.eu

Experience Probes: Immersion and Reflection Between Reality and Virtuality

Keywords: HCI Evaluation Methods, Immersive Experiences, Momentary Assessment, Mixed Reality Environments

This research addresses the issue of the memory-experience gap, the disconnect between momentary perceptions and post experience reporting as relates to HCI research methodologies and the study of immersive technology-mediated experiences in particular. The paper presents an overview of contemporary understanding of immersion and examines HCI methods that investigate participant experiences. We introduce Experience Probes, an integrated design and evaluation methodology that affords momentary reporting by blending states of reflection and immersion in a structured activity situated within the immersive experience. A pilot study is presented that examines an immersive soundscape installation and an Experience Probe enacted through participantauthored sound maps. The maps provide data for thematic analysis, and are coded for signs of self-perception and a sense of place to evaluate participants' sensations of presence and immersion. Preliminary results are discussed in relation to the reality-virtuality continuum and suggest that the reflective act of reporting, and the experience of immersion within the soundscape installation are not mutually exclusive. This research seeks to extend HCI methods by overcoming the memory-experience gap in the evaluation of technology-mediated experiences. .

Introduction

Immersion and reflection are often considered as opposing [START_REF] Carlson | Performance: a critical introduction[END_REF] or alternating states [START_REF] Csikszentmihalyi | Beyond boredom and anxiety[END_REF] of experience, and while these aspects of perception in lived experiences have been investigated long before the emergence of digital media, the development of augmented-and virtual reality technologies has revitalized investigations into the interrelated concept of presence [START_REF] Riva | Extending the Self through the Tools and the Others: a general framework for presence and social presence in mediated interactions[END_REF]. In a technology-mediated environment, there is often no escaping the existence of devices, equipment and interfaces, the technological materiality that drives and delivers digital content. Attention to these aspects of the experience can remind participants that they are engaging with a generated or semisynthetic environment and hamper the sense of immersion. Likewise, in the evaluation of experiences that elicit immersion and flow, these states can potentially break when a participant is interrupted to answer questions or reflect. Study of these experiences relies heavily on post-experience questionnaires and qualitative debriefings. However, research in cognitive science has shown that post-experience reporting can be biased by a number of factors, leading to a condition known as the experiencememory gap [START_REF] Miron-Shatz | Memories of yesterday's emotions: Does the valence of experience affect the memory-experience gap?[END_REF]. In this paper we propose a new method of Experience Probes (EP) that integrates design, experience and evaluation, engaging participants in the virtual world through reflective, investigative activity. EP is demonstrated with a pilot study in an immersive soundscape installation to demonstrate the potential for examining participants' perceptions from within an immersive experience through the analysis of participant-authored mapping artifacts.

One key observation of this work is that visitors to the soundscape installation, when engaged in listening and sound mapping exercises (reflecting), overwhelmingly describe ambience and experience of the virtual place rather than that of the installation, often expressing a feeling of 'being there' (immersion). Another important finding from these preliminary results is that the sense of presence is proportionally represented in participant reporting even when specific attention is given to the technological materiality of the experience. Results are examined with reference to the realityvirtuality continuum [START_REF] Paul Milgram | A Taxonomy of Mixed Reality Visual Displays[END_REF] and portray participants' momentary experiences, untainted by the experience-memory gap. This work contributes to the investigation of technology-mediated environments with a method to bind the experience of immersion in a technology-mediated environment with the activities of evaluation.

Related Work

Definitions of immersion range from "the experience of being transported to an elaborately simulated place" [START_REF] Murray | Hamlet on the Holodeck: The Future of Narrative in Cyberspace[END_REF] to "unreflective absorption in an activity" [START_REF] Kwastek | Aesthetics of interaction in digital art[END_REF]. These two definitions mirror the shift in focus in two decades of HCI research, from usability to experience [START_REF] Bødker | Third-wave HCI, 10 years later---participation and sharing[END_REF][START_REF] Waterworth | Dealing with user experience and affective evaluation in hci design: A repertory grid approach[END_REF]. Presence, initially examined as the perception of being 'in' a place, achievable by effectively masking the medium of experience [START_REF] Riva | Extending the Self through the Tools and the Others: a general framework for presence and social presence in mediated interactions[END_REF] has similarly evolved in HCI literature to address perceptions of agency within that place, including social, cultural and material dimensions [START_REF] Spagnolli | Understanding the Human Involvement in Mediated Interactive Environments[END_REF]. Contemporary study of technologymediated environments questions what exactly constitutes immersion and experience and how to measure them [START_REF] Al-Shamaileh | Investigating a multi-faceted view of user experience[END_REF][START_REF] Bilda | Collaborative analysis framework for evaluating interactive art experience[END_REF][START_REF] Forlizzi | Understanding experience in interactive systems[END_REF][START_REF] Höök | Sense and sensibility: evaluation and interactive art[END_REF][START_REF] Väänänen-Vainio-Mattila | Towards Practical User Experience Evaluation Methods[END_REF][START_REF] Peters | An exploration of user engagement in HCI[END_REF][START_REF] Vermeeren | User experience evaluation methods: current state and development needs[END_REF]. Kristina Höök addresses the challenges in [START_REF] Höök | Sense and sensibility: evaluation and interactive art[END_REF] decrying the mismatch between traditional HCI methods that are largely concerned with usability and the concept and aims of evaluation in interactive art. Her recognition that traditional HCI methods need expanding to encompass the lived experiences of technology are reinforced in [START_REF] Kaye | Evaluating experience-focused HCI[END_REF][START_REF] Macdonald | Changing perspectives on evaluation in HCI: past, present, and future[END_REF], specifically concerning interactive art installations and mixed reality environments in [START_REF] Georgiou | The development and validation of the ARI questionnaire: An instrument for measuring immersion in location-based augmented reality settings[END_REF] experience design in [START_REF] Zeljko Obrenovic | Sketching interactive systems with sketchify[END_REF] and prototyping interactions in [START_REF] Buchenau | Experience prototyping[END_REF].

Immersion in any form, be it narrative, theatrical or technology-driven is related to the 'willing suspension of disbelief', the idea that an observer consciously disregards the fact that a situation is beyond the ordinary reality in order to follow or participate, accepting the logic of the experience regardless that it may deviate from that of the real world [START_REF] Mateas | A neo-Aristotelian theory of interactive drama[END_REF]. This is essential to understanding technology-mediated experiences as virtual worlds, games and interactive installation involve hybrid situations that fluctuate along the reality-virtuality continuum [START_REF] Paul Milgram | A Taxonomy of Mixed Reality Visual Displays[END_REF]. The real and the virtual in this model are located at the extremes and the space between regarded as mixed reality. Augmented reality, the overlay of virtual elements on the real world is positioned towards the real environment. Augmented virtuality, the implementation of objects and elements from the real world within the virtual, is located towards virtuality..

Considering immersion as a complete captivation [START_REF] Murray | Hamlet on the Holodeck: The Future of Narrative in Cyberspace[END_REF], reflective thought and contemplation of the immersive experience might seem to inhabit the opposite end of the continuum [START_REF] Hansen | Contemplative interaction: alternating between immersion and reflection[END_REF]. This is congruent with Schon's perspectives of reflection-in-action, the conscious renegotiation of knowledge and practice within an activity, and reflection-on-action, the post-experience review of knowledge gained through an activity [START_REF] Schön | The reflective practitioner : how professionals think in action[END_REF]. However there remains considerable debate over design and evaluation of immersive experiences. Numerous studies [START_REF] Bilda | Collaborative analysis framework for evaluating interactive art experience[END_REF][START_REF] Candy | Interactive Experience, Art and Evaluation, in Interactive Experience in the Digital Age: Evaluating New Art Practice[END_REF][START_REF] Edmonds | Human Computer Interaction, Art and Experience, in Interactive Experience in the Digital Age: Evaluating New Art Practice[END_REF][START_REF] Jacucci | ParticipArt: Exploring participation in interactive art installations[END_REF][START_REF] Waterworth | Dealing with user experience and affective evaluation in hci design: A repertory grid approach[END_REF] examine participation and collaboration in interactive art installations attempting to quantify emotional reactions and social engagements with techniques such as Positive and Negative Affect Schedule (PANAS), Collaborative Analysis Framework and the Repertory Grid. Other methods employ video-cued recall [START_REF] Bilda | Collaborative analysis framework for evaluating interactive art experience[END_REF] to supplement questionnaires and multimodal investigations such as Gaver's design probes, cultural commentators [START_REF] Gaver | Cultural probes and the value of uncertainty[END_REF] and polyphonic assessment [START_REF] Gaver | Cultural commentators: Non-native interpretations as resources for polyphonic assessment[END_REF] draw on various lines of qualitative analysis taking into account the subjective interpretations of participants. The Sensual Evaluation Instrument (SEI) [START_REF] Isbister | The sensual evaluation instrument: developing an affective evaluation tool[END_REF] introduces nonverbal assessments of emotions and the affective properties of computational artifacts, another unique approach to addressing the methodological deficit. However the majority of HCI research into experience evaluation relies on post-experience reporting which fails to access the thoughts, feelings and emotions of participants from within the lived experience.

The importance of extracting participants' perspectives momentary impressions is related to the potential, and sometimes marked dissonance between the emotions, sensations and the perceptions of the moment and those recalled afterward [START_REF] Kahneman | Living, and thinking about it: Two perspectives on life[END_REF]. This dissonance is referred to as the memory-experience gap [START_REF] Miron-Shatz | Memories of yesterday's emotions: Does the valence of experience affect the memory-experience gap?[END_REF] and it has driven the development of methods to address the immediate experience in cognitive science research [START_REF] Csikszentmihalyi | Flow: The Psychology of Optimal Experience[END_REF] and behavioral medicine [START_REF] Smyth | Ecological Momentary Assessment Research in Behavioral medicine[END_REF]. The discrepancy between momentary and retrospective data in reporting is attributed to a variety of experiential, environmental, memory and cognitive factors [START_REF] Miron-Shatz | Memories of yesterday's emotions: Does the valence of experience affect the memory-experience gap?[END_REF]. An excellent introduction to the topic in the framework of HCI and user experience evaluation can be found in [START_REF] Bruun | Mind the Gap! Comparing Retrospective and Concurrent Ratings of Emotion in User Experience Evaluation[END_REF].

3

The Technology-Mediated Environment: Soundscape

This pilot study of Experience Probes examines the immersive soundscape installation Listening to the Walkable City that was constructed in an underground car park as part of the COOP2016 conference in Trento, Italy (Fig. 1). Its material components were a circular array of six audio monitors on stands, a north arrow marked on the floor in the center and a bamboo street-sweeping broom. Several chairs and tables were located to the sides supporting computer, audio equipment, and a coffee machine. The car park was accessible by stairs leading down from a small courtyard at the conference venue. The real environments portrayed in the soundscape were pedestrian stairs in Hong Kong that ascend relatively steep inclines in the city and are capped at each end by auto traffic. These are public spaces interspersed with small parks, public toilets and lined with trees, local small businesses and outdoor restaurants. The social and cultural activity of the stairs are the focus of the research project "Hong Kong Stair Archive: Documenting the Walkable City" [START_REF] Christ | [END_REF] that grounded initial investigations into the stairs soundscapes. The soundscape was composed of street recordings made on the stairs following several listening surveys of the area. Recordings were made mornings, afternoons and evenings on three separate stairs, and the final soundscape installation condensed a nearly 300m transect of the city into an exhibition circle approximately eight meters in diameter. The soundscape installation loosely reflected the real world situation in Hong Kong, with an outdoor restaurant located at the bottom of the stairs on a small street with delivery trucks and pedestrian traffic. The stairs led 'up' through the installation area, with sounds of local businesses on each side, and footsteps tripping up and down slightly off-center. At the southern, top of the staircase could be heard the sounds of a heavily trafficked road, with further in the distance a jackhammer and school playground. The street sweeper could be heard in two locations on the stairs.

Sound recordings on site were made with a handheld audio recorder with background sound s of traffic, construction, birdsong, rain and the outdoor restaurants captured by a wide audio image. Isolated sounds were recorded using a very narrow audio image and included footsteps, the rattling of delivery carts and the street sweeper. Audio clips were filtered and assigned unique positions and animated trajectories within the soundscape installation to mimic the natural dynamics of the stairs' sonic environment. Additional sounds were contributed by the installation space, such as the operation of the coffee machine, voices of participants and researcher, the occa-sional opening of the car-park garage doors, and the action of a broom sweeping the concrete floor.

Experience Probe

A total of 34 participants undertook the experience probe. Demographic data was not collected, however participants were roughly balanced male and female, between 30 and 60 years of age, mostly self-selected from the conference attendees. The conference was not sound specific, though many of the participants had prior experience with qualitative data collection methods. Once in the installation space, participants were allowed a few minutes to explore before being offered a clipboard and pen and invited to take part in the evaluation. The probe activity was orientated on a printed form with basic instructions and a map circle with a 'North' arrow. This arrow was replicated on the floor of the installation and orientates between the virtual soundscape, the car park installation space and the evaluation map (Fig. 2).

Participants were asked to "Describe (in any language) some of the sounds that you hear. Number them 1,2,3… and mark their position on the map." Additionally they were asked to "Describe the ambience that this installation presents." and after, to "Describe your experience, impressions, or comment on this installation." Resulting evaluations were written in English, Italian, German, and one in Danish. Participation was largely individual with little discussion or collaboration during the mapping exercise. Several participants sat in one of the chairs to write in more detail, though most remained standing. Instructions were left deliberately simple, and participants took various approaches to completing the exercise. The probe produced 34 participant-authored maps that were translated into English by native speakers. Transcripts were thematically analyzed using AtlasTi, first grouping descriptions of sound sources together, such as "broom" and "sweeping", "auto", "car" and "traffic" and then examining for inscriptions of senses of self and place. Self-perception was identified for example in "I feel", "feeling" and "I am", references to memories such as "in some ways it remembers a bit Beijing or Hong Kong" and creative or imaginative declarations, as in "These sounds are really inspiring for writing poetry!" and "…crimescene". Sense of place was noted in comments such as "It seems to be in a shop overlooking the street in a pedestrian district..." and "People passing by or exiting the workshop; happy because their work time has finished." Frequencies were determined not by word occurrences, but by the number of participants reporting in each category, as many participants repeated words and phrases. Participants' maps were further sorted by the method used to map, for example numbering of sounds or speakers, showing motion, and text entered direct on the map.

Results and Discussion

The descriptions provided by participants reflected a true-to-life perception of the actual place in Hong Kong, highlighting the capacity of soundscape to accurately transmit ambience. The most frequently recognized sound within the installation was the street sweeper, reported by 31 of 34 participants (91%). This was followed by sounds of traffic (82%), water and rain (70%), nature sounds (67%), voices (61%), footsteps (52%) and noise (52%). Birdsong, construction and car horns were among the other frequently reported sounds. Participants' ability to identify the elements of the soundscape, however, was to be expected, as the installation was a straightforward arrangement of familiar urban sounds in an enclosed listening environment.

Our main interest is in participants' reporting of these sounds in relation to their awareness of self and presence in the environment. Self-perception (41%) was identified in statements such as "If I close my eyes I have the impression of find myself elsewhere" and "Impression that around you is an oriental market in action". Sense of place, or first-hand experience of the ambience of the soundscape (38%) was marked in texts such as "It seems to be in a shop overlooking the street in a pedestrian district [...]" and more imaginatively, "A gutter, a water loss slips unseen in a silent narrow street, on the margin of the big and noise metropolitan arteries […]" Nearly a third (29%) of participants expressed both self and place, clear signs of presence in the virtual that signal immersion. Several participants addressed the overlap of real and virtual presence directly, for example "During the walk eyes and site are not important: I was looking at the ceiling the man next to me kept eyes closed. It is strange the difference between actual location (car parking) and the world presented by sounds." and "I felt I was walking a bit in circles; maybe moving back-and-forth; am I looking for an address?; The sounds move from one side to another; but I felt more like it was me that was moving."

Examining how the participants enacted their mapping, we observe another important overlap between reality and virtuality on the continuum. Of the 34 participants, one group of 18 (52%) numbered their sounds and marked them on the map, while a second group of 10 participants numbered the six audio monitors on the map and described the sounds they perceived were emanating from each source. Many of this second group of participants stood in front of each audio monitor and listened intently, clearly orientating on the material element of the installation, towards the reality end of the continuum. Yet reporting of self-perception and sense of place in the virtual remained consistent across both groups, shown in Table 1 The percentage of participants simultaneously reporting reflection and immersion remained constant, regardless if individuals were consciously attending to the installation environment or were focused on the virtual world of the soundscape. This supports the idea that a hybrid state of real-virtual, reflective-immersive experience is being initiated through the probe. That 38% of participants described movement within the soundscape, and 26% depicted motion on the map further suggests that participants were cognizant of their own position, as listeners, in relation to both the real and the virtual environments.

This preliminary investigation of the Experience Probes (EP) technique demonstrates the potential to engage participants in reflective practice and gather data from directly within the immersive experience. The probe's participant-authored maps provide sufficient detail to establish that nearly one third of participants experienced self reflection and sense of place, with some participants reporting from a state that clearly fluctuates along the reality-virtuality continuum. The proposition that one can reflect while immersed is borne out even among participants acting with acute awareness to the technological mediation of the installation. Though the use of pen and paper mapping worked well for the small-scale investigation of soundscape installation, it may not be suitable for examining all immersive experiences, particularly those with active visuals or tangible interactions. Yet the EP method is applicable to other domains with some adjustment to the form of data collection. Where a handheld device is appropriate for engaging participants, EP can be enacted through a digital interface to facilitate data collection, storage and filtered retrieval for analysis and visualization. If collaboration or cooperation is fundamental to a particular experience, EP can be implemented as a game or playful engagement. In any form, central to implementing a successful probe is to ensure that the reflective practice that produces data is mediating and not disturbing the immersive experience.

Conclusion

With the increasing technological mediation of our everyday lives, we inhabit what has become a Mixed Reality Environment. HCI as a field has pressing needs to develop new practices to design for and evaluate experiences in this environment. This research presents Experience Probes, an approach to momentary assessment of participant experiences in transition zones between reality and virtuality. The case study presents a soundscape installation and describes the facilitation of a hybrid state among participants in which the experience of immersion is investigated through conscious real-time reflection on a mapping exercise. Expressions produced in this state have been examined for insights into participants' perceptions and we describe how participants express a sense of self-perception as well as a sense of place, intimating their presence and immersion in the virtual space. We surmise from this that with this method the reflective activity of reporting may not necessarily detract from immersion. Experience Probes thus represent an expansion of traditional HCI methodologies to include assessment of participants' momentary perceptions, bypassing the memory-experience gap to evaluate technology-mediated immersive experiences.

Fig. 1 :

 1 Fig. 1: Installation Environment in Trento, Italy and Real Environment, Hong Kong

Fig. 2 :

 2 Fig. 2: EP Mapping Form and North Marker on Floor

Fig. 3 :

 3 Fig. 3: Experience Probe Sample Results

. ( Table 1 :

 (1 Number and % of Participants, Map Style, Reporting Self and Place)

			Reporting	Reporting	Reporting
		Participants N.	Self	Place	Self & Place
	Overall	34 (100%)	14 (41%)	13 (38%)	9 (26%)
	Mapping	18 (52%)	6 (33%)	6 (33%)	5 (27%)
	Sounds
	Mapping	10 (29%)	5 (50%)	7 (70%)	3 (30%)
	Speakers
	other	6 (17%)	3 (50%)	0 (0%)	0 (0%)

Acknowledgments. Special thanks to the organizers of COOP2016 and to Melissa Cate Christ. This work was supported by a fellowship from TIM.


Jianlong Zhou

Syed Z Arshad

Simon Luo

Fang Chen

Effects of Uncertainty and Cognitive Load on User Trust in Predictive Decision Making

Keywords: Trust, uncertainty, cognitive load, predictive decision making

Rapid increase of data in different fields has been resulting in wide applications of Machine Learning (ML) based intelligent systems in predictive decision making scenarios. Unfortunately, these systems appear like a ‗blackbox' to users due to their complex working mechanisms and therefore significantly affect the user's trust in human-machine interactions. This is partly due to the tightly coupled uncertainty inherent in the ML models that underlie the predictive decision making recommendations. Furthermore, when such analytics-driven intelligent systems are used in modern complex high-risk domains (such as aviation) -user decisions, in addition to trust, are also influenced by higher levels of cognitive load. This paper investigates effects of uncertainty and cognitive load on user trust in predictive decision making in order to design effective user interfaces for such ML-based intelligent systems. Our user study of 42 subjects in a repeated factorial design experiment found that both uncertainty types (risk and ambiguity) and cognitive workload levels affected user trust in predictive decision making. Uncertainty presentation leads to increased trust but only under low cognitive load conditions when users had sufficient cognitive resources to process the information. Presentation of uncertainty under high load conditions (when cognitive resources were short in supply) leads to a decrease of trust in the system and its recommendations.

Introduction

Trust has been found to be a critical factor driving human behavior in human-machine interactions with autonomous systems [START_REF] Hoff | Trust in Automation Integrating Empirical Evidence on Factors That Influence Trust[END_REF] and more recently in modern complex highrisk domains such as aviation and military command and control [START_REF] Marusich | Effects of Information Availability on Command-and-Control Decision Making Performance, Trust, and Situation Awareness[END_REF]. It is also one of the most important factors in management and organizational behavior for all personal and business decision making as well as for efficiency and task performance [3] [4]. Trust is influenced by the types and format of information received by humans, their individual approaches to develop and determine trust, and aspects such as system capability and reliability [START_REF] Wheeler | Trusted Autonomy: Concept Development in Technology Foresight[END_REF].

Various definitions of trust have been used. One of the most widely cited definition of trust is from Lee and See [START_REF] Lee | Trust in Automation: Designing for Appropriate Reliance[END_REF], which defines trust as -the attitude that an agent will help achieve an individual's goals in a situation characterized by uncertainty and vul-nerability‖. This definition shows that uncertainty is tightly coupled to trust. Uncertainty indicates it is impossible to determine whether the information available is true or not. There are many variants of uncertainty. The term could refer to statistical variability, noise in the information, nondeterministic relationship between action and consequences, or the psychological reaction to difficult problems. In human-machine interactions, uncertainty often plays an important role in hindering the sense-making process and conducting tasks: on the machine side, uncertainty builds up from the system itself; on the human side, these uncertainties often result in -lack of knowledge or trust‖ or -over-trust‖. Such human's biased interpretation can be partially resolved if we can make uncertainty transparent to users. Furthermore, system transparency is regarded as one vital aspect in maintaining human's trust in and reliance on autonomous systems [START_REF] Mercado | Intelligent Agent Transparency in Human-agent Teaming for Multi-UxV Management[END_REF] [START_REF] Ososky | Determinants of system transparency and its influence on trust in and reliance on unmanned robotic systems[END_REF]. A user might be risking too much by completely ignoring uncertainties and having complete faith in autonomous systems. On the other hand, trivializing autonomous systems or having high uncertainty perception on autonomous systems could possibly dismiss the incredible potential of autonomous systems. Adobor [9] showed that a certain amount of uncertainty is necessary for trust to emerge. Beyond that threshold, however, increase in uncertainty can lead to a reduction in trust. This midrange proposition suggests that there may be an optimal balance between uncertainty and trust.

Moreover, Parasuraman et al. [START_REF] Parasuraman | Situation Awareness, Mental Workload, and Trust in Automation: Viable, Empirically Supported Cognitive Engineering Constructs[END_REF] showed that human cognition constructs such as Cognitive Load (CL) and trust are often invoked in considerations of function allocation and the design of automated systems. For example, in task situations of modern complex high-risk domains, users often need to make decisions in limited time. Therefore, they often make decisions under high cognitive load besides trust issues in such task situations. It was found that a higher cognitive load worsens the situation in relation to trust building [START_REF] Khawaji | Trust and Cognitive Load in the Text-chat Environment: The Role of Mouse Movement[END_REF]. However, it is still not clear how trust varies under both high cognitive load and various uncertainty conditions.

Next we look at decision making, which is now an important research topic in HCI with the fast growing use of intelligent systems [START_REF] Smith | Human-Centered Design of Decision-Support Systems[END_REF]. Rapidly increasing data in fields such as finance, infrastructure and society has motivated users to try integrating -Big Data‖ and advanced analytics into business operations -in order to become more analytics-driven in their decision making. Much of machine learning (ML) research is inspired by such expectations. As a result, we continuously find ourselves coming across ML-based appealing viewgraphs and other predictions that seem to work (or have worked) surprisingly well in practical scenarios (e.g. AlphaGO's beating with professional GO players in 2016 and 2017). So far these machine learning success stories originate from ML technical experts or computing professionals (e.g. Google DeepMind). For many of non-ML users, ML-based predictive analytics software is like a -black box‖, to which they simply provide their source data and (after selecting some menu options on screen) colorful viewgraphs and/or recommendations are displayed as output. The -black box‖ approach has obvious drawbacks: it is difficult for the user to understand the complicated ML models [13,[START_REF] Zhou | Making Machine Learning Useable by Revealing Internal States Update -A Transparent Approach[END_REF]. It is neither clear nor well understood that how trustworthy is this output, or how uncertainties are handled by underlying algorithmic procedures. As a result, the user is more or less unconfident in the ML model output when making decisions based on the ML model output and thus also unconfident in the ML models themselves.

From this perspective, Winkler [START_REF] Winkler | The Importance of Communicating Uncertainties in Forecasts: Overestimating the Risks from Winter Storm Juno[END_REF] emphasized the importance of communicating uncertainties in predictions (as imprecision and uncertainty are unavoidable in predictive analytics). He believed that the consideration of uncertainty is greatly necessary in making rational decisions. It was also found that the presentation of automation uncertainty information helped the automation system receive higher trust ratings and increase acceptance of the system [START_REF] Beller | Improving the Driver-Automation Interaction An Approach Using Automation Uncertainty[END_REF]. This display might improve the acceptance of fallible systems and further enhances human-automation cooperation. However, it remains unclear whether different types of uncertainty (e.g. risk and ambiguity) affect trust building, and if yes how they affect trust building, especially in predictive decision making. Here risk refers to situations with a known distribution of possible outcomes, and ambiguity is the situation where outcomes have unknown probabilities.

This paper aims to investigate the effects of uncertainty on user trust under various cognitive load levels in predictive decision making. Two uncertainty types of risk and ambiguity are presented with predictive model results in a decision making scenario. This follows the user method and approach to design cognitive systems, as reviewed by Candello [START_REF] Candello | User Methods and Approaches to Design Cognitive Systems[END_REF], and used in [START_REF] Arshad | Investigating User Confidence for Uncertainty Presentation in Predictive Decision Making[END_REF]. This user study was deployed as a simulation (derived from the case study) of water pipe failure history analysis for future pipe failure prediction. It shows that both uncertainty presentation and cognitive load levels affect user trust in predictive decision making. The investigation results can be used to design effective user interfaces for ML-based intelligent systems and improve the acceptability of ML techniques by users.

Related Work

The research in human-machine trust and similar cognitive engineering constructs has a rich history [START_REF] Parasuraman | Situation Awareness, Mental Workload, and Trust in Automation: Viable, Empirically Supported Cognitive Engineering Constructs[END_REF]. Several of the efforts in this area can be traced back to Rouse's [START_REF] Rouse | Adaptive Aiding for Human/Computer Control[END_REF] ideas about adaptive aiding, that later, among other things evolved into more advanced HCI techniques. Also that psychophysiology was proposed for adaptive automation [START_REF] Byrne | Psychophysiology and adaptive automation[END_REF] and then trust and self-confidence argued into adaptive automation [START_REF] Moray | Adaptive Automation, Trust, and Self-Confidence in Fault Management of Time-Critical Tasks[END_REF].

Winkler [START_REF] Winkler | The Importance of Communicating Uncertainties in Forecasts: Overestimating the Risks from Winter Storm Juno[END_REF] demonstrated, with the help of several effective examples (from different fields), that probabilities are needed to understand the risk associated with potential decisions as well as to determine measures such as expected payoffs and expected utilities. LeClerc and Joslyn [START_REF] Leclerc | The cry wolf effect and weather-related decision making[END_REF] successfully demonstrated that adding a probabilistic uncertainty estimate in public weather forecasts improved both decision quality and compliance (to evacuation instructions in cases of severe weather threats). Uggirala et al. [START_REF] Uggirala | Measurement of trust in complex and dynamic systems using a quantitative approach[END_REF] studied humans using systems that include uncertainties by having the users rate their trust at each level through questionnaires. Their study showed that trust relates to competence and an inverse relation to uncertainty, meaning that an increase in uncertainty decreases trust in the systems.

Allen et al. [START_REF] Allen | The effect of cognitive load on decision making with graphically displayed uncertainty information[END_REF] investigated the effects of communicating uncertainty information with users using different representations on cognitive tasks. Uncertainty information is typically presented to users visually, most commonly in graphical format [24] [25].

Edwards et al. [START_REF] Edwards | Decision making for risk management: a comparison of graphical methods for presenting quantitative uncertainty[END_REF] compared different graphical methods from presenting quantitative uncertainty in decision making tasks. The representation of uncertainty can have significant impact on human performance. It was shown that when the representation of uncertainty for a spatial task better matches the expert's preferred representation of the problem even a non-expert can show expert-like performance [START_REF] Kirschenbaum | Visualizing Uncertainty The Impact on Performance[END_REF]. This is actually a very good motivation for trying to figure out the preferred representation of uncertainty by different user groups.

Decision making under uncertainty is widely investigated in decision theory [START_REF] Damghani | Decision Making Under Uncertain and Risky Situations[END_REF], where uncertainty is usually considered as probabilities in utility functions. de Visser and Parasuraman [START_REF] Visser E De | Adaptive Aiding of Human-Robot Teaming Effects of Imperfect Automation on Performance, Trust, and Workload[END_REF] conducted two experiments to examine the effects of automation reliability and adaptive automation on human-system performance with different levels of task load by using a high-fidelity multi-UV (uninhabited vehicles) simulation involving both air and ground vehicles. User trust and self-confidence were higher and workload was lower for adaptive automation compared with the other conditions. It was found that human-robot teams can benefit from imperfect static automation even in high task load conditions and that adaptive automation can provide additional benefits in trust and workload.

However, little research has been done on the effects of uncertainty, especially different types of uncertainty such as risk and ambiguity uncertainty, on user trust in predictive decision making under various cognitive load levels. With the use of a case study of predictive decision making for the water pipe failure budget planning, this paper investigates user trust changes under variations of both uncertainty types and cognitive load levels. Two types of uncertainty presentations (risk and ambiguous) and four cognitive load levels are introduced in the study to learn their effects on user trust in predictive decision making.

Experiment

Experiment Data

This research used water pipe failure prediction as a case study for predictive decision making (replicated in lab environment). Water supply networks constitute one of the most crucial and valuable urban assets. The combination of growing populations and aging pipe networks requires water utilities to develop advanced risk management strategies in order to maintain their distribution systems in a financially viable way [START_REF] Li | Water Pipe Condition Assessment: A Hierarchical Beta Process Approach for Sparse Incident Data[END_REF]. Pipes are characterized by different attributes, referred to as features, such as laid year, material, diameter size, etc. If pipe failure historical data is provided, future water pipe failure rate is predictable with respect to the inspected length of the water pipe network [START_REF] Li | Water Pipe Condition Assessment: A Hierarchical Beta Process Approach for Sparse Incident Data[END_REF]. Such models are used by utility companies for budget planning and pipe maintenance. However, different models with various uncertainty conditions may be achievable resulting in different possible budget plans. The experiment is then set up to determine what uncertainty conditions may influence the user's trust during the decision process. The prediction models were simulated following the models such as Weibull and Hierarchical Beta Process (HBP) [START_REF] Li | Water Pipe Condition Assessment: A Hierarchical Beta Process Approach for Sparse Incident Data[END_REF].

Experimental Data

In this study, models are simulated and based on different pipe features (e.g. size or laid year). The model performance curve was presented to let the participants evaluate different models. The model performance is the functional relationship between the inspected length of the network and the percentage of failures detected by the model. Fig. 1 shows the performances of two sample models, where the -blue model‖ outperforms the -red model‖, because the former detects more failures than the latter for a given pipe length.

ML models are usually imperfect abstractions of reality. As a result, imprecision can occur in the prediction through model uncertainty. Model uncertainty here refers to an interval within which the true value of a measured quantity would lie. For example, in Fig. 2 Model uncertainty usually spans as a band in the model performance diagram as shown in Fig. 2. By considering model uncertainty, the relationship between two models may have two cases as shown in Fig. 2: non-overlapping models (see Fig. 2a), and overlapping models (see Fig. 2b). In Fig. 2b, the interval of the model with lower uncertainty is subsumed in the interval of the model with higher uncertainty, whereas in Fig. 2a, the two bands are disjoint.

Task Design

According to the water pipe failure prediction framework, we investigated the decisions made by users under varied conditions. Each user was asked to make a budget plan, i.e. a budget in terms of network length to be inspected, using the failure prediction models learned from the historical pipe failure records. Two ML models were provided for each estimation task. Participants were required to make decisions by selecting one of two presented ML models and then making a budget estimate based on the selected ML model. The budget estimate needs to meet the following requirements:  To inspect as short length of pipes as possible (low cost);  To be as precise in budget estimate as possible (higher accuracy would reflect greater confidence in estimation).

In this study, a module named Automatic Predictive Assistant (APA) is introduced to the user as a new module ‗under testing' phase. The APA is a simulated module which reads in the information provided by the ML models, and then recommends a typical decision (of average accuracy) for the participant. Users can choose to trust, modify or totally ignore the recommendations of APA. The participant needs to evaluate whether she trusts the estimation recommended by the APA. If she does not trust the APA, she is asked to provide her own estimation. Fig. 3 shows the screenshot of a task performed in the study.

The actions a participant needed to do during a task session include answering questions to validate understanding of machine learning performance and uncertainty, and to validate cognitive load levels introduced, as well as making decisions according to information presented. In summary, each task is divided into following major steps: Fig. 3. Screenshot of a task performed in the study.

1) The participant is firstly asked to study the ML performance diagram (in the middle of the screenshot in Fig. 3) and answer questions on model performance and uncertainty to validate her understanding of the information presented. 2) Next, the APA recommendations (at the right side of the screenshot in Fig. 3) are displayed and again the user understanding validated. 3) Finally, the participant is required to estimate the budget by selecting an ML model and its estimation accuracy. If he/she does not trust the recommendations from the APA, he/she is required to provide his/her own estimations (at the left side of the screenshot in Fig. 3) based on the ML performance displayed at the first step. Subjective trust ratings are obtained immediately after this step.

Participants were encouraged to reach the best budget estimates they could as quick as possible.

In this study, cognitive load was introduced by asking participants to remember a random number digit sequence for the duration of task time and reciting it after the task. This dual-task load inducing technique is quite popular in decision making scenarios [START_REF] Deck | The effect of cognitive load on economic decision making: A survey and new experiments[END_REF]. The cognitive load level was determined based on the number of digits being remembered. Four cognitive load levels were applied in this studyfrom low to high, the number of random digits to be remembered ranged from three, five, seven and nine. Three-digit number is for lowest load condition and nine-digit is for the highest load.

There were three different uncertainty visualizations (no uncertainty, nonoverlapping uncertainty, and overlapping uncertainty). Each condition was performed under four different cognitive load levels. Each task was performed for three rounds. All together 36 estimation tasks (3 uncertainty conditions x 4 cognitive load levels x 3 rounds) were conducted by each subject. Three additional training tasks were also conducted by each subject before the formal tasks. The order of tasks was randomized during the experiment to avoid any bias. Slides-based instructions on the concepts of predictive models and uncertainty as well as predictive decision making were presented to each participant before the task time.

Participants and Apparatus

Forty-two participants were recruited from three groups with different background, with the ages ranging from 20 to 57 years: Fourteen participants were ML researchers (experts in ML or data mining research), nineteen were non-ML researchers (participants who were researchers but not in ML or data mining), and nine administrative staff. These three user groups constituted the majority of the users for this particular type of predictive decision making scenario we considered. All were requested to make predictive decisions (using historical data visualized on screen) about the optimal length of pipe (thus budget estimation) to be checked in order to minimize water pipe failures. Information was presented on a 21-inch Dell widescreen monitor. Fig. 3 presents a screenshot of a task performed in the study.

Data Collection

After each decision making task, participants were asked to rate their trust in APA recommendations and also confidence in their own decisions (using a 9-point Likert scale where 1= least trust, and 9 = most trust). Besides trust subjective ratings, cognitive load rankings for each task from subjects were also collected using a 9-point Likert scale (1= least mental effort, and 9 = most mental effort) for load validation purposes.

Results

Fig. 4 and Fig. 5 depict the summary visualization of trust measured via subjective responses of all 42 subjects. Trust values were normalized with respect to each subject to minimize individual differences in rating behavior. Since we had more than two dependent samples, we first performed Friedman test and then followed it up with post-hoc analysis using Wilcoxon signed-rank tests (with a Bonferroni correction) to analyze differences in participant responses of trust for various conditions.

Trust and Uncertainty: Fig. 4 shows normalized trust values over the uncertainty treatments. Control task had only point prediction lines (refer to Fig. 1) and no uncertainty was presented. Risk uncertainty was presented by models with non-overlapping uncertainty (see Fig. 2a) and ambiguity by overlapping uncertainty models (see Fig. 2b). When participants experienced uncertainty type ambiguity (rightmost group of columns in Fig. 4), Friedman test for cognitive load level conditions showed a statistically significant difference in trust among four CL levels, , p < .006. Then post-hoc Wilcoxon tests (with a Bonferroni correction under a significance level set at p<.013) was applied to find pair-wise differences between levels in trust. The adjusted significance alpha level of .013 was calculated by dividing the original alpha of .05 by 4, based on the fact that we had four load level conditions to test.

The post-hoc tests found that for uncertainty condition of ambiguity, participants had significantly lower trust under high cognitive load (CL4), with p < .001, compared to that of low load (CL1). More details of this result and its implications for subject groups appear in discussion ahead.

Trust and Cognitive Load: Fig. 5 shows normalized trust values over cognitive load levels. Here we are interested only in the extreme load levels administered, namely CL1 (the lowest) and CL4 (the highest), as they are the most relevant for automated cognitive load management [START_REF] Arshad | Interactive Mouse Stream As Real-Time Indicator of User's Cognitive Load[END_REF]. Friedman's test of cognitive load level conditions of the lowest (CL1) and highest (CL4) both gave statistically significant differences in trust among three uncertainty conditions, , p < .004 and , p < .006 respectively. Then post-hoc Wilcoxon tests (with a Bonferroni correction under a significance level set at p<.017) was applied to find pair-wise differences between uncertainty conditions. The adjusted significance alpha level of .017 was calculated by dividing the original alpha of .05 by 3, based on the fact that we had three uncertainty conditions to test. The post-hoc tests found that for low cognitive load condition (CL1), trust in the condition of ambiguity was significantly higher (p < .006) than that of risk condition (Fig. 5, leftmost group of three columns). Whereas, for high cognitive load (CL4) condition, trust in risk condition was significantly lower (p < .003) than control condition. More details of this result and its implications for subject groups appear in discussion ahead.

Discussion

As discussed in earlier sections, trust is a challenging concept to study and investigate in, therefore, we opted to study human-machine trust in a specialized predictive decision making scenario. Predictive decision making support and automated aids have become quite popular with the advent of new machine learning based intelligent applications. Now that machines are becoming more intelligenthuman-computer interaction must also evolve accordingly. Only by working together as a trusted team can humans and machines improve efficiency and productivity.

Trust and Uncertainty: Generally, in an automated predictive decision making scenario, humans are required to make future oriented decisions based on the information or recommendation presented on the screen by a machine learning (and data crunching) model/algorithm that mostly works on historical data behind the scenes (appearing like a black box to user). Since these decisions are about the future, there can be no absolutely correct answers -but only better and more appropriate ones based on a more precise understanding of the underlying data presented during the decision making process. Therefore, better presentation and adequate communication of uncertainty inherent in the underlying ML process can improve the trust of the user in the system and lead to better and effective decisions. In our case, we experimented with visualizing and communicating two forms of uncertainty, namely, risk and ambiguity. Risk is a form of uncertainty where all probabilities related to outcomes are known. The user, with the help of these known probabilities, can be expected to make better and well-informed decisions quickly. Such risk type uncertainty was represented by non-overlapping models (see Fig. 2a). The other type of uncertainty we experimented with was ambiguity, which was represented by overlapping models (see Fig. 2b) and where probabilities of outcomes were either unknown or not clearly stated. The control condition was the case where models were presented without any uncertainty component.

Looking at the overall results (Fig. 4), no clear trends can be observed for risk type uncertainty condition, but a clear trend of falling trust can be seen for uncertainty of type ambiguity as cognitive load level increases. It can be said that under low cognitive load (implying greater availability of cognitive resources), users felt more confident analyzing and interpreting the ambiguity type of uncertainty and therefore appear to trust the judgement/recommendation of the automated predictive assistant as it made more sense to them. However, under high cognitive load, the users might find themselves almost at the edge of their working memory capacity. Limited cognitive resources would result in lesser understanding of the ambiguity type of visual. This in turn is indicated by reduced trust in the system and its recommendations. This phenomenon seems to be in line with findings that the better the person understands the system and it's working the greater the person is willing to trust it [START_REF] Kirschenbaum | Visualizing Uncertainty The Impact on Performance[END_REF].

Further drilling down deeper into this trust (over ambiguity type uncertainty) phenomenon into subject groups (administration, machine learning experts and nonmachine learning experts) also leads to an interesting insight (see Fig. 6). Clearly the level of trust for ambiguity type uncertainty presentation appears to drop for all subject groups as cognitive load increases. High cognitive load appears to impact the trust same for all administrative staff and experts (whether they be machine learning or non-machine learning).

Important lessons here for improved trust can be to either avoid ambiguity type of uncertainty representation altogether or present it only when condition of low cognitive load. Also another direction could be to look for alternate ways of visualizing ambiguity type of uncertainty.

Trust and Cognitive Load: It is well known that human performance can be significantly affected by high cognitive or mental workload [START_REF] Chen | Robust Multimodal Cognitive Load Measurement[END_REF]. Cognitive workload is the load on working memory that the user experiences when engaged in a cognitive problem. In our case, the trust in decision making is influenced by a cognitive phase where user tries to make sense of the model data/visuals presented. Since the decision making task was soft time bound, the user must make efficient use of available cognitive resources in order to complete the task. Here we look at the two extreme conditions where most cognitive resources were expected to be available (CL1) and where least cognitive resources were expected to be available (CL4). As stated earlier in the re-sults section, Friedman test for both these extreme conditions (CL1 & CL4) turned out to be significant. In low load condition (CL1), trust for ambiguity type uncertainty was significantly higher than risk type uncertainty (see leftmost group of columns in Fig. 5). The trend seems to be the same for all subject subgroups (see Fig. 7). Trust, under low load conditions, seems to be consistently higher for all groups whenever uncertainty of ambiguity type is presented. However, on further testing, only non-ML group (rightmost group of columns in Fig. 7) yielded significantly (p<.006) higher trust level uncertainty types ambiguity to that of risk. A limitation here could be the lower number of subjects in groups other than non-ML experts. These findings go on to support the idea discussed earlier that uncertainty of type ambiguity can be readily processed by users only under low cognitive load conditions. Likewise in high load condition (CL4), trust for risk type uncertainty was significantly lower than control condition of no uncertainty presentation (see rightmost group of columns in Fig. 5). The trend seems to be similar for all subject subgroups (see Fig. 8). Trust, for both uncertainty conditions, seems to be consistently lower for all groups with respect to control condition. On further testing, only non-ML group (rightmost group of columns in Fig. 8) yielded significantly (p<.003) lower trust level uncertainty type risk to that of control. A limitation here could be the lower number of subjects in other groups than non-machine learning experts. Of the total 42, there were 9 administrative staff, 14 machine learning experts and 19 non-machine learning experts.

These findings go on to support the idea that people trust less what they have not had time to understand. Visuals presented in control condition are straightforward with no complicationhowever, they are simple, but only at the cost of hiding away the uncertainty inherent in the ML models. Once attempts are made to communicate the uncertaintythe trust seems to increase from control to risk and ambiguity uncertainty only under conditions of low cognitive load (see Fig. 7) and decrease under conditions of high cognitive load (see Fig. 8).

User Groups: The three user groups of administrative staff, ML experts and Non-ML experts constitute the majority of the users of such predictive decision making interfaces [START_REF] Arshad | Investigating User Confidence for Uncertainty Presentation in Predictive Decision Making[END_REF]. Of the people involved in predictive decision making, administrative staff is expected to be least knowledgeable of statistical probability and its representations. Non-ML experts may or may not be knowledgeable of statistical probability but they can be expected to be familiar with model representations. Finally the ML ex-perts are expected to be the most knowledgeable of statistical probability and also of uncertainty inherent in ML models. In our case, from the total of 42, there were 9 administrative staff, 14 machine learning experts and 19 non-machine learning experts. The administrative group by itself is too small in number for any meaningful inference. ML-experts were reasonably good in number but nothing significant would be inferred. Only the Non-ML group appears to significant results in certain conditions as discussed above. Overall, we can say that uncertainty presentation can lead to increased trust but only under low cognitive load conditions when user has sufficient cognitive resources to process the information. Presentation of uncertainty under high load conditions, when cognitive resources are short in supply can lead to lowering of trust in the system and its recommendations.

In order to incorporate these findings in HCI and real-world applications, the user interface for an ML-based intelligent system needs to include the following components:

 Components which show uncertainty of ML models. This could help users increase trust in their decisions;  Feedback on user trust and load levels that allow users interfaces to adapt accordingly.

These components may be incorporated into the framework of adaptive measureable decision making proposed in [START_REF] Zhou | Measurable Decision Making with GSR and Pupillary Analysis for Intelligent User Interface[END_REF], thereby introducing trust levels into an adaptive decision making process and allowing for efficient and informed decisions.

Therefore, besides decision quality as demonstrated in [START_REF] Helgee | Improving Drug Discovery Decision Making using Machine Learning and Graph Theory in QSAR Modeling[END_REF], the revealing of user trust levels in predictive decision making also benefits the evaluation of ML models. From this perspective, this study made -black-box‖ ML models transparent through revealing user responses to ML models, but not directly explain how ML algorithms process data to get outputs with visualizations or feature contributions [START_REF] Štrumbelj | Explaining prediction models and individual predictions with feature contributions[END_REF][START_REF] Zhou | Making machine learning useable[END_REF][START_REF] Krause | Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models[END_REF], where domain users still have difficulty to understand those complex visualizations and abstract numbers. The revealing of user trust in a predictive decision making scenario is more meaningful for both ML researchers and domain experts, and therefore help improve the acceptance of ML solutions by users.

In summary, this study showed that uncertainty and cognitive load played significant roles in affecting user trust in predictive decision making. Furthermore, various uncertainty types had different effects on user trust perceptions. These findings have at least two benefits in real-world applications: 1) to design intelligent user interface of predictive decision related applications in HCI. The user interface, which shows user trust in decision making in real-time, would help users make informed decisions effectively; 2) to evaluate ML models in ML research areas by measuring what is the user trust level in decision making based on ML output.

Conclusions and Future Work

This paper investigated the effects of uncertainty of ML models and cognitive load on user trust in predictive decision making in order to design effective user interfaces for ML-based intelligent systems. A user study found that both uncertainty types, as well as the cognitive load levels affected user trust in decision making. Furthermore, various user groups showed different trust perceptions under both uncertainty and cognitive load conditions.

Our future work will focus on analyzing physiological signals, such as Galvanic Skin Response (GSR) and Blood Volume Pulse (BVP) signals, as well as behavioral signals (mouse movement) of participants for indexing user trust levels during MLbased decision making. The relationship between the trust and the task performance will also be analyzed. Our ultimate goal is to set up a framework of measurable trust in decision making in order to dynamically adjust trust levels in ML-based intelligent systems.

  (a), in order to inspect 20% of the pipes in length, the uncertainty interval of the failure rate is [ ] for the blue model, and about [ ] for the red model: the red model is said to have less uncertainty in prediction than the blue model because the red model has smaller uncertainty interval than the blue model.

Fig. 1 .

 1 Fig. 1. Performance curves of ML models.

Fig. 2 .

 2 Fig. 2. Predictive models with uncertainty: (a) non-overlapping models, and (b) overlapping models.

Fig. 4 .

 4 Fig. 4. Trust over Uncertainty Presented; Control (No Uncertainty), Risk (Non-Overlapping Uncertainty) and Ambiguity (Overlapping Uncertainty).

Fig. 5 .

 5 Fig. 5. Trust over Cognitive Load Levels.

Fig. 6 .

 6 Fig. 6. Trust for Ambiguity type uncertainty.

Fig. 7 .

 7 Fig. 7. Trust over Subject Groups (Low CL).

Fig. 8 .

 8 Fig. 8. Trust over Subject Groups (High CL).

Acknowledgements

Authors thank all volunteer participants for the experiment. This work was supported in part by the Asian Office of Aerospace Research & Development (AOARD) under grant No. FA2386-14-1-0022 AOARD 134131.