Joel Harman

Ross Brown

Daniel Johnson

Improved Memory Elicitation in Virtual Reality: New Experimental Results and Insights

Eliciting accurate and complete knowledge from individuals is a non-trivial challenge. In this paper, we present the evaluation of a virtual-world based approach, informed by situated cognition theory, which aims to assist with knowledge elicitation. In this approach, we place users into 3D virtual worlds which represent real-world locations and ask users to describe information related to tasks completed in those locations. Through an empirical A/B evaluation of 62 users, we explore the differences in recall ability and behaviour of those viewing the virtual world via a virtual reality headset and those viewing the virtual world on a monitor. Previous results suggest that the use of a virtual reality headset was able to meaningfully improve memory recall ability within the given scenario. In this study, we adjust experiment protocol to explore the potential confounds of time taken and tool usability. After controlling for these possible confounds, we once again found that those given a virtual reality headset were able to recall more information about the given task than those viewing the virtual world on a monitor.

Introduction

The ability to accurately gather knowledge is a non-trivial challenge with relevance to a variety of domains [START_REF] Davis | Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review[END_REF]. Furthermore, if this information is not gathered effectively, there is the potential for many complications to arise. For example, when attempting to formalize business practices, working with incorrect or incomplete information may lead to higher construction costs, longer development times or poor-quality products [START_REF] Smith | The role of tacit and explicit knowledge in the workplace[END_REF].

Many approaches have previously been explored which aim to elicit information from stakeholders [START_REF] Zowghi | Requirements elicitation: A survey of techniques, approaches, and tools[END_REF]. These methods, however, usually produce a trade-off between the quality of information gathered and the time required to do so [START_REF] Davis | Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review[END_REF]. For example, questionnaires can usually be administered easily, but often yield inaccurate or incomplete information. Role-plays on the other hand, have been shown to be an effective means of gathering accurate information [START_REF] Costain | Experiencing the elicitation of user requirements and recording them in use case diagrams through role-play[END_REF], but are used less often in a business setting [START_REF] Zowghi | Requirements elicitation: A survey of techniques, approaches, and tools[END_REF]. It has been suggested that this may be due to issues related to the potentially high setup costs and co-location issues related to role-play [START_REF] Davis | Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review[END_REF]. Despite these issues, in this paper we will be considering a modified role-play approach due to the benefits it affords when looking to elicit accurate and complete information.

When performing a role-play, the individual steps in a task are usually executed in a specific sequence. The concept of memory chaining proposes that we are able to recall a piece of information more effectively if we are already thinking about the task it precedes [START_REF] Mace | Involuntary memory chaining versus event cueing: Which is a better indicator of autobiographical memory organisation?[END_REF]. This would suggest that even potentially mundane tasks, which may be ignored in other elicitation approaches (e.g. interviews), may initiate memory chains which could potentially yield relevant information. Furthermore, the rich context afforded during an in-situ role-play may also be sufficient in achieving a situated cognition response. Situated cognition is the theory that all knowledge is, to some extent, tied to the situations and contexts in which it was learned [START_REF] Brown | Situated cognition and the culture of learning[END_REF].

In our proposed approach, we look to leverage many of the benefits of a standard role-play. Rather than conduct the role-play within the real world, however, the session is instead conducted within a virtual environment closely resembling the real world location. By doing this, we aim to achieve many of the elicitation advantages associated with role-play, while mitigating many of the common detractors. Specifically, this approach may be preferable when a real world role-play would incur high setup costs, excessive risks, or have issues with co-location. For example, when looking to role-play a complex task, it may be difficult to arrange a time when all parties involved can be in the same location concurrently, without disrupting existing work commitments. This is the third in a series of related studies exploring the potential recall benefits associated with proving situated context via a virtual world role-play. In our first study [START_REF] Harman | Augmenting process elicitation with visual priming: an empirical exploration of user behaviour and modelling outcomes[END_REF], we found that when comparing recall ability of participants provided with the virtual world stimuli and participants that weren't, participants given the stimuli were able to recall more information. In this first study, we conjectured that the stimuli provided by the virtual world was able to assist the users in recalling more information. Furthermore, we conjectured that if a more immersive interface was used (e.g. a virtual reality headset), a larger effect may potentially be observed. We later tested this theory in the second study and compared the recall ability of participants provided with a virtual reality headset with participants provided with a desktop display (monitor) [START_REF] Harman | The role of immersion during situated memory recall within virtual worlds[END_REF]. The results of this experiment found that participants given the virtual reality headset were able to recall more information, but the experiment did identify some potential confounds which may have affected our results. In particular, we found that there were significant differences in both time taken and usability. Without further exploring these issues, we were unable to say with any degree of certainty whether the observed recall results were meaningfully due to the difference in viewing mechanism immersion, or some other secondary effect.

In the study presented in this paper, we aim to further explore this phenomenon and clarify the potential effect of confounds we found in our prior studies. In particular, we believe that the virtual world design and experiment setup we chose to use in our prior studies may have been too difficult for novice users, leading to issues with usability and subsequently affecting the time taken by participants experiencing these issues. As the participants given the virtual reality headset were using a new interface, we believe that this issue was more pronounced within this group, leading to the differences in usability and time taken between the two groups. In this study, we look to obtain insight into these issues by making the task easier to learn, understand and execute for participants.

Situated Cognition and Virtual Worlds

Applying the concept of situated cognition within virtual worlds has been extensively explored in literature. When situated cognition was first proposed in 1989, it considered the concept of situated learning [START_REF] Lave | Situated learning: Legitimate peripheral participation[END_REF] to be a related concept [START_REF] Brown | Situated cognition and the culture of learning[END_REF]. Situated learning postulates that students need to be taught information within the contexts in which it will be applied.

Situated learning research has primarily considered virtual worlds for their potential applications to distance learning [START_REF] Dede | Designbased research strategies for studying situated learning in a multi-user virtual environment[END_REF]. The goal of these environments is to place students within virtual classrooms, in order to better situate them within an environment they associate with a student-teacher learning dynamic. Some research even suggests that this approach may be an improvement over typical distance-learning communication methods, such as conference calls [START_REF] Dickey | Teaching in 3D: Pedagogical affordances and constraints of 3D virtual worlds for synchronous distance learning[END_REF].

Outside of the context of learning, there has been an absence of literature which has explored the theory of situated cognition and the appropriateness of virtual worlds for achieving the necessary degree of situated context. We found one theoretical paper, however, which discusses the appropriateness of existing measures, such as immersion, and their importance in achieving effective situated cognition responses within virtual worlds [START_REF] Carassa | A Situated Cognition Perspective on Presence[END_REF]. Immersion, within the context of this study, is the degree to which a person's senses are engaged [START_REF] Mania | The effects of levels of immersion on memory and presence in virtual environments: A reality centered approach[END_REF]. For example, a movie with sound may be considered more immersive than one without, as it engages the person's sense of hearing. As this study discusses the potential for assisting memory recall within virtual worlds via situated cognition, we have decided to primarily explore whether differences in immersion may affect user recall performance.

Research Questions

In this study, we have looked to explore how differences in viewing immersion may affect participant behaviour and memory recall ability. This had led to the formation of two distinct research questions:

-RQ1: How do changes in viewing immersion affect memory recall performance of users when they are asked to describe information while roleplaying within a related virtual world?

-RQ2: How do changes in viewing immersion affect the behaviour of participants when they are asked to describe information while role-playing within a related virtual world?

We have chosen to explore potential behavioural differences as our prior studies have suggested that there may be a relationship between participant behaviour and recall performance [START_REF] Harman | The role of immersion during situated memory recall within virtual worlds[END_REF]. To examine the effect of this change in immersion, we will be comparing a virtual reality setup, using an HMD, with a standard desktop display. From the two above research questions, we have constructed two main hypotheses after exploring existing theory within related literature. The first hypothesis is:

-H1: Users asked to recall information while viewing a related virtual world within an HMD will recall more information than those viewing the virtual world on a desktop display.

In prior work, we found preliminary evidence to suggest that the context provided by a virtual world was able to assist participants with recalling information [START_REF] Harman | Augmenting process elicitation with visual priming: an empirical exploration of user behaviour and modelling outcomes[END_REF]. We conjecture, however, that while the virtual world provided some benefit, it did not, wholly, afford the necessary context to the user. Specifically, we believe that the embodiment and immersion afforded to the user by virtual reality will better situate the user and assist them in recalling more effectively. In addition to recall improvements, we believe that the added embodiment and immersion provided by the HMD will also result in certain behavioural changes between the two groups. This has led to the formation of our second hypothesis:

-H2: Users provided with the HMD will be more exploratory in their approach than those provided with the desktop display.

To better evaluate this hypothesis, we will be exploring two sub-hypotheses. These are:

-H2(a): Users provided with the HMD will traverse a larger portion of the virtual world than those provided with the desktop display.

-H2(b): Users provided with the HMD will adjust their view within the virtual world more often than those provided with the desktop display.

While we do not necessarily believe that virtual reality innately makes a participant more exploratory, we do believe that the interaction mechanisms it affords makes it easier for participants to operate in a way in which they are familiar and experienced. While this hypothesis was somewhat supported in our prior experiments [START_REF] Harman | Augmenting process elicitation with visual priming: an empirical exploration of user behaviour and modelling outcomes[END_REF], we look to confirm that these effects remain present if the user does not need to continually halt movement to enter descriptions of their actions. Furthermore, we believe the evidence for increases in emotions [START_REF] Riva | Affective interactions using virtual reality: the link between presence and emotions[END_REF], telepresence and sexual presence [START_REF] Renaud | Measuring sexual preferences in virtual reality: A pilot study[END_REF] when using an HMD, rather than a desktop display, are all examples of how users appear to behave more naturally within virtual reality. We wish to understand if this more natural behaviour may lead to observable increases in recall ability for the subjects.

Virtual worlds are synthetic environments which provide users with an avatar through which they can explore and interact with other users, or the environment itself, to perform various tasks [START_REF] Duncan | A taxonomy of virtual worlds usage in education[END_REF]. For this study, we have constructed a virtual world which aims to be representative of a real world airport.

A common issue related to research within this area, however, is that the virtual worlds can be difficult for participants to use when they are unfamiliar with the features and have not been provided with extensive training within the environment [START_REF] Virvou | On the usability and likeability of virtual reality games for education: The case of VR-ENGAGE[END_REF]. To mitigate this issue, we have developed a virtual world specifically to explore this recall phenomenon.

The virtual world was developed using the Unity3D [START_REF]Unity -Game Engine[END_REF] game engine. The environment was constructed using a mix of both constructed and prebuilt assets available via the Unity 3D asset store. Furthermore, we have also looked at trying to improve the usability of the virtual world in previous experiments [START_REF] Harman | Model as you do: Engaging an S-BPM vendor on process modelling in 3D virtual worlds[END_REF] and made modifications to improve usability where possible. We have endeavoured to reconstruct the airport as best as possible. For this reason, we have included both critical areas (check-in, security, boarding and the plane itself), and noncritical areas (parking, shops, luggage collection and restrooms). Figure 1 shows screen captures of the developed virtual world. To navigate the virtual world, participants from both the HMD and desktop display conditions were both given an Xbox 360 game controller. This controller had two main functions:

-The left joystick controlled avatar movement.

-The right joystick controlled avatar view changes.

We chose to use a game controller for this study as it allowed both treatment groups to use similar mechanisms for both navigating and viewing the environment. It must be noted, however, that as a core component of the HMD is to assist with adjusting avatar view, the HMD was still responsible for view changes within this condition and the vertical view component of view change was disabled on the controller joystick. Furthermore, both treatment groups moved at the same constant speed and had the same turning speed when using the controller.

Unlike all prior experiments we have conducted to explore this phenomenon, the virtual world which was used in this experiment did not afford any interaction mechanisms to the user. Previously, participants were required to first look at an object or person within the virtual world, press a button to select the object and then write down the actions that they would normally complete with the object. While this approach greatly assisted in structuring the user responses, this did also present issues regarding usability. In this experiment, participants instead dictated all actions they would usually complete as they traversed through the virtual world.

Experiment Procedure and Measures

Participants who agreed to participate in the study were invited into a lab. They were then either presented with a 24 inch desktop monitor, or an HMD (Oculus Rift DK2), determined by random assignment. In addition to this, they were also provided with a game controller. A microphone was placed on the desk to record everything the participant said throughout the experiment. A between-subject design was chosen as we believed that participation in one treatment would meaningfully affect participant performance in the subsequent treatment. Upon entering the lab, participants read and signed the associated consent form and answered a series of control questions (e.g., knowledge of the boarding process, prior experience with virtual worlds). Following this, participants were placed into a virtual building, where they were trained to both view and traverse the environment. Participants were then instructed that for the experiment, they would be role-playing the task of boarding an airplane within a virtual airport. While doing this, participants were asked to verbally describe all the actions they would normally take within the environment, being as detailed and specific as possible. Participants were then placed into the airport to begin the experiment. No explicit time limit was placed on participants during their role-play. After participants were happy with their description of the task, they were asked to complete questionnaires related to presence and tool usability.

Recall Measure

The recall measure is the main item of interest in this study. To measure recall, we will be exploring the number of tasks described by participants within the two groups. This task count was generated by transcribing the logs from each session and manually assigning the phrases used by participants into individual tasks. For example, if a participant said "I hand my check in information to the boarding staff", it would be counted as a single task. If the participant had instead specified "I hand my check-in information and luggage to the boarding staff", however, this would instead be recorded as two different tasks (the giving of the check-in information and the giving of the luggage). We chose to use this approach as we emphasized to participants during the session that they should aim to be as specific as possible when describing the actions that they would need to take.

To provide further insight into this result, we will also be comparing the total number of words used by participants in the two treatment groups. While this is not necessarily a measure of recall performance, it does provide an objective measure of how much the participants in both groups spoke. This may provide some validation for our recall measure if the differences between the two groups are similar (e.g. Group A described more than Group B because they spoke more), or create interesting insights if the number of words spoken result is not reflected in recall performance (e.g. Why did Group A describe less than Group B, despite talking more? What were they saying?).

To ensure that both treatment groups had a similar understanding of the task, participants were asked to both state the number of times they had been on a plane in the last five years and to rate their subjective knowledge of the airplane boarding task. As we did not prime participants with a description of the task prior to the experiment, and each participant may have their own nuanced personal approach for completing the task in the real world, we have opted to not examine the results for correctness. We chose not to prime participants as it provides a much more ecologically sound basis for the study. If we had primed participants, it would not be measuring long-term memory ability, but instead short-term memory of the description they were given. We have, however, removed tasks from our first measure where the statement was immediately identified as erroneous (e.g. the participant described the same task several times in a row or backtracked to try and correct a mistake). Finally, it should be noted that this task, while exploring recall ability, is technically measuring task performance. We believe that this is valid, however, as memory tests often require the participants to constrain the information they recall to the requirements of a task (such as a quiz) (e.g. [START_REF] Roediger | Test-enhanced learning taking memory tests improves long-term retention[END_REF], [START_REF] Ivanoiu | Memory evaluation with a new cued recall test in patients with mild cognitive impairment and Alzheimers disease[END_REF]).

Exploration

We have chosen to examine how the users traversed the environment as we believe that this may provide insight into the behavioural approach taken by the participants. This is important as we believe differences in behaviour may influence recall performance. In our prior study, we found that participants given the HMD treatment explored a larger portion of the environment than those given the desktop display [START_REF] Harman | The role of immersion during situated memory recall within virtual worlds[END_REF]. To measure this, we once again split the virtual environment into 216 5m x 5m segments (virtual world units were designed to be approximately equivalent to real-world units). For this analysis, we compare the average number of segments traversed by the two groups.

In our prior study, we found that participants given the HMD traversed more of the environment than those given the desktop display. As the way the two groups traversed the environment did not fundamentally change in this study, we expect a similar result will be observed.

Change of View

We will be exploring change of view factors in order to better understand how the two treatment groups examined the virtual environment. This paper was primarily motivated by the concept that inserting a user into a situation with the appropriate context would assist them in recalling information related to that situation. For this to be an effective approach, however, the user must view and understand this context. For this reason, we conjecture that, to some degree, viewing more of the environment may result in participants recalling more information about the task.

For this study, participants given the desktop display adjusted their view entirely with the provided game controller, while participants given the HMD adjusted their view with both the controller and the HMD. We will be considering view change as the average change of vertical viewing angle per minute. We have chosen to use only the vertical component of view change as the participants, in addition to making horizontal view changes to examine the environment, also adjust their view horizontally when they need to turn. This means that horizontal view changes do not necessarily indicate that a user is looking to adjust their view of their avatar specifically to view a different part of the environment. As vertical view changes were exclusively performed in order to explore the environment, we believe that this is the more accurate measure in this scenario. To measure these view changes in both groups, we will be examining the rotation made by the avatar, rather than the raw HMD tracking data. We have chosen to do this as we wanted to keep this measure consistent between the two treatment groups. Furthermore, the functions controlling avatar jitter automatically removed HMD jitter, and should therefore more accurately reflect the intentional view changes made by the users.

In our prior study, we found that participants given the HMD adjusted their view more often than those given the desktop display. As the way the two groups view the environment did not change in this study, we expect to see a similar result in this study.

Time Taken

This time taken measure considers only the time taken from when the participant begins describing the task to when they finalise their description. The time taken for all other activities (e.g. questionnaire completion) is not included in this measure. We have chosen to examine the time taken by both groups in the virtual world as its relationship has remained uncertain in our prior experiments. One existing study found that participants who described tasks faster tended to perform better [START_REF] Claes | Tying process model quality to the modeling process: the impact of structuring, movement, and speed[END_REF], but their approach did not provide the situated context central to the approach presented in this paper. In our prior work, we found that participants given the HMD both took longer to complete the task and performed better. As we found no existing literature which explored a potential link between the time spent viewing stimuli and recall performance, we were unable to adequately determine whether the differences in recall performance were due to differences in time taken between the two treatment groups, or due to the levels of immersion provided by the associated interfaces.

In our prior paper, we present several reasons why participants given the HMD may have taken longer to complete the task. For example, we identified that usability issues identified by participants given the HMD may have accounted for this difference in time taken. The design of this study aims to eliminate the differences in usability we found in our previous study by allowing participants to describe information verbally, rather than interacting with the world itself. For this reason, we will once again be examining time taken to determine whether there remains a difference in time taken by the two groups, despite the differences in experiment design.

Presence

Presence, as related to immersive virtual reality, is considered to be the concept of transportation. People are considered present when they report a sensation of being, to some degree, in the virtual world (e.g., you are there) [START_REF] Schumie | Research on presence in VR: a survey[END_REF]. We have chosen to measure presence in this study as existing literature suggests there may be a link between presence and recall performance [START_REF] Carassa | A Situated Cognition Perspective on Presence[END_REF]. In this study, we will be administering the Witmer and Singer presence questionnaire [START_REF] Witmer | Measuring presence in virtual environments: A presence questionnaire[END_REF]. This questionnaire was chosen despite some criticism regarding its efficacy [START_REF] Schubert | The experience of presence: Factor analytic insights[END_REF], as it remains the most widely administered survey for measuring presence in virtual worlds designed for non-game related activities.

In our prior study, we found inconclusive presence findings, with our quantitative results suggesting the HMD condition experienced lower presence while our qualitative findings suggested they experienced higher presence. We conjectured that this difference was due to the complexity of the task. Presence studies usually involve very simple tasks, where usability is not likely to be an issue. As this study aims to provide a much easier experience for the user to comprehend, we hope that the presence findings obtained in this experiment will not suffer from the inconsistencies encountered in our prior study.

Usability

We have chosen to measure usability as HMDs have been known to suffer from challenges with usability among novice users [START_REF] Sutcliffe | Evaluating the usability of virtual reality user interfaces[END_REF]. If a user has difficulty using the virtual environment, it may have ramifications on recall performance in addition to other aspects of the session. We have measured usability with the IBM usability satisfaction survey [START_REF] Lewis | IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use[END_REF] as it remains a widely used measure for usability of software systems.

Furthermore, we are particularly interested in examining whether there are any usability differences between the two treatment groups as we found the HMD condition to have considerably lower usability in our previous study [START_REF] Harman | Augmenting process elicitation with visual priming: an empirical exploration of user behaviour and modelling outcomes[END_REF]. In this experiment, we have chosen to use a much simpler system for gathering information from participants, which we believe may mitigate this difference in usability between the two groups.

Participants in this study were randomly assigned to either the HMD or desktop display conditions, with each condition having 31 participants. For comparing between the two conditions, we will be using two-tail Students-t tests when the data is both continuous and passes the Shapiro-Wilk W normality test (p < 0.05). If either of these conditions is not met, the two-tail Mann-Whitney U test will be used instead.

The average age of the participants was 24.84 (SD = 5.02). No significant difference was found between the age of those given the HMD and those given the desktop display (p = 0.39). Perceived understanding of the airport boarding scenario was quite high, with an average response of 5.40 (SD = 1.63) on a 7point Likert scale. No statistically significant difference was found in perceived understanding between those given the HMD and those given the desktop display (p = 0.41).

Given the exploratory nature of the research presented and the relatively small sample size, we have not applied a Type I error correction (e.g., Bonferroni) to our analysis. Instead, we have elected to provide effect size calculations (Cohen's d) for all readings to provide the reader with a sense of the magnitude of all presented findings. Following [START_REF] Cohen | A power primer[END_REF] [START_REF] Cohen | A power primer[END_REF], we treat all effect sizes between 0.2 and 0.5 as small, 0.5 and 0.8 as medium and greater than 0.8 as large. Due to the increased possibility of a Type I error, the findings presented in this study should be interpreted with a greater degree of caution. For consistency, we have used existing conversion formulas to convert the r effect sizes calculated for the Mann-Whitney U tests into comparable Cohen's d values [START_REF] Rosenthal | Parametric measures of effect size[END_REF].

Recall

In this study, we were primarily interested in comparing the recall ability of participants presented with the HMD and the desktop display. To evaluate H1, we compared the average number of tasks specified by both treatment groups. This test found that participants given the HMD were able to recall a larger number of tasks (M = 10.50, SD = 2.48) than those given the desktop display (M = 8.97, SD = 2.72, U = 327, z = 2.15, p < 0.05, d = 0.57).

To further explore this result, we also compared the average number of words spoken by both treatment groups. This test found that participants given the HMD spoke a larger number of words (M = 212, SD = 47.3) than those given the desktop display (M = 189, SD = 20.2, U = 272, z = 2.93, p < 0.005, d = 0.80).

Exploration

To evaluate Hypothesis 2(a), we have compared the mean amount of the environment traversed by the two groups. Results from this test found that participants given the HMD traversed a larger number of segments (M = 88.2, SD = 27.4) than participants given the desktop display (M = 66.0, SD = 20.9, t(60) = 3.58, p < 0.001, d = 0.91).

Change of View

To evaluate Hypothesis 2(b), we have measured the average amount participants adjusted the vertical view of their avatar. Results from this test found that participants given the HMD adjusted the vertical view of their avatar more often per minute (M = 755, SD = 404) than those given the desktop display (M = 198, SD = 71, t(60) = 7.56, p < 0.0001, d = 1.92).

Presence

Participants reported higher presence in the HMD condition (M = 5.18, SD = 0.42) than the desktop display condition (M = 4.83, SD = 0.53, U = 282, z = 2.79, p < 0.05, d = 0.76).

Usability

No statistically significant difference in usability was found between participants given the HMD and participants given the desktop display (p = 0.16).

Time Taken

No statistically significant difference in time taken was found between participants given the HMD and participants given the desktop display (p = 0.09).

Discussion of Results

The primary aim of this study was to further confirm whether improvements to interface immersion (specifically viewing immersion) may directly translate into better long term episodic memory recall. This work has been motivated by the continuing need to develop elicitation techniques which can effectively gather knowledge from individuals. Accurately gathering knowledge is a core component of many domains [START_REF] Davis | Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review[END_REF].

The results of this experiment supported our first hypothesis that participants given the HMD would be able to recall more information about the given scenario than those given the desktop display. It must be noted, however, that when asking participants to recall information, it is possible that we may have been exposed to issues regarding cognitive load. Cognitive load refers to the total amount of mental effort being used with regards to working memory. For example, if a task was to greatly tax working memory capability, it would be considered high cognitive load. It is possible that while participants were experiencing higher cognitive load at more complex parts of the task, they may have become less focused on accurately articulating their thoughts verbally. This may have inadvertently lowered the recall scores of participants in both treatment groups, as they may not have been talking as much during the more complex parts of their description. This is somewhat supported, as participants in our previous study did tend to recall more information when asked to write down their knowledge of the task in a more structured manner. As both treatment groups were asked to describe the same task, however, the two treatments should have required similar levels of cognitive load. For this reason, we do not believe this finding invalidates the observed results. While the observed recall findings do suggest that the immersive system of virtual reality may be able to assist with allowing users to better recall information, we have also explored several other items of interest to better explain the observed result.

In addition to looking at memory performance, we have also explored behavioural differences between the two groups to determine whether any differences in behaviour may have also potentially affected recall performance. When examining how the two groups traversed the environment, we found that participants given the HMD explored more of the environment than those given the desktop display (supported by a large effect size). When examining how the two groups viewed the environment, we found that participants given the HMD adjusted their view more often than those given the desktop display. As participants given the HMD both traversed more of the environment and viewed more of the environment, we believe these findings support H2(b) and indicate that the participants given the HMD were more exploratory in their role-play approach. While it is likely this difference in view change was due to the ease at which the participants given the HMD were able to adjust their view to examine the environment, we do not believe that this diminishes the significance of the finding. The ability to easily change view is a fundamental affordance provided by the HMD. While we cannot say, with any degree of certainty, that these differences did result in better recall performance, these findings do seem to be consistent with existing literature which suggests that providing participants with adequate levels of context is important to best achieve a situated cognition effect [START_REF] Brown | Situated cognition and the culture of learning[END_REF].

No statistically significant difference in usability scores was found between those given the HMD and those given the desktop display. While this result is in no way conclusive, this suggests that the possible link between time taken and recall performance may not be as significant as our previous experiment suggested. This is an positive result, as one of the main reasons for running this study was to eliminate, or at least partially mitigate, the difference in usability between the two groups. Ensuring similar usability between the two treatments reduces the chance that usability differences may have affected any of the other results presented in this study.

In addition to usability, we also looked to explore any potential difference in time taken between the two groups, as differences in time taken presented a potential confound in our prior studies. In this experiment, we found no statistically significant difference in time taken between the HMD and desktop display conditions.

Finally, participants given the HMD reported higher subjective presence scores than those given the desktop display. These findings are consistent with prior research [START_REF] Schumie | Research on presence in VR: a survey[END_REF], which discussed a possible link between presence and memory performance within virtual worlds. Due to the increased levels of presence experienced by those given the HMD, we conjecture that the higher presence provided by the immersive properties of the HMD interface may be responsible for the improved recall ability. Further work will be required to better explore this phenomenon and adequately identify the exact mechanism, or mechanisms, facilitating better recall while immersed within virtual reality.

Synthesis from Overall Results

In this section, we will synthesise the findings discussed above with the data gathered in our prior studies. By doing this, we aim to identify patterns across different studies which may provide further insight into the overall approach. Specifically, we will be looking at the outcomes of this study and two prior studies to identify potential patterns and discuss possible reasons for differences in results across each of these studies. To assist with this, Table 1 provides a brief overview of the three experiments as well as a brief summary of the pertinent results. As we will be discussing multiple studies in this section, we will refer to the studies as they are numbered in Table 1, referring to the initial study we conducted as Study One, our prior study which also investigated recall differences between HMDs and desktop displays as Study Two and the study which was presented in this paper as Study Three.

Study One provided the initial motivation for the approach which was subsequently explored in Study Two and Study Three. Study One provided evidence that suggested the context provided by virtual worlds may assist in improving recall ability. Due to the differences in experiment design between the first study and the other two studies (e.g. differences in the virtual world visuals), we will not be making direct comparisons between the results of the first study and the results in the other two studies. We did feel that including this study in Table 1 was warranted, however, as it provided the initial results which assisted in designing the two subsequent studies. As both Study Two and Study Three had many similarities, however, we will be primarily comparing the results of these two studies and discussing potential causes for any differences in observed results.

In both Study Two and Study Three, we found that participants given the HMD were able to recall more information about the airport boarding task than those given the desktop display. This provides further evidence that the immersive qualities of the HMD were able to improve memory recall capabilities of participants.

Furthermore, in both Study Two and Study Three we found that participants given the HMD were also more exploratory in their approach, choosing to both traverse more of the environment and look at more of the environment. In Study Two, however, we found significant differences in the traversal patterns of the two groups. This was not the case in Study Three, whereby both groups traversed all areas of the airport, despite the HMD group traversing more on [START_REF] Harman | Augmenting process elicitation with visual priming: an empirical exploration of user behaviour and modelling outcomes[END_REF] This first study compared the recall ability of participants provided with a virtual world (viewed on a desktop display) with participants provided with a 2D process description environment. This study found:

-Participants given the virtual world recalled more information.

-Participants given the virtual world completed the task faster.

Study 2: HMD Compared with Desktop Display (Interactions) [START_REF] Harman | Model as you do: Engaging an S-BPM vendor on process modelling in 3D virtual worlds[END_REF] This second study compared the recall ability of participants provided with a virtual world viewed on either an HMD or a desktop display (monitor). Participants described information by interacting with objects and writing descriptions. This study found:

-Participants given the HMD recalled more information.

-Participants given the HMD viewed and traversed more of the environment.

-Participants given the desktop display completed the task faster.

-Participants given the HMD reported lower subjective usability scores.

-Participants given the HMD reported lower subjective presence scores. The third study also compared the recall ability of participants provided with a virtual world viewed on either an HMD or a desktop display (monitor). Unlike in the second study, however, the participants could not interact with objects in the virtual world. Instead, users dictated the actions they would usually perform with each of the items or people as they traversed the environment. This study found:

-Participants given the HMD recalled more information.

-Participants given the HMD viewed and traversed more of the environment.

-No significant difference was found in time taken between the two groups.

-No significant difference was found in usability between the two groups.

-Participants given the HMD reported higher subjective presence scores.

average. Furthermore, we found that both treatment groups traversed more of the environment than in Study Two. This may be due to the removal of the user interface, which forced users to stop moving and enter information into the system intermittently.

In Study Three, we found no statistically significant differences in usability between the HMD and desktop display conditions. While not conclusive, this is a positive result, as the differences in usability identified previously in Study Two put many of our observed results into question. Issues with usability can manifest in a variety of ways and may have affected many, if not all of our results in some way. By simplifying the experimental setup and user requirements for this experiment, we have greatly mitigated any potential confounds posed by usability.

In Study Three, we also found no statistically significant difference in time taken between the HMD and desktop display conditions. Once again, this result is contrary to what we found previously in Study Two. This difference in time taken in our prior study was problematic, as we were unable to dismiss the possibility that the observed recall differences were not due to the differences in time viewing the given virtual world stimuli. This meant that it may not have been the immersive qualities of virtual reality producing differences, as we discussed in our questions and hypothesis construction. In our prior experiment, we conjectured that a major reason for the difference in time taken between the two groups was due to the significantly lower usability scores reported by those given the HMD. If the participants given the HMD were finding the virtual world harder to use, it may mean that they would require longer to complete the task. This suggests that the observed difference in recall ability between the two treatment groups was likely a direct result of the way the two groups viewed the environment, rather than simply the time they spent viewing the virtual world stimuli.

Finally, Study Three found that participants given the HMD reported higher subjective presence scores than those given the desktop display. In Study Two, our presence findings were inconclusive. The subjective presence questionnaire suggested people given the HMD experienced lower levels of presence, while the semi-structured interviews suggested that they were experiencing higher levels of presence. After reflecting on the usage of the presence survey, however, we believe that the task we provided to participants in Study Two was not well suited to this particular questionnaire. After reviewing many of the papers which use this particular survey, the tasks given to participants tended to be very simple and contained a very unobtrusive user interface. In Study Two, however, the participants had to use a variety of menus and object interactions to describe their desired actions. We believe that these menus may have inadvertently affected the user presence scores on the chosen survey. Despite this, however, we have no way of knowing for certain which treatment group experienced greater levels of presence in Study Two. In Study Three, however, we have removed all menus from the task and the HMD subjective presence scores are now also significantly higher than the desktop display scores.

We believe that the study described in this paper has meaningfully clarified and explored the issues identified in our prior studies. Furthermore, this work has provided further rigor to our prior results and provided further evidence of a link between the added immersion provided by virtual reality and improved recall performance.

Design Insights

As we have now constructed multiple virtual worlds with varying interfaces, we believe it is prudent to discuss some of the insights we have gathered during the development, testing and refinement of these environments. While there are numerous design decisions made when looking to develop a virtual environment for eliciting information, our work in this area has generated particularly beneficial insights regarding how the virtual environments should be constructed and how the information should be elicited from the user. Despite significant advancements to the tools and resources available for constructing these environments, the time required to accurately reconstruct environments manually is still greatly prohibitive to the overall approach. The environment used for this experiment was constructed in a matter of hours, but further time may be required for more complex environments, especially if objects within the environment needed to be created. Unless the virtual world reconstruction had already been developed for other purposes (e.g. training simulations), it would likely still be easier to conduct real-world role-plays, rather than manually attempt to reconstruct these environments. These environments do not manually need to be constructed, however, and may instead be generated via 3D scanning techniques (e.g. from a Matterport depth camera [START_REF] Matterport | Immersive Spaces for Real-world Applications -Matterport[END_REF]). In our experiments, we chose not to use this approach as we did not want user experience within the real-world airport we chose to scan to confound our results. We have worked with these scanned environments, however, and we believe that they provide a sufficient level of detail for this particular approach, while greatly reducing the time required to generate these environments.

After working with both a structured approach (where participants manually entered their descriptions of individual tasks) and an unstructured approach (where participants dictated their descriptions of tasks), we also believe that providing some structure to the way in which participants provide their descriptions of each task is beneficial. In the experiment described in this paper, we believe that there may have been times where participants were role-playing the tasks they wished to complete within the environment, but forgot that they also needed to also describe their actions aloud. While this may have also occurred in our previous experiments, we do not believe it was so pronounced. This may be because participants given the structured approach believed they were constructing a list of required tasks for later review, while those dictating their actions believed they were describing their actions forthrightly. This is somewhat supported as many participants which described their actions using the structured approach refined their descriptions over time, while those dictating their descriptions were much less likely to make revisions. For these reasons, we believe that a structured approach is preferable when attempting to elicit information. This structure, however, may not necessarily have to be achieved by having participants manually enter descriptions. For example, this may instead be achieved by having the system only record while the user holds a button down. When the user then releases the button, the system then stores the description and allow the user to review and modify all of their prior recordings. Specifically, we believe that the key difference between these two approaches is the ease in which participants can review and revise their previous descriptions.

While we believe that these two areas are important when looking to develop a virtual world for elicitation, considerable future work is required to adequately explore the many design challenges associated with constructing a virtual world for this purpose.

Conclusion and Future Work

In this study, we have continued to explore the potential of using a virtual world role-play approach in order to assist in effectively eliciting knowledge. This approach was motivated by the theory of situated cognition, which postulates that by situating a user within a specific context, it becomes easier for them to recall related information. This study aimed to solidify the results of prior studies by exploring potential confounds related to usability, presence and time taken.

The results from this study indicated that participants using the HMD both described more tasks and spoke a larger number of words than those given the desktop display. When comparing the behaviour of the two groups, we also found that participants given the HMD also tended to traverse more of the environment and modify their view more often (supported by large effect sizes). Unlike in our prior study, no statistically significant difference in time taken or usability was found between the two treatment groups. In addition to this, the HMD condition reported higher presence than the desktop display condition. These results supported our prior conjecture that the interaction mechanisms presented by the virtual world in our previous experiment may have inadvertently affected the observed results.

In this experiment, however, we were measuring performance based on the dictation provided by participants. This does not necessarily adequately align with the actual recall ability of participants. There is a possibility that the areas of the task which would normally require high cognitive load may have resulted in participants actively describing less. Future work will be required to determine the effect that varying cognitive load may have on the ability of participants to adequately describe their knowledge. Furthermore, as both this study and prior studies used the same airport scenario, further research is required to adequately explore whether the observed results may generalise into other complex spatial scenarios, such as warehouse inventory management or hospital medical processes.

Fig. 1 .

 1 Fig. 1. Screen captures of the developed virtual world airport.

Study 3 :

 3 HMD Compared with Desktop Display (No Interactions)

Table 1 .

 1 Description and results summary for all related studies.

	Study 1: No Context Compared with Virtual World Context


K P Sachith

Aiswarya Gopal

Alexander Muir
email: alexander.muir@ammachilabs.org

Rao R Bhavani
email: bhavani@amrita.edu

Contextualizing ICT based Vocational Education for Rural Communities: Addressing Ethnographic Issues and Assessing Design Principles

Keywords:

Recently, combining Information and Communication Technologies (ICT) with Technical Vocational Education and Training (TVET) for a low literate population is gaining interest, as this can lead to more effective socio-economic development. This strategy can more easily provide employment and bring community wide change because of the improved quality and relevance of education material. Although TVET providers are present throughout India that uses some ICT, challenges remain for prospective students including illiteracy, language, resource limits and gender boundaries. Providing TVET that is accessible to low-literate people in rural village communities requires a shift in the design of ICT so that it is universally useable, even for communities like tribal India that has a largely oral culture. In this article, we detail the design and development of an ICT driven TVET model for a mostly illiterate audience in rural India and measure its efficacy. Through our ethnographic and usability study with 60 low-literate oral and novice village users, we present the issues faced and the solutions we incorporated into our new model. The results show that users performed better in the vocational course units with the solutions incorporated.

Introduction

Today, Information and Communication Technology (ICT) is becoming increasingly available to people around the globe, especially in developing countries. Smaller devices such as tablets and smart phones are available for increasingly cheaper rates, and are reaching all groups of people. There is increasing interest by researchers and NGOs in applying computers, mobile phones, and tablets for serving economically poor populations in the field of Information and Communication Technology for Development (ICT4D).Technology has proven to be an effective tool in alleviating poverty, in areas such as education, networking, and employment.

Vocational training and skill development are connected to both economic productivity and social wellbeing [START_REF] Work | ILO Asia-Pacific Working Paper Series Subregional Office for South Asia , New Delhi Employment Challenge and Strategies in India An Assessment in the Framework of ILO ' s Global Employment Agenda[END_REF].TVET helps people to gain employment, especially those with limited formal education [START_REF] Bhavani | Vocational education technology: rural India[END_REF]. However, it is difficult to make TVET readily accessible to economically and educationally challenged communities, at first simply for its novelty, but more importantly because most technology has been designed for formally education, literate, and urban populations [START_REF] Coley | Ensuring Training-Transfer and Personal Performance after TVET : A Strategy Within the Rural , Indian Context for Post-Training Engagement of TVET Students for Long-term Success[END_REF]. ICT driven TVET is a promising solution to accelerate learning for people with low literacy, as compared to conventional models because it is more easily adapted to multimedia instruction, but should be designed with the end user in mind. ICT elements like multimedia textbooks were found to be advantageous in reducing average learning time and in enhancing understanding of a subject matter when presented properly [START_REF] Ng | Multimedia textbook for virtual education environment[END_REF].

The User Interface (UI) of the ICT devices for TVET, need to be designed with the level of literacy of the target users in mind. The design process needs to be re-thought for the ICT4D context, where many of the basic assumptions that underpin the methods may not always hold true for low literate community. For example, many illiterate users are unfamiliar with the concepts of lists and the hierarchy used in modern UIs [START_REF] Medhi | A comparison of list vs. hierarchical UIs on mobile phones for non-literate users[END_REF].This paper shall discuss orality and its roots, as well as abstract versus operational thinking and its implications for ICT4D [START_REF] Price | Multimedia technologies and solutions for educational applications: Opportunities, trends and challenges[END_REF].

This paper will present the observations and results of a series of usability studies on low-literate, rural audiences using two units from a vocational training course for toilet construction in five villages in India. The tests were conducted to measure the efficacy of the package which included "pre-training" activity, contextual videos, and a supporting game module. Five tests1 were conducted at different villages across the country from 2016 to 2017, with 60 illiterate and semiliterate participants. In the consecutive tests, problems were observed and analyzed at two different levels: user and the system level. Some solutions for these problems were included in the next trial of the test. The study discovered various issues related with the package we tested, and refinements were made iteratively on the instructional video, and design changes were incorporated into the UI of the game module. The paper also presents a pretraining module as a solution that empowers low literate rural audiences to more easily take up ICT for using vocational training. The module enables low literate users to physically use the devices, trains the user to listen to and answer questions, and introduces and familiarizes the user with the UIs and its components from the start.

Related Works

Introducing ICT into TVET has accelerated learning in populations with low-or semi-literacy. Bhavani, et al. (2013) has demonstrated that using ICT in vocational education for rural populations is more flexible, accessible, and provides a high amount of scalability [START_REF] Bhavani | Vocational education technology: rural India[END_REF]. From the same study, it has been observed and analyzed that there are advantages to using interactive multimedia textbooks over standard formal school textbooks. ICT becomes a powerful medium for providing TVET, especially for the rural sector, when compared with the traditional classroom setup designed for urban areas.

The target audience for ICT4D includes illiterate or semi-literate populations. For this reason, ICT4D researchers have proposed doing away with text-based instruction, and have instead recommended non-textual designs [START_REF] Medhi | Full-context videos for first-time, non-literate pc users[END_REF]. In this section, we shall discuss the various modalities available or introduced to overcome the issues faced by low-literate or semi-literate users, especially understanding and using abstract symbols and following navigation procedures. UIs that use voice, graphics, and video have been proposed to better meet the requirements of the target population. Audio-output as a means of instruction (as opposed to text) has become primary topic for research, as it is a more natural means of expression well suited to learners who may struggle with literacy (ibid). Multimodal interfaces such as the VideoKhethi, introduced by Cuendet et al, a mobile system, helps farmers to find and watch agricultural extension videos in their own language and dialect, have been developed for use on smart phones with graphics and touch interfaces including speech recognition. This was designed to help improve agricultural practices among rural groups [START_REF] Cuendet | VideoKheti: making video content accessible to low-literate and novice users[END_REF]. Exploration of touch gestures to enable a better and natural experience for first-time; low-literate users have been conducted. Research has demonstrated that soft-key mapping on devices with keypads is more challenging for low-literate users [START_REF] Medhi | Designing Mobile Interfaces for Novice and Low-Literacy Users[END_REF]. According to the same study, ICT devices, like tablets, exploit the advantages of touch gestures, and needs no or comparatively less training for first time users, when compared with text driven devices like laptops or mini-computers.

Oral users are not necessarily acquainted with the system of abstract learning from books or other formal systems. Rather, learning often happens in the original environment, in concrete situations, and through practical experience. This means that learning and teaching happens in an informal, yet very immersive way where oral and visual transmission of knowledge is the primary mode of delivery. With this in mind, it is important to properly characterize the target audience. Sherwani, et. Al tries to better understand ICT4D users in terms of their thinking, communication and culture [START_REF] Sherwani | Orality-Grounded HCID: Understanding the Oral User[END_REF].In their work, the illiterate community is not considered as less privileged, but referred as "oral users." Orality talks about how people think, communicate, and learn in a culture where writing has not become prevalent. Orality theory claims that writing has so fundamentally transformed consciousness in literate cultures that we-literate researchers-are unable to grasp how oral cultures and people function. When one develops a user centered design for oral users, consideration should be given to cultural practices of community knowledge building and transmission which includes: (a) organization and transmission of information, (b) learning information, and (c) remembering information.

Our Work

Our team is a South India-based organization, concentrating on developing technology enhanced vocational and skill development courses for empowering women, especially in villages and rural areas across the country [START_REF] Coley | Ensuring Training-Transfer and Personal Performance after TVET : A Strategy Within the Rural , Indian Context for Post-Training Engagement of TVET Students for Long-term Success[END_REF]. We provide vocational courses using ICT devices, such as tablets and laptops, reinforced by expert trainers and post-training support. The educational model we use is a virtual media enhanced vocational course, which has been developed end to end by the team at AMMACHI Labs. The vocational training model follows a multimodal learning approach in which the user learns through verbal and nonverbal (audio, visual and textual) representations of the material. In order to test the efficacy of our approach, we took two units from our Rural Toilet Builder vocational course, focusing on training rural women in India to build toilets for their community.

Target Community

Usability tests were conducted at 5 different villages across the country from 2016 to 2017, with illiterate and semiliterate women participants (see Table 1). All of the women were from families that are financially supported by men and/or women working for daily wages, and in unskilled, manual work. Over 50% of our participants had little or no literacy, and less than 15% could read complex sentences. To test the level of literacy, participants were given a sheet of paper with some sentences on it in their mother tongue, and requested to read out what they could. As for the participants' familiarity with technology, mobile phones are the most common technology that these women may have seen or handled. None of them had previous experience in using tablet or a computer. The primary language for communication was Kannada, Tamil, and Hindi (depending on the location). Low-literate users think, communicate, and learn in a culture where writing has not become internalized. In order to make them eligible for a TVET model, the user were first introduced to the material through a physical training where they used physical gestures to answer questions. The training was taken to the next level by asking question through images and answering through picking the correct image. After this the same set of questions were asked through tablet devices and answered by touch-gestures, slowly equipping them to use technology-enhanced models for learning.

Experimental Setup -Application

The teaching application was driven by an animated character that is similar to a "traditional" village lady. This means she was wearing clothing similar to what our target population wears (sari), has a similar skin, hair, and eye color, wore common accessories (no fancy gold ornaments, etc.), and behaved in ways that were familiar to them. The language of instruction was also their mother tongue. A character narrator was introduced to the users so that they could more easily connect to the application in a grounded way, rather listening to instructions through a disembodied voice. The interface was designed with minimal text elements, and supported by graphics, animation and audio intelligible to a person who cannot read [START_REF] Manly | Literacy and memory decline among ethnically diverse elders[END_REF].

Classroom and Interactive Games

The classroom provides the theoretical knowledge of the vocational trade the student is learning, and the necessary concepts for the eventual occupation they are training for. Each classroom contains multimedia content: videos displaying recorded demonstrations and explanations by experts in the field, accompanied by animations, images, and some text. It gives theoretical knowledge on a specific component in the course through video footage, accompanied by text, audio, 2D and 3D illustrations and animations. In the toilet building course, the video included real women from a village in Tamil Nadu recorded in handling the different elements of construction. This is an important part of our training videos, as it boosts the confidence of the target students to see people they can identify with doing these new, sometimes difficult skills. The course also includes simple, interactive games in between classroom chapters that use 2D illustrations and animations to create interactive exercises where users navigate through practical, real-world scenarios and apply the theoretical knowledge gained through the classroom. This module requires the user to answer questions, played by audio, on the content provided in the instructional video. The response is given through a touch gesture, and is succeeded by a feedback screen which shows the correct response. This exercise is repeated to reemphasize the concepts to the user. The user interface does not provide negative reinforcement for wrong responses from the user, in order to avoid reducing the student's enthusiasm; rather, it guides the user to the correct answer in case they make an error.

Practical assessment

Following the classroom and intervening games, the user's understanding is assessed. The units we selected for the usability tests required both cognitive and operational skills, and the assessments reflect both. The student was assessed in an environment that allowed them to use the skills they were learning, and which was conducive for assessing cognitive skills. Cognitive skills were assessed with oral questions by the field staff on concepts taught in the training. Operational skills were assessed by asking the student to physically demonstrate the proper use of the tools explained in the classroom in the real world, where they are observed using the tools.

Experimental Setup -System Setup

The vocational training courses, and our usability tests, require a classroom set up to deliver the educational content to the participant. In terms of location, both urban areas and rural communities in India frequently lack both steady electricity and quiet, undisturbed spaces for maintaining a classroom environment. Rural areas are frequently difficult to reach and face a lack of facilities including secure rooms and poor internet connectivity. Our TVET design already addresses some of the environmental and social challenges that working in a rural, remote area bring, such as limited electricity, etc., through our innovative deployment model discussed at length in our other publications [START_REF] Coley | Ensuring Training-Transfer and Personal Performance after TVET : A Strategy Within the Rural , Indian Context for Post-Training Engagement of TVET Students for Long-term Success[END_REF]. However, even after building local trust, providing training in low-infrastructure environments and encouraging enthusiasm from users, we still faced the issue of the users not fully understanding the original UI.

Iterations on Design Principles, Ethnographic Issues and Solutions incorporated

The first and second usability study came up with issues which we have classified as:

Educational

The participants were mostly illiterate, first-time users of technology. The teaching methodology and the UI which was initially used were not effective enough with the target population. Many students failed to understand the concepts in the training or even how to use the UI. The primary challenge faced by participants was to think and consider concepts in abstract terms. This could be due a lack of formal schooling and exposure to abstract ideas in a formal setting. This is reflected in popular learning and development theories which argue that abstract thinking is learned mostly through formal, modern education models [START_REF] Strauss | Theories of cognitive development and their implications for curriculum development and teaching[END_REF].For example, user was able to understand the usage of the tool, i.e., a plum bob to check the level of the surface. But their knowledge was only restricted to checking the level of the bricks which was introduced to them, not to other materials. The users were also not familiar with a model of education where they needed to pay attention to a video, process the knowledge, and then apply it in the successive sections. Even for games, which were slightly easier to grasp, the participants struggled to understand that the games were meant to test concepts from the associated video classroom. The participants typically answered from their existing knowledge and life experience, even if it contradicted what was just taught in the classroom.

Interface Related Issues

Classroom

In the original classroom, the animations and the video footage used abstract symbols like a tick mark, crosses, and concentric circles to indicate correct or wrong procedures and to bring attention to a particular portion in the visual. This was initially not understood by the users, as it was the first time they were introduced to illustrations and animations using these abstract symbols. The video was narrated in the regional language, but followed a more formal style that did not include some the regional terms or slang they were comfortable with. This caused the participants to misunderstand some topics discussed in the video. Following the initial user tests, the animations in the video were redesigned to be more contextual, and less abstract for better understanding in the successive tests. The operational skills were re-emphasized by repeating the instruction on proper usage of tool in the video. Some of the important changes in the iteration we made included changing the camera angles for the user so that they felt they were a part of the process, using close and tight shots for validation scenarios, using expressions and hand gestures to convey correct and wrong procedures, and included some consequences of doing a step in the wrong manner (see Snapshot 1 and 2).

User Interface

The original interface tested used buttons and abstract symbols to operate and navigate through the application. The concept of "navigation" was new to them. The application was navigated using buttons which used minimum text in the regional language. Abstract symbols such as ticks or cross marks were used as icons to give feedback to the user on a particular question. The game modules in this initial test introduced help videos in the introduction screen of each unit. This did not help the users much because this population is not familiar with the concept of "help videos". The user played the games with touch gestures, and each answer was followed by a feedback screen which showed the correct response. This was repeated to re-emphasize the concept to the user. A correct response was appreciated by the animation character by gestures to motivate and increase the engagement of the user. In the paper prototype session, we introduced the abstract symbols that are used in the course through images on paper. Large "Yes" or "No" symbols (tick and cross) and the user interface were first introduced on paper. The participants were asked to touch the icon on the paper as their response to the set of questions. The same set of questions which was asked in the first and the second phase of the pre-training were reproduced on the tablets. While the participants seemed to grasp the abstract symbols and progression of the activity, there were still some issues in using the technology itself. Specifically, it was difficult for the participant to correctly understand the touch and the drag gesture, often applying too much pressure, not understanding the consequence of using too many fingers, or not touching for the appropriate length of time.

Quantitative and Qualitative Results

Rate of successfully answering questions Figure 3a and 3b

The above graphs (figures 3a and 3b) show the results of Study 2 and Study 5, which are two comparable user studies. Study 2 was the first time the challenges by students were fully documented, and by Study 5 all of the improvements to the UI, including the pre-training, were tested. The results from studies 3 and 4 were helpful and provided more insight to us on what we could improve, however results are not usable for comparison cases in this analysis. The results are shown in terms of the participant's literacy levels: For the non-literate group, there is a dramatic change from 0% of questions being successfully answered, to 80% successfully answered. 30% of the questions were answered without help from the moderator. For the lowliteracy and literate groups, there is also a notable shift. For low literacy participants, the percentage of questions answered without any help goes from 25% to 63%. During later studies the participants were noticeably more confident and less reluctant to use the system. These results are encouraging for future studies that can assess the learning of populations in other areas and in other contexts. The limitations of this study, including a limited population, on testing a single course (toilet building), and short time duration could be addressed in future work.

Conclusions

The results of comparing Study 2 and Study 5 (our two comparison cases) show a strong increase in the ease with which the non-literate students could engage with the tablet-based training. Contextualizing the content and introducing pre-training module helps students succeed because it gives them practice in using the interface to answer quiz-style questions. This approach demonstrates an initially effective strategy to address the challenges faced by illiterate and first time technology users in accessing vocational training. To date and using this model, we have successfully trained over 250 women who have collectively built over 250 toilets in 21 states across the country 2 .

Snapshot 1&2 :Photo 1 :

 1&21 Photo 1: Physical tick gesture Photo 2: Paper prototype

Table 1 :

 1 Overview of the series of user studies

	Study number, Loca-
	tion and Participant	Study Focus
	count
	1. Karnataka: 10	Get the toilet building training status of the program happening in
		Byse
	2. Tamil Nadu: 12	Educational Usability of the Plum Bob and Site Selection videos
		and Interactive quizzes. Need for Pretraining identified.
	3. Chhattisgarh: 8	Educational Usability of the Plum Bob and Site Selection videos
		and interactive quizzes.
		Pilot test of Pretraining.
	4. Andhra Pradesh: 8	Educational Usability of the Plum Bob and Site Selection videos
		and interactive quizzes. Pilot test of Pretraining.
	5. Tamil Nadu: 10	Educational Usability of the Plum Bob and Site Selection videos
		and interactive quizzes. Evaluation of Pretraining

The first 5 tests were conducted from date to date, by staff at [Anonymous], in the following locations: Byse, Karnataka;

tests at Ettimadai, Coimbatore, Tamil Nadu; Kondalgavu, Chattisgarh; Gudipadichervu, Andhra.