Ard W Lazonder
email: lazonder@edte.utwente.nl

Principles for Designing Web Searching Instruction

Keywords: learning, instructional design, information literacy, World Wide Web

Web searching is a timely topic which importance is recognized by researchers, educators and instructional designers. This paper aims to guide these practitioners in developing instructional materials for learning to search the Web. It does so by articulating ten design principles that attend to the content and presentation of Web searching instruction. These principles convey a mixture of insights gleaned from instructional theory, empirical research, and many hours of classroom experience. Together, these design recommendations elucidate the key characteristics of effective Web searching instruction, explaining not only what the instructional materials look like, but also why they look the way they do.

Principles for Designing Web Searching Instruction

Introduction

Web searching closely resembles the work of a detective. To trace relevant information, one has to ask the right questions, consult proper sources of information and creatively combine search outcomes. Unfortunately, few students can measure up to Sherlock Holmes in this respect. According to a recent survey, nearly half of the Dutch high school freshmen consider themselves incapable of operating Web browsers and search engines (Ten [START_REF] Brummelhuis | ICT monitor 1998/1999: Voortgezet onderwijs [ICT monitor 1998/1999[END_REF]. A comparative study further showed that these findings are consistent with the situation in many European countries [START_REF] Pelgrum | Curriculum and pedagogy. In ICT and the emerging paradigm for lifelong learning: A worldwide educational assessment of infrastructure, goals and practice[END_REF].

These studies imply that students should be taught to search for information on the Web. This training need is substantiated by the pervasive use of Internet in modern society. At school, the Web is increasingly being used as an arena for students to obtain information. Furthermore, a growing number of occupations require at least a rudimentary understanding of applications to access and navigate the Web. It is therefore important that educators seek opportunities to improve teenagers' Web searching skills. This in turn raises the question of how to foster these skills through the appropriate use of instructional strategies.

One might expect a good deal of direction on this issue from the research literature. Indeed, quite a few studies on Web searching have been published over the past years. Yet few attempts address the issue of how people learn to search the Web. And the studies that do, often do not give an explicit account of the principles used to design the instructional materials.

Useful guidance might also be gleaned from research on stand-alone information systems such as hypertext databases and online public access catalogues. However, it is unclear whether these insights can be generalized to Web searching. Being a global hypertext structure, the Web suffers from orientation problems, bandwidth problems, and download latencies [START_REF] Barbesi | A psychological investigation of long retrieval times on the World Wide Web[END_REF]. Another difference concerns the lack of indexing conventions. Controlled vocabularies, thesaurus classifications, and human indexing are uncommon for Web search engines [START_REF] Schacter | Children's Internet searching on complex problems: Performance and process analyses[END_REF]. This inconsistency makes Web searching so imprecise that even information specialists find it difficult to retrieve specific information [START_REF] Bruce | Searching for digital libraries in education: Why computers cannot tell the story[END_REF].

Deducing design principles from existing instructional materials would be another, yet equally unfeasible option. Although a multitude of materials designed to train students to become skilled Web searchers is available, few of these publications appear to have been developed from the research on the topic. For example, theorists acknowledge information literacy skills to be the key factor in information seeking, and research has repeatedly shown that students generally have little command of these skills (Elley, 1994;[START_REF] Guthrie | Roles of document structure, cognitive strategy, and awareness in searching for information[END_REF][START_REF] Thomas | Information literacy and information skills instruction[END_REF]. Yet commercially available Web searching manuals seem to be unaware of these training needs as they predominantly focus on operating Web browsers and search engines, leaving the information literacy skills largely unaddressed.

This paper aims to compensate for this seeming lack of design recommendations. It articulates ten design principles that attend to the content and presentation of Web searching instruction. These principles convey a mixture of insights gleaned from instructional theory, empirical research, and many hours of classroom experience. The principles are primarily intended for designing paper Web searching tutorials for youths between the ages of approximately 10 and 14 (hereafter, students). But their usefulness is restricted neither to this audience nor to the design of printed training materials. Some principles may also facilitate the design of other kinds of instructional support (e.g., online help); other principles may generalize to different user groups. These extensions are not addressed in this paper, but readers are encouraged to consider the design principles beyond the context in which they are presented.

The sections below introduce the design principles, explain their rationale, and illustrate their application in Web searching tutorials. It is worth noting that the principles do not offer a prescriptive blueprint designers can follow mechanically. Rather, they constitute a framework of design recommendations that support designers in their thinking about Web searching instruction.

Principles for selecting instructional content

Web searching calls upon various types of expertise. Web users should have a basic facility with the browser and search engine, possess knowledge of the topic being searched, and be capable of regulating their own search behavior [START_REF] Carroll | Expert Internet information access[END_REF][START_REF] Chen | Internet browsing and searching: Use evaluations of category map and concept space techniques[END_REF][START_REF] Marchionini | Information seeking in electronic environments[END_REF]. While it may seem a contradiction in terms, inexperienced Web users can be quite knowledgeable in some of these areas. To illustrate, a vast majority of the students is familiar with common software programs such as word processors, games, and graphical editors. The core skills involved in operating these programs are typical for contemporary point-and-click applications, including Web browsers and search engines. As a result, students without prior Web-searching experience may already master the elementary skills to access and navigate the Web.

Web searching instruction should always take the learners' prior knowledge into account. It should focus on the expertise learners need to acquire, and adapt to or even exploit the knowledge already in place. The principles in this section provide general directions for selecting the content of Web searching instruction. As audiences and circumstances are widely divergent, the principles suggest rather than impose the subject matter. Designers can, and in fact should, tailor each recommendation to their personal training situation.

Principle 1: Focus on locating Web sites

Web searching comes down to locating a Web site and, subsequently, locating information within that site. Site location involves either performing a keyword search, entering a URL, or browsing subject categories. The order in which these strategies appear mirrors the students' preferred choice of strategies. Students show a strong tendency to search for keywords (e.g., [START_REF] Fidel | A visit to the information mall: Web searching behavior of high school students[END_REF][START_REF] Wallace | On-line search in the science curriculum: Benefits and possibilities[END_REF]. In one of our studies, students used this strategy in nearly 70% of the occasions. Entering a URL occurred in one out of four attempts to locate a site, whereas the browsing strategy was used in only 4% of the trials [START_REF] Lazonder | Using minimalism to support the development of self-regulatory knowledge and skills[END_REF].

Once a potentially relevant site has been retrieved, the target information can be located through browsing and searching using the site's built-in search engine. As with site location, students strongly prefer searching to browsing. Yet they are often required to engage in browsing because not every site is equipped with a local search engine.

Prior experience with the Web yields superior search performance on the first part of the search process [START_REF] Lazonder | Exploring novice users' training needs in searching information on the World Wide Web[END_REF][START_REF] Lazonder | Differences between novice and experienced users in searching information on the World Wide Web[END_REF]. Experienced Web searchers are more proficient at inferring and entering URLs. They are also more skilled at operating browsers and performing keyword searches compared to inexperienced users. However, no gains of Web experience are apparent in the search for information within sites. This may be due to the fact that locating information requires a substantial amount of browsing -a data-driven search strategy that calls upon few analytical search strategies. Together, these findings argue that Web searching instruction should concentrate on locating Web sites, and only touch on locating information within sites.

Principle 2: Teach advanced system knowledge Web searching instruction need not elaborate on the procedural skills to operate Web browsers and search engines. Most students already have a basic facility with these retrieval tools, and those who don't can easily develop the required skills -even without instructional support [START_REF] Khan | Searching through cyberspace: The effects of link display and link density on information retrieval from hypertext on the World Wide Web[END_REF][START_REF] Lazonder | Exploring novice users' training needs in searching information on the World Wide Web[END_REF][START_REF] Marchionini | Information-seeking strategies of novices using a full-text electronic encyclopedia[END_REF]. However, this precept does not apply to the advanced options of the search engine. According to [START_REF] Bilal | Children's use of the Yahooligans! Web search engine II: Cognitive and physical behaviors on research tasks[END_REF], students generally possess inadequate knowledge of how to use search engines efficiently and effectively. Most students are unaware that the search engine's functionality goes beyond performing a single keyword search. And students that do try to use a search engine's advanced options are bound to make mistakes. Refining a search, using Boolean operators, and searching for multiple keywords are just a few examples of advanced options that are prone to error [START_REF] Lazonder | Exploring novice users' training needs in searching information on the World Wide Web[END_REF][START_REF] Spink | Searching the Web: The public and their queries[END_REF][START_REF] Wallace | On-line search in the science curriculum: Benefits and possibilities[END_REF]. Using these options correctly and functionally should therefore be addressed in Web searching instruction.

Web searching instruction should specify what an advanced option does and how it should be used.

In most cases, the reason why an advanced option has to be used is either self-evident or inherent in the selected search strategy. To illustrate, narrowing an existing search involves changing the search engine's search mode from 'New search' to 'Refine search'. Clearly, this switch needs no further explanation. In other cases the rationale is irrelevant to users because of software conventions. There is, for example, little use in explaining why Boolean operators should be entered the way they do.

Principle 3: Teach self-regulatory skills

The third principle specifies the self-regulatory skills students should acquire to become competent at locating Web sites. These skills were adapted from empirically verified models of how students search electronic environments such as the Web (e.g., [START_REF] Eisenberg | Information problem-solving: The big six skills approach to library and information skills instruction[END_REF][START_REF] Hill | A conceptual framework for understanding information seeking in open-ended information systems[END_REF][START_REF] Marchionini | Information seeking in electronic environments[END_REF][START_REF] Sutcliffe | Towards a cognitive theory of information retrieval[END_REF]. [START_REF] Lazonder | Exploring novice users' training needs in searching information on the World Wide Web[END_REF] used one such model as analytical framework for identifying the training needs of novice Web searchers. This study revealed that Web searching instruction should primarily address self-regulatory skills to monitor search behavior (e.g., check for typos in URLs or keywords, watch the status bar for Web page loading progress, and consult the 'number of hits' system cue) and evaluate search outcomes (e.g., use system cues such as relevance ratings, keywords found, and URLs to assess the merit of Web sites included in the hit list).

Other studies suggest that novice users should also be taught to select search strategies [START_REF] Bilal | Children's use of the Yahooligans! Web search engine II: Cognitive and physical behaviors on research tasks[END_REF][START_REF] Hill | Cognitive strategies and learning from the World Wide Web[END_REF][START_REF] Khan | Searching through cyberspace: The effects of link display and link density on information retrieval from hypertext on the World Wide Web[END_REF]Locatis, 1998, Watson, 1998). Inexperienced Web searchers generally engage in primitive search strategies that are either inefficient or unsuccessful. Our own observations support this stance: most of the students we observed employed rather naive search strategies. When given a search task, students systematically used the keywords from the task description as search term or URL. Obviously, this rather straightforward approach often proved unsuccessful, causing students to conclude that the target site was not available on the Web. Very few students were observed to apply sophisticated search strategies such as combining multiple keywords into a query or searching for keywords that are synonymous with the wording used in the task description.

These conclusions probably generalize to a larger audience. [START_REF] Lazonder | Exploring novice users' training needs in searching information on the World Wide Web[END_REF] study revealed various instances of inefficient search behavior by experienced Web searchers. Similar to novice users, experienced Web searchers hardly used system cues to evaluate search outcomes. Not did they monitor their search behavior on a regular basis. [START_REF] Spink | Searching the Web: The public and their queries[END_REF] analyzed over one million Web queries by over 200,000 users of the Excite search engine. Their findings indicate that most people use few search terms, few modified queries, and rarely use advanced search strategies. Together, these studies suggest that experienced users might benefit from instruction in self-regulatory Web searching skills.

Web searching instruction should inform students on the 'what, how, and why' of a self-regulatory skill [START_REF] Jong | Self-regulation: Regulation of the learning process and learning to regulate-a process view[END_REF][START_REF] Osman | Metacognition research and theory: Analysis and implications for instructional design[END_REF][START_REF] Shunk | Self-regulation and academic learning: Self-efficacy enhancing interventions[END_REF]. The 'what' and 'how' parts are typical of skill training programs, indicating which skill should be applied and how it should be performed. As can be seen from Figure 1, the instructional content of these parts depends on the nature of the user's activity. If the user has to perform a mental action (e.g., formulate a search term, recall the search task, determine the number of search outcomes), the 'what' part contains an elaborate description of that action. The 'how' part details the way in which this action has to be performed. In case the user has to make a decision (e.g., select a search strategy, choose the most relevant Web site), the 'what' part details the choice users have to make and (if appropriate) the alternatives from which they can choose. The 'how' part presents the selection rules users can apply to make a conscious choice. The 'why' part points out the relevance and efficacy of a self-regulatory skill, independent of the user's activity (i.e., action or decision). It informs students on the advantages of performing that skill by highlighting its facilitative effect on the quality of the search outcomes and the efficiency of search behavior.

- ------------------------------Insert Figure 1 about here -------------------------------Principle 4: Arouse and sustain motivation At the risk of stating the obvious, this principle attests that Web searching instruction can only be effective if students are motivated to learn. Clearly, this principle is not new: motivation has long since been acknowledged to play a pivotal role in learning and instruction. Furthermore, one might question whether it is in fact necessary to arouse motivation since students appear to be intrinsically motivated to learn to search the Web (e.g., [START_REF] Bilal | Children's use of the Yahooligans! Web search engine I: Cognitive, physical, and affective behaviors on fact-based search tasks[END_REF][START_REF] Watson | If you don't have it, you can't find it": A close look at students' perceptions using technology[END_REF]. In one of our classroom studies, students completed a motivational questionnaire prior to the instruction. Results indicated that the students' motivation was high: the mean scores ranged from 5.5 to 6.4 on a seven-point scale (Lazonder, 2001a).

There is, however, reason to believe that designers cannot trust blindly on the students' task motivation. In the aforementioned study, students' expectations about the course appeared to be at odds with the actual course content. Most students perceived Web searching as entering keywords and URLs and clicking hyperlinks. And their initial motivation scores showed that these were the topics they were anxious to learn. Unfortunately, their learning preferences were not met by the instruction.

Consistent with the third design principle, the instruction centered on the self-regulatory parts of the search process and paid virtually no attention to the hands-on skills. As a result, students' motivation decreased during the instruction. This would probably not have been the case if the students' motivation were based on the actual course content instead of on their beliefs about what the course would be like.

Having students perform a complicated search task prior to the instruction may arouse proper task motivation. This search task should be so arduous that pervasive use of self-regulatory skills is required for its completion. Among other things, this implies that students should be unable to locate the site via its URL. Furthermore, the target site should not appear on the first three or four pages of the hit list when students search for the most obvious keywords (i.e., the terms used in the task description). The target site should readily appear in the hit list only if students search for multiple keywords. And at least one of these keywords should be a synonym of the terms used in the task description. No requirements apply to the 'locate information' part of the task because these skills are not included in the training needs of new Web users.

The basic idea behind this search task is to confront students with their inability to solve this search problem. Once students recognize they need additional, self-regulatory skills to complete this task, they may become motivated to learn these skills. Evaluating the search task and demonstrating its solution seem vital in this respect. Either the teacher or the instruction manual should elucidate which skills were needed to complete the introductory search task. Students should also be given an advance organizer: a preview that details where these skills are addressed in the instruction.

Once the proper motivation is aroused, it should be sustained. This is especially important in view of the extensive amount of repeated practice required to acquire self-regulatory skills (see principle 8).

Learners typically resent rehearsing the same routine over and over again, causing them to skip most review exercises [START_REF] Carroll | The Nurnberg Funnel: Designing minimalist instruction for practical computer skill[END_REF]. The amount of repeated practice should therefore be well balanced, allowing students to master a skill without loosing interest. This may be achieved by integrating repeated practice in self-regulatory skills into course curricula. Such an integrated approach seems promising, especially since teachers increasingly ask their students to obtain information from the Web. This in turn will maintain students' motivation: their success in other school activities may depend on their proficiency to search the Web.

Principle 5: Do not overestimate the importance of domain expertise

Although some people 'surf the net' just for the sake of it, Web searching is no end in itself.

Rather, it is a means to obtain information on a particular topic. Knowledge of this topic and its underlying task domain is acknowledged to enhance search performance. Users with high levels of domain expertise are, for example, more proficient at identifying the key facets of a search task, using vocabulary of the task domain to formulate queries, and differentiating between relevant and irrelevant search outcomes [START_REF] Marchionini | Information seeking in electronic environments[END_REF]. Research has shown that these skills yield superior performance in terms of efficiency and effectiveness. That is, domain experts need less time to complete search tasks and produce a greater number of correct solutions [START_REF] Hirsh | How do children find information on different types of tasks?: Children's use of the science library catalog[END_REF][START_REF] Mcdonald | Navigation in hyperspace: An evaluation of the effects of navigation tools and subject matter expertise on browsing and information retrieval in hypertext[END_REF][START_REF] Patel | Effectiveness of expert semantic knowledge as a navigational aid within hypertext[END_REF].

There is, however, reason to believe that Web searching instruction need not always cater for differences in domain expertise. Knowledge of the task domain may, for example, be left out of account when teaching a homogeneous group of students to search the Web. In that case, within-group differences in domain expertise are so insignificant that they do not affect search performance (Lazonder, 2001a). Domain expertise is also less important when teaching the basics of Web searching. According to Hsieh-Yee (1998), domain expertise becomes a factor only after a certain amount of search experience had been acquired. Yet this is not to say that designers can completely ignore the task domain. The students' attitude towards the topics being searched has important consequences for the design of practice. Carefully designed training tasks can foster students in practicing distinct Web searching skills. This is especially important during the initial phases of learning when the unanticipated use of advanced search skills can easily disrupt or even obstruct learning. For assigned search tasks to be effective, they should comply with the students' interests and preferences. Assignments that do not satisfy this condition are likely to be abandoned by students. Insight into the topics students are anxious to search for may thus serve as a basis for designing authentic hence motivating practice tasks.

Principles for presenting the instructional content

The second set of principles aims to guide designers in developing materials that convey the instructional content detailed in the previous section. The majority of these principles was adapted from the minimalist approach to instruction (Van der Meij and [START_REF] Van Der Meij | Principles and heuristics for designing minimalist instruction[END_REF] and validated in various studies on initial computer skill training. In all of these studies, minimalist instruction was found to lead to faster training and better learning as compared to other instructional approaches (e.g., [START_REF] Carroll | The Nurnberg Funnel: Designing minimalist instruction for practical computer skill[END_REF][START_REF] Lazonder | The minimal manual: Is less really more?[END_REF][START_REF] Ramsay | Designing minimal computer manuals from scratch[END_REF]. Recently, Lazonder (2001a) showed that the functionality of minimalist instruction also applies to acquiring procedural skills to operate Web browsers and search engines.

Theoretical evidence suggests that minimalism holds promise for the development of selfregulatory skills. That is, the main characteristics of minimalist instruction correspond to the instructional conditions for self-regulatory skill learning. An early attempt to substantiate this theoretical evidence with practical findings failed to produce the anticipated effects (Lazonder, 2001b). However, this study did reveal valuable directions for improving the Web searching instruction. These suggestions are included in the principles presented below.

Principle 6: Teach procedural and self-regulatory skills concurrently

Self-regulatory skills may be taught in two fundamentally different ways. Instructional designers can either embed self-regulatory skills within lessons, or afford the opportunity for learners to learn and practice the skills in an adjunct course, detached from of the procedural skill instruction [START_REF] Osman | Metacognition research and theory: Analysis and implications for instructional design[END_REF]Puntambekar and Du Boulay, 1997;[START_REF] Shin | Promoting students' self-regulation ability: Guidelines for instructional design[END_REF]. Both theoretical and empirical evidence suggests that embedded instruction is the most effective way to develop self-regulatory skills (Bielazcyc et al., 1995;[START_REF] Hattie | Effects of learning skills interventions on student learning[END_REF][START_REF] Osman | Metacognition research and theory: Analysis and implications for instructional design[END_REF].

The joint practice of procedural and self-regulatory skills may provoke additional cognitive load. In fact, the coordinate handling of manual, keyboard and screen already requires considerable mental effort [START_REF] Chandler | Cognitive load while learning to use a computer program[END_REF]. Adding self-regulatory skills could make it even harder for students to allocate their attention to the relevant parts of the learning task. The success of embedded self-regulatory skills instruction thus seems to hinge on its potential to manage the students' cognitive load [START_REF] Osman | Metacognition research and theory: Analysis and implications for instructional design[END_REF].

Timesharing strategy is a potentially fruitful strategy to resolve the difficulty involved in simultaneously performing procedural and self-regulatory skills [START_REF] Perkins | Teaching cognitive and metacognitive strategies[END_REF]. Guided by the notion of parallel processing, timesharing instruction prompts learners to shift attention from performing procedural skills to execute a search, to performing self-regulatory skills to manage search behavior.

Timesharing instruction therefore provides self-regulatory skills instruction on the spot, fully integrated with the procedural skills to operate the search engine. Self-regulatory skills are introduced and explained only if they actually have to be performed. Students are thus prompted to perform a self-regulatory skill every time it is needed. This in turn teaches them the positioning of a given skill in the search process.

Due to this integrated presentation, self-regulatory skills instruction has to be signaled to indicate its distinct nature and to facilitate recognition. In Figure 1, two different icons are used for this purpose: a 'light bulb ' signals the 'what' and 'why' part, a 'hand' indicates the 'how' part.

Principle 7: Teach procedural and self-regulatory skills in context

Web searching instruction should be designed around real tasks. This principle reveals itself especially in the design of practice. Training tasks should include genuine activities that represent the core tasks of a domain. The students' interest in and understanding of these tasks is what incited them to learn to search the Web in the first place. By building the instruction from within these tasks, students' learning preferences are met and, consequently, their motivation to learn is sustained.

The need for contextual instruction is further substantiated by the conditions for self-regulatory skill learning. Research suggests that self-regulatory skills should be taught in context [START_REF] Boekaerts | Self-regulated learning: A new concept embraced by researchers, policy makers, educators, teachers, and students[END_REF][START_REF] Hattie | Effects of learning skills interventions on student learning[END_REF]Puntambekar and Du Boulay, 1997). Self-regulatory skills are always used within a given task domain. Their spontaneous application in that content area seems to depend on the students' conditional knowledge about when and where to use particular skills. Without this knowledge, the self-regulatory skills remain inert, and students may fail to invoke them during task performance. The instruction should therefore associate the self-regulatory skills with the circumstances in which they are applicable. This in turn may increase the students' perceived utility of a self-regulatory skill [START_REF] Shunk | Self-regulation and academic learning: Self-efficacy enhancing interventions[END_REF].

From a learner's perspective, tasks are 'real' when they represent the activities users intend to perform with an application. To illustrate, Dutch high school students predominantly use the Web in language classes to retrieve summaries and book reports, to find background information on authors and literary movements, to explain the meaning of proverbs and sayings and so on. Consequently, the practice tasks we designed for this target audience resembled these activities as much as possible and each task was embedded in an authentic setting (see Figure 2). In the instruction, search tasks were presented in an box at the beginning of each section, thus making it explicitly clear to students which search task they had to perform.

- ------------------------------Insert Figure 2 about here -------------------------------Principle 8: Provide ample opportunities for practice New software users show a strong desire to act. They want to learn the program by working with it, not by reading about it. Web searching instruction should comply with this request in various ways, depending on the type of skill. For procedural skills, it should provide students with an immediate opportunity to act. It should emphasize the operation of the program, leaving out nearly all information that does not directly relate to 'doing things'. To illustrate, the Web searching instruction we designed had no preface, long-winded introduction on the origin and current proliferation of the Internet, or general description of how data are digitally stored and transferred. Rather, students received their first instructions to act on page 2. The initial stress on learning by doing was maintained throughout the instruction, thus providing students with ample opportunities for practice.

Practice in procedural skills can further be enhanced by encouraging students to explore designated parts of the program. Action prompts are a fruitful means to invite students to discover new features of the browser and search engine. For example, detailed instructions on how to open subsequent result pages can be replaced by the prompt "Click on the digits at the bottom of the screen and see what happens", thus inviting students to infer the meaning of these links on their own. Explorations can also be encouraged in a "Do it yourself" chapter. We included such a chapter in one of our Web searching manuals. This chapter invited students to locate Web sites, but did not specify the action steps needed for site location. Nor did it inform students on the information they should locate within sites. Students were merely given a list of URLs and keywords they could use to rehearse the procedural skills addressed in the preceding chapters.

An action-oriented approach should be maintained in teaching self-regulatory skills. Because selfregulatory skills require significant effort to learn, Web searching instruction should offer ample opportunities for repeated practice [START_REF] Garner | Metacognition: Answered and unanswered questions[END_REF][START_REF] Vermunt | The regulation of constructive learning processes[END_REF][START_REF] Weinstein | Self-regulation interventions with a focus on learning strategies[END_REF]. Paradoxically, this call for repeated practice is at odds with practical findings on the acquisition of procedural skills. Lazonder (2001a) found that three 50-minutes training sessions can enhance procedural skills. Yet this amount of practice proved insufficient to develop self-regulatory Web searching skills. In view of these findings, significantly increasing the length of the instruction would be the most straightforward way to create opportunities for repeated practice. Unfortunately, this has a negative impact on the students' motivation. Integrating repeated practice in self-regulatory skills into other courses may be a more fruitful option.

Principle 9: Support error recovery New users inevitably make mistakes, and correcting these mistakes can be both time-consuming and frustrating. Web searching instruction should therefore teach how to 'do things' and how to undo the things that have gone wrong.

Prevention is better than cure. Web searching instruction should therefore help students to avoid making mistakes. Observations are a powerful means to signal situations, tasks and actions that are prone to errors. The results of these tests are used to further improve the instruction, for instance by further decomposing a task, providing hints, or including error information. For example, our Web searching instruction contained the hint "Please note: Your target site may be on a subsequent results page" to remind students that additional results pages are created in case their search produced more than ten hits. Although accessing additional results pages had been explained in a previous section, observations revealed that most students ignored these pages. This caused them to erroneously conclude that their search had failed to produce the desired outcomes.

Obviously, not every single error can be prevented. When actions are error prone or when errors are difficult to correct, error information can support users in recognizing and recovering from errors.

The 'proverbs and sayings' search task from Figure 2 serves as illustrative example of the need for error information. User observations indicated that students frequently chose "proverb" instead of "proverbs" as a search term. Surprisingly this search did not produce the target site. Because this mistake is difficult to avoid, error information was included to support students in dealing with this error. It is shown as example 1 in Figure 3. As can be seen from this example, error information supports the detection, diagnosis and correction of an error. The problem part directs the students' attention to the screen to check if that particular error has been made. The cause part explains what accounted for the error. The solution part suggests how students can recover from the error. Also note that errors in performing procedural and self-regulatory skills are supported in similar fashion.

Early detection of an error is especially important for its correction. Error information is therefore presented directly after the actions it refers to rather than in a separate trouble shooting section.

Because students who have not made the mistake also read the error information (and also because some find it difficult to discern error information from action steps), an icon may be used to signal its presence.

- -------------------------------Insert Figure 3 about here ------------------------------- -------------------------------Insert Figure 4 about here --------------------------------Two fading techniques may be used to decrease the amount of self-regulatory skills instruction.

The first one pertains to the specificity of the instructional support. In this respect, De [START_REF] Jong | Self-regulation: Regulation of the learning process and learning to regulate-a process view[END_REF] proposed to fade self-regulatory skill instruction from a full description of a self-regulatory skill, through a brief description, to a question that prompts learners to perform that skill (see Figure 4). The second fading technique concerns the locus of control. In keeping with the aforementioned call for genuine learning activities, students should always perform practice tasks in full. However, their share in performing the self-regulatory parts of a search may vary as a function of their prior learning experiences. We applied this strategy in one of our Web searching manuals. On the first search task, students merely had to identify the number of hits. The instruction detailed all other self-regulatory activities, explicitly stating the keyword students should enter, the buttons to they should click, and the Web site they should retrieve. In the course of practice, the responsibility of performing these skills was gradually transferred from the instruction to the learner.

Conclusion

The design principles articulated in this paper aim to guide practitioners in developing instructional materials for learning to search the Web. The principles offer an integrated set of design recommendations that were based on a mixture of theoretical, empirical and practical evidence. Most principles are rather straightforward. And that is exactly what design recommendations should be like.

While practical constraints may complicate the application of a design principle, its description should not interfere with the designer's understanding of instructional specifications.

There is, however, a potential danger to presenting design principles as explicit guidelines for this might raise the impression that designing Web searching instruction merely comes down to applying well-tried solutions. Clearly, this is not the case: design always involves a certain level of creativity. It requires a constant balancing between approved ideas and new, original input or expressions that arise from the need to optimally adjust instructional materials to the features of the search system, the task domain or the target population. Practitioners should therefore treat the principles outlined in this paper as starting points for their thinking about the design of Web searching instruction, rather than as a set of trusty techniques they can follow mechanically.

Additional empirical and practical evidence is needed to further validate the current set of design principles. The design principles may thus serve a research agenda for scholars in the field of education and information technology. Research should primarily address the issue of how selfregulatory skills should be treated in Web searching instruction. Theoretical evidence suggests that the presentation principles outlined in this paper facilitate the development of self-regulatory skills. Their instructional efficacy has been established in various tasks domains. With regard to Web searching, these principles have been found to enhance procedural skills (Lazonder, 2001a); their efficacy in fostering the development of self-regulatory Web searching skills remains to be shown.

The principles attending to the selection of instructional content were extrapolated from studies on Web searching. Assessing their validity therefore seems less imperative even though information technology is evolving at a mind-boggling rate. One reason is that these principles hold no reference to specific browsers and search engines. Furthermore, the instructional content mainly consists of selfregulatory skills. Because of their generic nature, self-regulatory skills can be applied with different search systems (e.g., the selection of search strategies is largely independent of the search system).

The advent of more sophisticated browsers and search engines is therefore unlikely to affect the training needs of inexperienced Web users. Nor will the use of these systems affect a user's level of self-regulation. While users can develop procedural skills by interacting with a search system, instructional support and ample opportunities for guided practice are needed to cultivate selfregulatory Web searching skills. This paper exemplified how such instructional support can be designed. You have to give a talk on one of Kees van Kooten's books. Your friends particularly enjoyed the book Zwemmen met droog haar. You have decided to read a book review first to see if you share their opinion. Find this book review on the Web. Your best friend is looking for a summary of Phileine zegt sorry by Ronald Giphart. This summary should contain at least four quotations from the book. Help your friend and find this summary on the Web.

As of August 1998, the German spelling has been reformed. Your German teacher has asked you to list the four most important changes. Search for these changes on the Web.

Your Dutch teacher has asked you to explain the meaning of proverbs and sayings. The saying Als een pareltje in het goud zitten is not listed in your dictionary. Search the Web for the meaning of this saying.

The Duitsland Institute Amsterdam had recently launched the Duitsland Informatie Net (DUIN), a Web site especially intended for students from secondary education. On DUIN you'll find information on various topics regarding Germany. Two famous German hiphop/rap groups are introduced in the section on popmusic. What are the names of these bands? And what are the Web addresses of their homepages?

-

  Figure 4.

Example 1 (

 1 user activity = action) Before you start browsing the hit list, check how many Web sites were found. Only the first ten search outcomes appear on screen. If your search produced more than ten Web sites, you should evaluate all of these sites. Your target site might be in the second or third set of search outcomes. Look at the blue banner. It shows "how many URLs match your search". Worded differently, it tells you how many Web sites were found. Example 2 (user activity = decision) Choose your next action. You can (1) start a new search, (2) open a relevant Web site, or (3) refine your search. Your choice depends on the number of relevant Web sites. Use the guidelines below to decide on your next action: • No relevant sites → start a new search • Few relevant sites → inspect the most relevant site • Many relevant sites → refine your search The how part holds three selection rules that guide users in choosing their next action. These rules merely assist users in making a decision. They do, for example, not specify the exact number of relevant sites. Notice that the why part is left out. At this point users must choose a next action and there is no true reason why they should (or should not) make this choice. The what part specifies the choice users should make. It also includes the alternatives from which to choose. The how part explains how users can determine the number of hits.The what part is presented in the first sentence. It is elaborated in the subsequent sentences, together with the reason for performing this self-regulatory activity (i.e., the why part).

Figure 1 .

 1 Figure 1. Annotated examples of self-regulatory skill instruction.

Figure 2 .

 2 Figure 2. Illustrative Web search tasks.

Figure 3 .

 3 Figure 3. Two examples of error-information.

PROCEDURAL SKILLS

SELF-REGULATORY SKILLS

Full description

Before you start browsing the hit list, check how many Web sites were found. Only the first ten search outcomes appear on screen. If your search produced more than ten Web sites, you should evaluate all of these sites. Your target site might be in the second or third set of search outcomes.

Look at the blue banner. It shows "how many URLs match your search". Worded differently, it tells you how many Web sites were found.

Brief description

Before you start browsing the hit list, look at the blue banner to see how many Web sites were found.

Your target site might be on an additional results page.

Question

How many Web sites did you find?


Casper D Hulshof
email: c.d.hulshof@utwente.nl

Tessa H S Eysink

Sofie Loyensº

Ton De Jong

ZAPs: Using interactive programs for learning psychology

ZAPs are short, self-contained computer programs that encourage students to experience psychological phenomena in a vivid, self-explanatory way, and that are meant to evoke enthousiasm about psychological topics. ZAPs were designed according to principles that originate from experiential and discovery learning theories. The interactive approach that is offered invites students to engage in subject matter through exploration, experience, and discovery of psychology. In an empirical study the effectiveness of different ZAPs for teaching psychology was examined. A group of students who worked with complete ZAPs was compared to a control group who worked with a ZAP from which the 'activity' component was removed. Posttest results showed that the control group outperformed the experiment group. However, on a retention test the differences between the groups disappeared. The results show that, on the long run, relatively good learning effects may be expected from working with ZAPs.

Introduction

Interactive software has become an established part of almost any standard curriculum. At the level of university education, this is especially true for the 'hard' sciences, notably physics and chemistry. In both disciplines, extensive use is made of computer simulated laboratories to enable students to explore phenomena that are difficult or impossible to reproduce in the real world.

There is a large body of literature that supports the claim that interactive learning fosters learning processes that are hard to obtain with more traditional teaching methods. Traditional teaching methods, which are also called expository, focus on acquiring factual knowledge. Examples of these are attending lectures and studying textbook material. A different view of learning, already expounded by Jerome Bruner in 1961 [START_REF] Bruner | The act of discovery[END_REF] focuses on gaining understanding of a knowledge domain through mindful construction of 'personal' knowledge. Two main recent undercurrents in educational science can be distinguished that underlie the ideas set forth by Bruner. These are experiential learning and discovery learning.

Experiential learning refers to knowledge acquisition as a process of knowledge construction [START_REF] Kolb | Experiential Learning[END_REF]. Learners themselves play a role in learning processes by actively engaging in the learning material. The theory of experiential learning stresses the need to start learning from concrete experiences that can be reflected upon. It is only after having had these experiences that a learner can make an abstract conceptualization of what it all meant. An experience involves a learner performing actions in a situation and observing the effect of these actions. Reflection means understanding the results of actions in a particular situation in such a way that predictions can be made regarding the outcome of similar actions in similar situations. Abstract conceptualization means gaining understanding of a general principle, so that predictions can be made about the effect of other actions in other situations. Although it may sound as a very abstract process, experiential learning may be one of the most concrete types of learning. In daily life we continuously experience new situations, from which we abstract information that we apply in other situations. Experiential learning has relevancy to many different domains, the domain of psychology is no exception. In a way, the psychological processes that we deal with in everyday life can be described as experiential processes. In other situations, such as classroom situations, computers can be useful tools in eliciting experiential learning processes. Interactive software is instrumental in achieving experiential learning on the computer. This may be even more true for the psychological domain than for physics and chemistry. Psychology has human beings as its subject, that is, it deals with ourselves. This means that psychological phenomena may be best understood by subjectively experiencing them and by reflecting upon that experience. One way the computer can foster experiential learning of psychology is by presenting phenomena and having learners interact with them. Examples are visual illusions such as the well-known Ponzo illusion (shown in Figure 1), of which different boundary conditions can be explored. In the description of ZAPs below, this point will be more thoroughly explicated. Another way to use the computer for experiential learning is by using a computer simulation. Experiential learning in the context of a computer simulation is also known as 'discovery learning' [START_REF] De | Scientific discovery learning with computer simulations of conceptual domains[END_REF].

##################### Insert Figure 1 about here #####################
Discovery learning requires students to devise one or more hypotheses and test their validity by performing a number of relevant experiments. Over the years, the computer has proven itself invaluable in the creation and testing of various types of simulations to foster discovery learning about a conceptual domain [START_REF] De | Scientific discovery learning with computer simulations of conceptual domains[END_REF]. The focus on conceptual domains sets restrictions on the type of domains that can be simulated. A domain can only be simulated if there is actually something to simulate, that is, a model of the domain. For that reason, most educational simulations have been restricted to physical, chemical, or economical domains. As stated above, the method of discovery learning can also be relevant to learning about psychological phenomena. Many different phenomena, ranging from biological to cognitive processes, can be described in terms of a model. An example is the gate-control theory of pain sensation [START_REF] Melzack | Pain mechanisms: A new theory[END_REF]. Common descriptions of this theory involve different types of fibers and synaptic 'gates' that together shape different types of pain sensation. The gatecontrol theory describes the relation between phenomena that occur at the physical level of neural interaction and conscious coping strategies at the psychological level. Such a description takes a lot of effort to understand. Also, a verbal description necessarily misses some optional possibilities or issues that one may be interested in. A functional model of the theory helps overcome these problems. Experimenting on a model of the gate-control theory supports deep understanding of a complex phenomenon. In addition, such models may be used to show how a computational description reveals the relation between biological and psychological levels of description.

In sum, it can be argued that it is possible to apply both the theory of experiential learning and discovery learning to the domain of psychology. This involves creating learning material that offers a direct experience with a phenomenon, or the possibility to perform experiments to discover an underlying model. In the ZAP project, the goal was to apply both principles to the design of interactive software modules (called 'ZAPs') in the context of the study of psychology. ZAPs are targeted for use by first-year psychology students. This means that for those students, ZAPs are designed to be easy to use, simple to understand, yet rich enough in content to contain a full 'experience'. The design and structure of ZAPs is only discussed in brief here. A more thorough description, in which the relation between implications from experiential and discovery learning theories and the construction of ZAPs is more fully explored, is discussed in a forthcoming article [START_REF] Hulshof | The ZAP Project: Designing Interactive Computer Tools for Instruction in Psychology[END_REF].

Figure 2 shows an example screen of a ZAP. In this case, the ZAP is about the topic of classical conditioning, a basic type of learning that was first systematically investigated in the early twentieth century by Ivan Pavlov.

##################### Insert Figure 2 about here #####################
The left part of Figure 2 shows a number of different components ('Introduction', 'Discovery', 'Theory', and 'Further Info'). These four components are part of all ZAPs. Three of the four components contain text and pictures. The goal of the text parts is to offer a short introduction to a topic to students (in the Introduction component), to offer a theoretical explanation (in the Theory component), and to offer further explanations and comment on related phenomena (in the Further Info component). The heart of all ZAPs is formed by the 'Activity component' (called 'Discovery' in Figure 2). ZAPs can be divided into three different types, based on the type of activity they contain: 'experience ZAPs', 'discovery ZAPs', and 'experiment ZAPs'. In experience ZAPs, learners directly experience in what way they themselves react to certain psychological phenomena. Examples of phenomena that can be experienced are illusions as displayed in Figure 1, but also higher mental processes such as biases in reasoning or problem solving methods. In discovery ZAPs, learners take the role of experimenter in a virtual laboratory. By setting up and performing experiments, a theory of the model underlying behavior of a virtual subject can be derived. Example phenomena that can be modeled in a discovery task are the already mentioned gate-control theory of pain and classical conditioning, but also human behavior such as the behavior of people whose corpus callosum has been cut (also called 'split brain patients'). In experiment ZAPs, learners do not perform experiments on someone else, instead they are participants themselves in a (classic) psychological experiment. An experiment consists of several trials in which the participant's task is to respond in various ways to stimuli that are presented. The end result is a data set that shows by comparison of different experiment conditions the way a theory of cognitive processing can explain results obtained from an experiment. Examples are the mental rotation experiments by [START_REF] Shepard | Mental rotation of three-dimensional objects[END_REF] which illustrate the relation between perception and mental imagery, and the Stroop effect [START_REF] Stroop | Studies of interference in serial verbal reaction[END_REF] which illustrates the interference between controlled and automated tasks.

During the development of ZAPs, prototype versions were extensively evaluated. Based on this evaluation, improvements were made to the textual components, and revisions were made to the activity component. The evaluations led to many favorable remarks, and the ZAPs have so far been very well received by both students and teachers.

However, an important question remains: do ZAPs contribute positively to learners' knowledge about psychological topics? To answer this question, an experiment was conducted. The current experiment focused on the contribution of the activity component of ZAPs (experience, discovery, or experiment) to the acquirement of psychological knowledge. Since the activity component takes up a central position in a ZAP, it was expected that learners would benefit from interacting with it. A number of studies in which the effectiveness of instruction based on experiential and discovery learning was studied have shown that its result is insight and deep understanding of a topic, not necessarily more knowledge than would be obtained in a more traditional setting (e.g., [START_REF] Swaak | Measuring intuitive knowledge in science: The development of the What-if test[END_REF]. The implication is that the gain from working with the activity component of a ZAP should be measured by looking at insight into and understanding of the psychological phenomenon that is covered by the ZAP. A ZAP without the activity component only consists of a structured text on a psychological topic. It was expected that studying the text would lead to a gain of factual knowledge about a psychological topic, as opposed to real understanding. In sum, positive learning outcomes from working with ZAPs were expected. It was expected that factual knowledge would be the same and insight in the psychological phenomena would be higher for learning with complete ZAPs compared to learning with ZAPs from which the activity component was removed. Furthermore, it was expected that, on the long term, students who were presented with complete ZAPs would better maintain their acquired knowledge and insight than students who received incomplete ZAPs. These results were expected for all three ZAP types (experience, discovery, and experiment).

Method

Participants

In total, 71 participants took part in the experiment: 30 participants were students enrolled in higher vocational education, and 41 participants were university students enrolled at the University of Rotterdam, both located in The Netherlands. Students had not yet received any formal instruction on the topics covered in the different ZAPs. For the higher education students, participation in the experiment was scheduled as part of the school's first introduction week. Because the university students volunteered for the experiment, they were financially compensated for their participation.

Design

A randomized two-group pre-post-retention test design was used for the experiment. A pretest was included, but it was differently constructed from the posttest and retention test so it was not possible to compare results on the pretest to the other tests. The reason for using a different pretest was to prevent participants from being influenced in the experiment by the questions. Two conditions were used, an experimental condition in which participants worked with complete ZAPs during the experiment (that is, texts and the interactive component), and a control condition in which participants only read the text components without the interactive component.

Participants were randomly assigned to the experimental condition (n=35) or the control condition (n=36). Participants in the experimental condition worked with complete ZAPs (that is, ZAPs consisting of a number of text components and an activity component). Participants in the control condition worked with ZAPs consisting of text components without the activity component. Participants were neither made aware of the fact that they were assigned to a particular condition, nor of the difference between the conditions. All participants worked with six different ZAPs during the experiment. To control for sequence effects or fatigue, the sequence in which participants worked with ZAPs differed for each person. It was not possible for participants to work through the ZAPs in any other sequence than the one that was offered to them.

Materials

From the ZAPs that were produced as part of the project, six were selected for use in the present experiment. The selected ZAPs were representative for the complete collection. For each type of ZAP (experience, discovery, or experiment) two representative ZAPs were selected. Table 1 lists the titles of the ZAPs that were selected for each type. A short description of the contents of the ZAPs is also given in the table.

##################### Insert Table 1 about here #####################
The largest difference between ZAPs in the experiment condition and the control condition was the presence of an activity component in the former. In addition, to cope with the absence of an activity from the control condition minor textual modifications were made to the text to remove any references to the activity. Also, instead of actually working with an activity, participants in the control condition read a short explanation that described the original experiment on which the ZAP was based. In both conditions, the ZAPs were opened within a web browser. There were no technical difficulties in opening or running the ZAPs during the experiment.

Tests

In total, three tests were administered to participants: a pretest, a posttest, and a retention test. All three tests covered the six topics dealt with in the ZAPs plus an additional topic about an additional seventh ZAP which was not used in the experiment. The seventh topic (on the 'serial position' memory effect) was used in order to control for experimental artifacts.

The pretest consisted of 14 open-answer questions, 2 for each topic, the aim of which was to measure the extent to which participants had prior knowledge about the topics that were covered in the different ZAPs. An example item from the pretest is 'What is the corpus callosum, and what is its function?'. It was expected that participants would have only little prior knowledge about psychology, since they had just entered their first year of higher education. Therefore, it was specifically stated in the instructions to the test that it would not matter if they answered 'no idea' to the questions. This was done to prevent participants from becoming frustrated or unmotivated. The test items were constructed in such a way that the probability of focusing participants' attention to specific parts of the ZAPs would be minimal. As a consequence, the style of the pretest was different from the posttest. The posttest consisted of 35 open questions. There were 5 questions for each of the 6 ZAPs covered in the experiment, and 5 additional questions about the additional seventh topic. The questions of the posttest concerned the topic of each ZAP used in the experiment. The 5 questions for each topic were divided over 2 'knowledge questions' about specific factual content, and 3 'insight questions' that involved predicting imaginary variations on the situation that was covered by the activity component of the ZAP. Knowledge questions covered the textual components of a ZAP. An example knowledge question (about the 2-4-6 Task) is: 'What is meant by inductive reasoning? Please provide an example of inductive reasoning'. Most insight questions were longer than the knowledge questions. They usually described a situation that might occur in the activity component, and asked participants to imagine and predict what would happen if the situation was modified in some way. An example insight question about classical conditioning is shown in Figure 3. All insight questions were constructed in such a way that they could be answered by participants in both the experiment and the control condition.

##################### Insert Figure 3 about here #####################
Answers to the open questions were scored afterwards. The 5 questions for each ZAP added up to 10 available points. The division for each ZAP was 4 points for the knowledge questions, and 6 points for the insight questions. Division of points over the questions varied with the content.

To measure the stability of knowledge gained in the experiment, a retention test was administered approximately ten days after the experiment. The retention test was similar to the posttest. About half the questions from the knowledge test had been slightly modified, to avoid complete equivalence between the tests. Again, participants answered 35 questions, part knowledge questions and part insight questions.

Process measures

While navigating through the ZAPs, all operations participants carried out (as measured in 'mouse clicks') were unobtrusively registered by the computer. This made analysis of the time subjects spent on different components of a ZAP possible. Because the navigation log data was combined with student identification data, it was also possible to compare the experiment and control condition with respect to navigation through ZAPs.

Procedure

The experiment was divided over two sessions. In the first session, participants worked individually at the computer. The session was completely web-based, which means that both the tests and the ZAPs were opened within a web browser. All participants first received an individual login code, and used this code to start the experiment session. After filling out a form with some personal information (name, age, prior education), the knowledge pretest started. Because most participants had little prior knowledge about psychological topics, completing the pretest took only a short time. Immediately after the pretest, participants started working with the first of 6 ZAPs. Depending on the login code, each participant worked with the ZAPs in a different sequence. To prevent participants from skipping a ZAP, they had to spend a minimum time of 3 minutes on working with a ZAP before they were allowed to continue to the next one. Early exits from a ZAP were not possible: attempts to do so were met with a warning, which asked the student to spend at least a minimum time with the ZAP. After completing all 6 ZAPs, the posttest was administered to participants. The questions on the posttest were delivered in the same order as the sequence of ZAPs that a participant had worked with. For example, a participant who had first worked with the cognitive dissonance ZAP and had finished with the split-brain ZAP would first receive 5 questions about cognitive dissonance and end with 5 questions about split-brain. The session ended as soon as the posttest was completed. The complete session took about two hours.

The second session took place approximately 10 days after the first. In this session, participants did not work with the computer. Instead, a retention test was administered as a pencil-and-paper test. The sequence in which the questions were printed on the question sheet varied: for each participant it was the same as the sequence of questions on the posttest.

Results

Two types of results are of interest in the context of the present experiment: students' performance on the tests (pretest, posttest, and retention test) and process data. Analysis showed no difference between the two student populations that participated in the experiment (from higher education and university education), so only results for the complete group are described in this section. Test results are described first, followed by process results.

Pretest. As was expected beforehand, students experienced much trouble in answering the pretest questions. Most questions where answered by almost all students with 'Don't know', or 'No idea'. Because of this it was not possible to compute a reliable value of Cronbach's α. The only question that showed relatively good answers were the ones on the topic of classical conditioning. The reason for this may be that this topic is regularly taught in Dutch secondary schools as part of the biology curriculum. Other results from the pretest revealed a number of misconceptions about psychological phenomena. For example, cognitive dissonance was seen as a pathological phenomenon (which is not the case), and split-brain patients were regarded as suffering from a multiple personality syndrome (which they are not). In general, the results show that prior knowledge about the different subjects that were subsequently covered in the ZAPs was very poor. Furthermore, the results on the pretest show that the experiment and the control condition are comparable, as they did not differ (F 1, 69 = .05, n.s.).

Posttest and retention test. Cronbach's α for the complete posttest was .81. Cronbach's α for the complete retention test was .73. The correlation between performance on the posttest and the retention test was .72. To measure differential learning gains from working with the ZAPs, posttest and retention test results for the experiment and control condition were compared. In Table 2 the results are shown, divided over the 3 ZAP types that were used (for each type, 2 ZAPs were used in the experiment, as shown in Table 1). This resulted in an average test score that could range from 0 to 10. On the posttest, the control condition and experiment condition scored 4.81 (SD=1.45) and 3.79 (SD=1.36), respectively. On the retention test, the averages are 4.16 (SD=1.25) for the control condition, and 3.87 (SD=1.09) for the experiment condition. A statistically significant interaction effect between the test (posttest or retention test) and condition (experiment or control) was found (F 2,57 = 4.40, p<.05).

##################### Insert Table 2 about here #####################
The results show that participants' total score in the experiment condition was lower on the posttest than that of participants in the control condition (F 1,67 = 10.18, p<.01). This difference was caused by both different performances on the knowledge questions (F 1, 67 = 9.13, p < .01) and on the insight questions (F 1, 67 = 6.97, p < .01). Differences between the conditions on the posttest are found for all three ZAP types, experiment ZAPs (F 1, 69 = 5.36, p < .05), discovery ZAPs (F 1, 68 = 7.54, p < .01) and experience ZAPs (F 1, 67 = 5.59, p < .05). Results on the retention test show different results. No significant difference appeared between the experiment and control condition (F 1, 61 = 1.28, n.s.). Also, no differences are found for the knowledge and insight questions, nor for the different ZAP types. To make the test results more clear, Figure 4 shows a graph of the comparison between the two conditions on the posttest and the retention test. The test scores that are displayed in the figure are average test scores per ZAP (that is, knowledge and insight questions, averaged for all ZAPs).

##################### Insert Figure 4 about here #####################
As can be seen from Figure 4, the control condition shows a significant decrease in performance from the posttest to the retention test (F 1, 32 = 10.83, p < .01), whereas the results of the experiment condition remain the same from posttest to retention test (F 1, 25 = .26, n.s.).

Posttest and retention test items of the seventh topic. As described in the Method section, the posttest and the retention test also consisted of items on a topic on which students did not receive a ZAP. This topic was used in order to control for experimental artifacts. The results on the questions about this topic were, as was expected, low. There was no difference between the experiment condition and the control condition (F 1, 59 = .01, n.s.), and the scores on the posttest and the retention test remained the same for both the control condition (F 1, 32 = .00, n.s.) and the experiment condition (F 1, 27 = .01, n.s.). When the results on this topic are divided between knowledge questions and insight questions, a difference between scores on these types of questions becomes clear, though. On the knowledge questions students score low and their score on this topic is much lower than their score on the topics they received ZAPs on. However, on the insight questions the difference between scores on topics they received ZAPs on and scores on the seventh topic is very small.

Process results. For all participants, time spent on reading a text component or on working with the activity component (in the experiment condition) was registered. In ZAPs of the experiment type, time was also registered for the extra data component that this type contained. When a participant had viewed a component for less than 5 seconds, the result was removed from the analysis. The results are displayed in Table 3.

##################### Insert Table 3 about here #####################
As can be seen from the table, students in the experiment condition on average spent 8:45 minutes working with a ZAP, compared to 5:42 minutes for students in the control condition. The time difference can be explained by the presence of an activity component in the experimental condition, which was absent in the control condition. However, it can be observed in the table that there are consistent differences between the conditions in the average time spent reading the textual components. In the control condition, participants spent more time reading the texts than students in the experiment condition. When the average time spent on ZAPs is divided over the three different types (experience, discovery, or experiment), it can be seen that there are large differences in the time participants spent working with each. For experience ZAPs and discovery ZAPs differences are the largest: participants in the experiment condition spent much longer working with experience ZAPs than with discovery ZAPs. For the experiment ZAPs the data is less clear because the experiment in the activity component takes a minimum time to complete for all participants. The short average time spent on the activity component of the discovery ZAPs indicates that participants may have had trouble understanding the rich content of the activity in the discovery ZAPs. Process data confirmed this hypothesis, because it showed that most participants performed only a few experiments in these ZAPs.

Discussion

Our expectations were to find no differences between conditions for the knowledge questions of the posttest and better scores on the insight questions of the posttest for the experiment condition. Furthermore, better conservation was expected on the long run for the experiment condition.

To start with the latter, the results partly confirmed the hypothesis. The interactive component in which students actively engaged with the subject matter was meant to lead to good understanding and integration of knowledge, leading to prolonged conservation. The results showed that the scores of students in the control condition decreased from posttest to retention test, whereas scores of the experiment condition remained the same over time. In the interval between the experiment and the retention test, students in the experimental condition better maintained their knowledge than students in the control condition. The combination of the textual parts with the activity in the experiment condition may support better understanding of psychological information, which means that memory of this information degrades more slowly over time.

The expectations concerning the posttest administered immediately after the experiment were not confirmed by the results. The interactive component was especially meant to improve insight in the topic of the ZAP, but students in the experiment condition performed less on the insight questions than students in the control condition.

However, the scores on the insight questions of the seventh topic show that the insight questions that were used might not have measured insight as was intended. The idea behind the design of the insight questions was to measure deep understanding of the subject matter. This was operationalised by asking students to imagine a variation of the standard situation that was used in the ZAP activity, and to answer a question based on this hypothetical situation. It was thought that, in order to come up with an answer, students needed deep understanding of the subject matter. However, students' scores on the insight questions of the seventh topic, the topic on which they had not received any instruction, were almost as high as the scores of students on the insight questions of topics on which they had received instruction. It can be hypothesized that answering the insight questions did not necessarily require deep understanding of the subject matter. The answers could possibly also be generated by correct reasoning. Most insight questions involved imagining the situation, after which it may have been possible to have reason follow its logical course. Possibly, insight was not necessary to successfully answer the insight questions, nor was instruction into the topic.

The finding that participants in the experiment condition scored lower on knowledge questions of the posttest than participants in the control condition was also not in accordance with prior expectation. The knowledge questions concerned knowledge which was presented to the students in the textual components of the ZAP. As these textual components were the same for both conditions, no differences were expected on the posttest between the experiment and the control condition. However, the fact that the experiment condition scored lower may, at least in part, be explained by the setup of the study. In the experiment condition, the students worked with a number of normal, complete ZAPs. Performance of this group was compared to a control condition, in which students were assigned only one task: to carefully read and understand a number of texts. Students in the first group faced the dual task of interpreting texts and carrying out an interactive assignment. It may well be that students in this condition saw the purpose of the text parts primarily as an illustration to the activity component. That would mean that these students focused less on the factual content of the texts, and more on the information pertaining to the task at hand. Evidence for this explanation can be observed by analysis of the time spent on each ZAP component: time spent on reading textual components-that were equivalent for both conditions-was systematically shorter for the experiment condition than for the control condition. On the whole, students in the control condition were given a more simple task than students in the experiment condition, since their only task was to read texts. They were aware of the fact that afterwards, they would receive questions on the content of these texts. In retrospect, the difference in tasks assigned to students in both conditions may be responsible for causing a better performance on the knowledge questions of the posttest for the control condition compared to the experiment condition.

Conclusion

ZAPs are positively welcomed by both students and teachers. Psychological phenomena that students usually only read about can be fully experienced and experimented with. In this respect, they succeed in evoking enthusiasm about a variety of psychological topics. Although the present study did not find beneficial learning gains for students who used complete ZAPs immediately after instruction, compared to results for students who used ZAPs without the interactive component, results on the long run were positive in the sense that students who used the complete ZAPs better conserved their knowledge. It may be hypothesized that this difference in conservation of knowledge will also be found on a long-term retention test. It should be kept in mind that ZAPs are originally meant to be integrated in existing courses. The small modules are highly suitable for this purpose. One or more ZAPs can easily be used in addition to existing learning materials. The value of ZAPs lies in the interaction they add to otherwise passively consumed texts, the combination of factual information with an attractive and interactive example, the possibility of easily integrating ZAPs in existing courses, and the added variety that makes a ZAP worthy of repeated visits. All these opportunities that ZAPs offer were not taken advantage of in the current study. Future studies should focus more on these aspects, so that the added value of ZAPs in existing curricula comes more to its right. From the beginning to time A, each trial the dog was presented with meat and the sound of a bell. From time A on, the dog was presented with both meat and a bell sound, and in addition a light was flashed. The procedure of presenting three stimuli was repeated a number of times, until time B. From that moment on only the lamp was flashed during trials. The graph shows the amount of salivation for each time period.

The question is: why is there a lack of salivation after time point B?

Figure 1 .Figure 2 .

 12 Figure 1. Example of the Ponzo illusion. Which horizontal line is the longest?

Figure

  Figure 3. Example insight question (on classical conditioning)

Acknowledgements

The development of ZAPs took place in the context of the ZAP project, which was supported by a grant from the Stichting SURF. The ZAP project team consisted of Henk Schmidt, Wilco te Winkel, Sofie Loyens, Eveline Osseweijer and Manon de Jong of the Erasmus University Rotterdam, and Jules Pieters, Jan Oosterhuis, Jakob Sikken, Marita Wesselink, Ben Reimerink, Ton de Jong, Casper Hulshof and Tessa Eysink of the University of Twente.