H S Eysink
email: eysink@edte.utwente.nl

Sanne Dijkstra

Jan Kuper

Running head: COGNITIVE PROCESSES IN SOLVING LOGIC PROBLEMS Cognitive Processes in Solving Variants of Computer-Based Problems Used in Logic Teaching

Keywords: Logical Reasoning, Problem Solving, Cognitive Processes, Computer-Based Instruction, Visualisation, Manipulation

. For all instructional conditions, the scores on the transfer tests showed a significant increase in understanding the conditional. Visualisation, operationalised as presenting only formal expressions or a geometrical reality in addition to these, showed no differences on the transfer test. If only presented formal expressions, about half of the participants needed to make drawings of the objects, especially when the problems increased in complexity. The manipulation condition, in which the participants could either construct a geometrical world or were presented a fixed world, significantly influenced the participants' cognitive processes in solving the logic problems. The students worked affirmatively and were tempted to stay in familiar situations. The results support the authors' view that visualisation facilitates cognitive processing. Moreover, the results are congruent with Piaget's theory of the development of knowledge of formal science concepts from the action with objects.

The central concern of logic is the correctness of human reasoning. Reasoning occurs in all sciences and in all possible contexts. The rules of logic are valid in all these situations. To use the same rules in every possible situation, they must be formulated in a general way, that is, in such a way that they are not restricted to a given context. This makes logic abstract and general.

Furthermore, the language of logic is formal. Agreements are made about symbols to be used and about the way these symbols can be connected to each other to form formulas in a formal and precise way.

Various studies have shown that a substantial part of all students have difficulties with learning and using these abstract and formal characteristics of logic. Students experienced logic education as being difficult, too abstract and boring (e.g., [START_REF] Goldson | A review of several programs for the teaching of logic[END_REF][START_REF] Fung | Why computer science students find formal reasoning frightening[END_REF]. Besides this, [START_REF] Barwise | Computers, visualization, and the nature of reasoning[END_REF] stated that students often saw logic as the manipulation of logical expressions by applying formal, meaningless rules. They do not get sufficient practice in finding the relation between abstract representations and real-life meanings and therefore have difficulties in applying abstract principles to everyday phenomena [START_REF] White | ThinkerTools: Causal models, conceptual change, and science education[END_REF]. This resulted in students not grasping any real understanding of the concepts and rules of logic.

In addition to this, studies have shown that abstract reasoning is difficult to improve. Only near-transfer effects (Cheng & Hollyoak, 1986) or effects of years of formal training [START_REF] Lehman | A longitudinal study of the effects of undergraduate training on reasoning[END_REF]) have been found. [START_REF] Freudenthal | Revisiting mathematics education: China lectures[END_REF] supposed that abstract reasoning is difficult to improve, because common-sense ideas often obstruct scientific ideas. In everyday life, people develop naïve notions about logical reasoning. Sometimes, these often ill-defined concepts and rules do not meet the rules of logic. If the learners develop certain ideas, it is difficult to change their minds and to convince them they should replace the old (incorrect) knowledge with the new (correct) knowledge. Teachers in logic are often confronted with the problem of how to teach students to solve logic problems and to translate the real world statements into formal statements. Moreover, they have to make clear to the students how to use the logical connectives, conjunction (∧), disjunction (∨), conditional (→) and negation (¬). To clarify the meaning of logical expressions, [START_REF] Barwise | The language of first-order logic: Including the Microsoft Windows program Tarski's World 4.0 for use with IBM-compatible computers[END_REF] constructed a reality of computer-generated geometrical objects, which students could use to construct logical statements and to check whether these statements were true. This world of geometrical objects, which was labelled Tarski's World, was presented on a screen and could be constructed and manipulated by using a mouse. Tarski's World has been reviewed [START_REF] Goldson | A review of several programs for the teaching of logic[END_REF] as being an easy and fun to use programme that is 'capable of teaching a great deal about a formal language, its interpretation, models, counterexamples and consequence'. [START_REF] Van Der Pal | Balancing situativity and formality: The importance of relating a formal language to interactive graphics in logic instruction[END_REF] designed an instruction for learning formal logic in which Tarski's World was used. In addition to formal expressions, students could manipulate objects in this geometrical world. This instruction was compared to an instruction in which only formal logical expressions were given.

Results showed that students who were given the experimental instruction, performed better on transfer tests than students who were given the formal instruction. In the experimental instruction, however, two instructional variables were confounded: (a) the use of a geometrical reality, and (b) the manipulation of the objects. Thus, it was possible that the effect could be caused either by one of the two variables, by both or by an interaction of the two. The purpose of the present study was to investigate which variable was critical. In this respect, two issues received attention. The first concerned the extent of visualisation, that is, the effect on students' performances of presenting a geometrical reality in addition to formal expressions. The second concerned the extent of manipulation, that is, the effect on students' performances of manipulating the objects presented. Both instructional variables were supposed to facilitate the students' problem solving process and thus the development of the students' knowledge and skills.

Abstraction and Reality

In order to learn formal logic, cognitive development has to be in a stage in which formal concepts can develop and in which the logical operators can be used in a meaningful way. [START_REF] Piaget | Epistémologie génétique[END_REF] called this stage the formal operational period and he claimed that it is reached at about the age of twelve. During this stage, children start to think abstractly. They can formulate hypotheses without actually manipulating concrete objects, and when more adept in this, they can test hypotheses mentally. They can generalise from a real object to another and from a real object to an abstract notion.

Many adults, however, still have problems learning abstract, formal concepts without any reference to real world objects. They are unable to solve formal problems, in which only symbols are used. They can only do mental operations with real (concrete) objects, events or situations. It was estimated that this is the case for 40 to 70 percent of all adults [START_REF] Pintrich | Implications of psychological research on student learning and college teaching for teacher education[END_REF][START_REF] Freudenthal | Revisiting mathematics education: China lectures[END_REF] also recognised the difficulties learners have when studying mathematics. He proposed to connect the formal, abstract mathematics to reality, so that the learners could infer these formal concepts from this reality. This led to the suggestion, that learners need to be offered concrete problem situations, which can be imagined and can be used to develop mathematical knowledge and skills, so that the learners will understand the concepts and be able to work with them.

To study the relationship between abstract concepts and reality, it has to be clear how reality is described in formal sciences, and what will be the best possible way to represent this relationship. The essential characteristic of logic is that it can be applied to all situations and to all worlds. The world or set of worlds is the reality and the formal language describes this reality.

The way in which this reality can be represented can range from a direct, everyday reality via an entirely pre-structured reality (e.g. geometrical figures) to complete abstraction (e.g. abstract mathematical objects, elements, sets). The drawback of learning in an everyday life context is that students are tempted to pay attention to irrelevant aspects of the problem. Students' prior knowledge consists of ideas about often ill-defined concepts and rules. When solving problems in an everyday life context, the students will use these naïve ideas in which pragmatic aspects as preferences, intuition, and hidden assumptions can play a role. Language is also permeated with conversational implicatures [START_REF] Grice | Logic and Conversation[END_REF], that is, sentences often express suggestions without explicitly stating them. When learning logic by solving logic problems in an everyday life context, students will use their everyday life ideas and expressions about what is correct reasoning, whereas they should learn to abstract from the given context and learn to reason according to the rules of logic.

Giving reality as a complete abstraction also shows some drawbacks. Students then receive abstract, conceptual knowledge that is isolated from the situations in which this knowledge is normally used. Students will not always understand what the concepts and rules are about and the knowledge will not be imbedded into prior knowledge. They may only learn to shuffle the abstract symbols without comprehending what they are doing and why. When trying to overcome this abstraction by imagining concrete objects for abstract expressions, they will use their own situations and in doing so mistakes can be made.

Although a common goal of logic education is to be able to apply logic and to reason logically in all situations, the use of a geometrical world to learn to reason logically will probably support learning: it makes all operations possible and at the same time shows what happens when certain operations are applied. The world is completely defined, in such a way that errors can be precluded, since irrelevant characteristics of the problem situation are left out of the context. Abstract principles are related to concrete meanings, so that meaningfulness and understanding can be reached. Because the context is controlled, unwanted characteristics of the context will have no influence. [START_REF] Stenning | Contrasting the cognitive effects of graphical and sentential logic teaching: Reasoning, representation and individual differences[END_REF] added to this that a geometrical world shares the property of specificity with the internal representations used by humans in their reasoning. It is the specificity of the world that makes it cognitively manageable and more concrete.

A Short Description of Logical Notions

Logic is the science of (both human and machine) reasoning, which tries to discover conditions by which conclusions are justified and correct. In order to reach precision, contemporary logic is presented in a formal, mathematical way. This paper uses the language of first-order logic. The (formalised) language of this logic contains names (a, b, …, x, y, …) to denote individual objects, and predicates to express properties of objects and relations between these objects. For example, Large is a predicate and Large(a) says that object a is large. Likewise, Larger(a, b) says that object a is larger than object b, and Between(x, y, z) expresses that object x is positioned between the objects y and z.

Expressions of this form are called (elementary) propositions. They can be combined into more complex propositions by the following connectives: negation (¬, not), conjunction (∧, and), disjunction (∨, or), and conditional (→, if … then …, sometimes called implication). For example, the formula Larger(a, b) → Smaller(b, a) says that "if a is larger than b, then b is smaller than a".

Propositions express states of affairs about the world and they can either be true or false.

The connectives are truth functional, that is, the truth or falsehood of a complex proposition is completely determined by the truth or falsehood of the propositions of which it is composed, as described by the truth tables given in Table 1. - ----------------------------------Insert Table 1 about here -----------------------------------So, for example, if p is true, then ¬p is false. And also, if p is true and q is false, then p ∧ q is false, but p ∨ q is true. Notice that p → q is only false if p is true and q false. In all other cases p → q is true.

As mentioned above, the formal language contains names to denote objects. There are two kinds of names: constants (a, b, …) denote specific objects (it is assumed that one knows which objects are denoted), whereas variables (x, y, …) denote arbitrary objects (one does not know which objects are denoted). This difference is exploited by the two quantifiers, the universal quantifier (∀, for all) and the existential quantifier (∃, there is at least one). For example, ∀x Large(x) says that all objects x are large, whereas ∃x Large(x) expresses that there is at least one object x which is large. A more complicated example is ∀x (Cube(x) → Large(x)). Literally, this formula says that "every object x is large, if it is a cube". In everyday language this is normally said as "every cube is large", or "all cubes are large". In this formula all occurrences of the variable x are said to be within the scope of the quantifier, and all occurrences of x are bound by the quantifier. If a variable is not bound by a quantifier, it is called free. If a formula contains a free variable, one cannot know whether this formula is true or false, since it is not known which object is denoted by this variable. For example, we do not know whether Large(x) is true or false, since we do not know which object is denoted by x. On the other hand, when x is bound by a quantifier, as in ∀x Large(x), we can know whether this formula is true or false, since now x ranges over all objects, and for each individual object it can be checked whether it is large or not.

In general, one can determine whether a formula is true or false whenever that formula does not contain free variables. Such formulas are called sentences.

Tarski's World

The computer-based learning environment Tarski's World was designed by [START_REF] Barwise | The language of first-order logic: Including the Microsoft Windows program Tarski's World 4.0 for use with IBM-compatible computers[END_REF]. By defining a world of visible, geometrical objects with certain characteristics and relations, users learn semantic structures as studied by logic and they learn to determine the truth value of formulae. Within the programme, all situations are completely defined and the programme is able to provide feedback on syntactic aspects and the truth of logical formulae. A typical example is shown in Figure 1.

- ----------------------------------Insert Figure 1 about here -----------------------------------The problems that the students have to solve lead to the construction of logical expressions.

By examining types of errors and sequences of errors, the students' reasoning process could be mapped out. For example, in Figure 1 sentence 1, the problem "There is a small tetrahedron or a medium dodecahedron" leads to the logical expression

∃x ((Tet(x) ∧ Small(x)) ∨ (Dodec(x) ∧ Medium(x))) (1)
If programmed accordingly, the programme is able to register all errors the students made when constructing a formula and when reasoning. As can be seen from the T's and F's at the left of the expressions in the sentence module, two of the displayed expressions are true, whereas one is false. Expression 1 claims that there must be a small tetrahedron or a medium dodecahedron.

As object a is a small tetrahedron, the sentence is true in the world given. But as can be seen in the inspector module, the student thought Expression 1 was false. The programme provided feedback by telling the student that the expression was indeed syntactically correct, that it was indeed a sentence, but that the answer that the sentence was false, was incorrect. These three errors, (a) whether the logical expression was syntactically correct, (b) whether the logical expression was a sentence, and (c) whether the truth value given by the student to this logical expression was correct, were being recognised by the feedback mode of Tarski's World.

Another example in Figure 1 is sentence 2, "All cubes are large" which has a hidden conditional. Another way of saying "All cubes are large" is "For all objects it holds that, if it is a cube, then it is large". The correct translation is given in the logical expression

∀x (Cube(x) → Large(x)) (2)
However, many subjects do not recognise the hidden conditional and render "All cubes are large" as

∀x (Cube(x) ∧ Large(x)) (3)
which means "All objects are large cubes". In this case, the student made a faulty translation and Tarski's World is not able to recognise this. This brings us to two errors not being recognised by the feedback mode of Tarski's World. These errors are (d) errors in the translation of the Dutch sentences into logical expressions, and (e) errors during the problem solving process of finding an answer. The last two types of errors could only be traced by analysing the log files in which all the students' actions were logged.

The Visualisation Variable

In order to study whether a simple world of geometrical objects facilitates logic learning, two instructional conditions were designed and constructed. In one condition, the students were presented a geometrical world in addition to Dutch sentences that had to be translated in firstorder logic. In the other condition, the students were given a sentential world, that is, a textual description of a world, in addition to Dutch sentences that had to be translated in first-order logic. The authors assumed that the students who were given a geometrical world would outperform the students who were only presented a sentential world. The students in the first group would be able to visually check their reasoning in the available world and easily retain the steps made. When only presented a sentential world, imaginations arise automatically on the basis of the verbal descriptions. Johnson-Laird (1989) called these imaginations the mental models of discourse, making explicit the structure of the situation as it is imagined instead of the exact sentence. This results in a higher load on working memory and thus a greater chance of making errors and less concentration on the conceptions and operations of logic.

Learning results were measured by administering a transfer test. It was supposed that students understood the subject matter, if they were able to apply their newly acquired knowledge and skills to new situations in which the subject matter was not learnt. The transfer test consisted of items that measured the students' ability to apply the rules of logic to everyday life problems. Psychologists have long been sceptical about the extent in which logical skills generalise to domain independent reasoning skills as people use in everyday life (see [START_REF] Nisbett | Teaching reasoning[END_REF] for an overview). It is widely held that for most people teaching logic only influences the algebraic symbol shuffling skills. Logic is seen as a syntactic mechanism of reasoning and because humans do not reason syntactically, teaching logic will neither help them to reason, nor to understand what their reasoning means. However, the authors assumed that if instruction is given in which the reasoning can be applied to a geometrical world, understanding is reached, so that this knowledge and these skills can also be used adequately in everyday life problems.

In addition to the transfer results, the authors' interest concerned the acquisition process together with the errors that were made within this process. Therefore, the authors decided to study the errors that the students made during the course of the problem solving process, although they did not have a specific prediction about the errors that would be made. It was assumed that presenting a simple world of geometrical objects would not influence the number or type of errors concerning syntax. However, it was hypothesised that the problem solving process of the students given a geometrical world was different from that of their colleagues in the other condition, because the former could use the geometrical world to check the steps in the reasoning process and to use the objects to retain the steps made, whereas the latter had to imagine the world themselves and had to cognitively operate on the imagined objects. This makes the problem solving process more difficult which will manifest in students needing more time to solve the problems, needing more checks of logical expressions in worlds and making more errors.

The Manipulation Variable

To solve logic problems, knowledge of abstract objects as well as the skill to perform logical operations has to be developed. [START_REF] Piaget | Epistémologie génétique[END_REF] Experience, then, referred to the relation between the characteristics attributed to the objects by manipulating them or operating on them, and not to the characteristics the objects already possessed. In this sense, knowledge was seen to be abstracted from the operations as such and not from the physical features of the object. For instance, a child learned the concept of order by ordering different balls to size. In this case, size was a feature all balls possessed, order was added by operating on the balls. The child understood that operating on the balls did not change the characteristics of the balls themselves.

At a certain moment, the applications of operations on physical objects become superfluous and the logic-mathematical operations are being integrated in symbolic operators, which can be applied in different contexts. Therefore, from a certain moment, pure logic and mathematics are left, for which no (concrete) experience is needed. Formal concepts and operations can be GM. It was supposed that the students in the third condition would profit most from the instructions.

Method

Participants

The participants were 66 first-year social science students (38 male, 28 female; mean age 19.4 years, SD 1.0). They volunteered for the experiment for which they were paid a fee of 50

Dutch guilders (approximately $ 25). None of the students had any experience in computer programming or logic.

Learning Environment

The computer-based learning environment Tarski's World 4.1 for Windows [START_REF] Barwise | The language of first-order logic: Including the Microsoft Windows program Tarski's World 4.0 for use with IBM-compatible computers[END_REF] was used. Tarski's World provided an introduction in first-order logic. In the problems to be solved a well-defined, simple world of three kinds of geometrical objects (cubes, tetrahedrons and dodecahedrons) was used. Participants could change the size of the objects (small, middle or large) and the position of the objects (to the left of, to the right of, at the back of, in front of, and between). The learning environment consisted of four main components (see The programme was adapted to fit the experimental design by making three versions that corresponded with the three conditions. In all three conditions, the students had to translate Dutch sentences into first-order predicate language. These sentences had to be checked in a world. In the sentential, non-manipulation condition SN, this world was given by a textual description. In the geometrical, non-manipulation condition GN, the world consisted of geometrical objects, which could not be manipulated. In the third, geometrical, manipulation condition GM, students had to construct and manipulate the world themselves.

The following changes in Tarski's World were made: (a) the menu bar was made invisible, so that students were not able to give commands themselves; (b) the programme was translated from English into Dutch, so language could not interfere with the results; (c) worlds and sentences were automatically loaded and saved when starting and finishing a task; (d) in the geometrical, non-manipulation condition a certain world was given which could not be manipulated by the students; and (e) in the sentential condition the geometrical world was made invisible.

The instruction accompanying Tarski's World was provided in the browser of Netscape

Communicator 4.06. The changes in the browser were as follows: (a) the menu options were disabled, so that students could not navigate completely freely in the browser nor surf on the internet; (b) the browser was linked to Tarski's World, so commands in one programme resulted in actions in the other programme.

Learning Materials

The learning material comprised the conditional. Two sentences p and q can be combined into a new sentence with the symbol of the conditional. The new sentence will look like p → q; its English counterpart is "If p, then q".

Tests and Questionnaires

To measure the students' knowledge of the meaning of the conditional, a transfer test of eleven items was administered. Figure 2 -----------------------------------

Insert Figure 2 about here -----------------------------------

The remaining seven items included one Reduced Array Selection Task (RAST, Johnson-Laird & [START_REF] Johnson-Laird | A theoretical analysis of insight into a reasoning task[END_REF], two items to be solved best by using set theory and four items in which a statement was given and the students had to decide whether this statement was true or false or whether you could not tell from the information given. Figure 3 shows a typical example of the latter.

-

---------------------------------- Insert Figure 3 about here -----------------------------------
The students had to complete three comparable versions of the transfer test, namely a pre-, post-and retention test. These tests were designed to measure the knowledge gained after the various instructions. All students received the same tests.

Two questionnaires were administered. The first concerned the former education of the students on mathematics and logic. The second questionnaire was an evaluation of the instruction in combination with Tarski's World.

All tests and questionnaires were administered on the computer.

Log Files

Two log files were generated during the experiment. The first log file logged all the actions of the students while working in Tarski's World. It logged the status of the sentences and the matching world every time the students checked this combination on Well Formed Formula (WFF), sentence and/or truth. The second log file logged all the actions of the students while working in the browser. This, among others, concerned answers of students on two questionnaires, answers of students on the transfer tests, and time registration.

From these log files, different results could be taken. First, the mean time students were working on the problems during general training and experimental problems could be calculated.

Second, the mean number of checks per student could be computed. As the subject matter concerned the conditional with the general format p → q, these checks could be divided into the four possible truth-falsity combinations (1 → 1, 1 → 0, 0 → 1, 0 → 0, in which 1 = true, and 0 = false). Third, the mean number of errors in the final answer per student could be determined.

These 'final' errors could be divided into two kinds: (a) 'indicated' errors, that is, errors indicated by the feedback module of Tarski's World, but deliberately ignored by the students; and (b) 'non-indicated' errors, that is errors not indicated by the feedback module of Tarski's World, so that the students did not detect them. When the latter type of errors occurred, it was mostly because Tarski's World did not check whether the logical expressions were correctly translated from the Dutch sentences.

Design and Procedure

Experimental Conditions. The subjects were randomly assigned to one of three conditions.

In all conditions, the students were given Dutch sentences, which they had to translate in first order predicate language. In the sentential, non-manipulation condition SN, the students had to check the sentences to be true or false in a given, sentential world. In the geometrical, nonmanipulation condition GN, the students had to check the sentences to be true or false in a given, static geometrical world. In the geometrical, manipulation condition GM, the students had to make geometrical worlds in which they had to check the sentences to be true or false. In all conditions, students were allowed to use scrap paper if they wished.

Procedure. The experiment was held in three consecutive sessions; the first (pre test, instruction and general training) and the second (exercises and post test) on two successive days and the third (retention test) three weeks later.

The first session started with an introduction after which the students had to complete a questionnaire about their previous education in mathematics and logic. This questionnaire was followed by a pre test, which consisted of eleven puzzles measuring the knowledge of the For this, model progression was used, an idea introduced by [START_REF] White | Causal model progressions as a foundation for intelligent learning environments[END_REF]. One of the general principles of model progression is to structure the rich information source and to keep the environment manageable by not introducing too many ideas at one time. Model progression entails starting with a simplified version of a model and gradually offering more complex versions of the model. In this case, the model was the field of predicate logic. The concepts were introduced in the following order: (a) predicates and constants, (b) connectives and parentheses, (c) quantifiers and variables, and (d) conditional. If the students had any questions, assistance was given by one of the experimenters present.

The second session started with six problems that had to be solved by the students in the three conditions. The first two problems were presented to refresh the knowledge acquired the day before. Consequently, four exercises addressed the conditional. In all the exercises students were asked to translate Dutch sentences into first-order logic and to check the truth of the sentences in the geometrical or sentential world. After the students had completed the exercises, a post test was administered. In this post test, the students were again tested on their knowledge of the conditional. The post test consisted of the same type of items as used in the pre test. Also the second questionnaire in which the instruction and Tarski's World was evaluated, was administered. Three weeks after the experiment, the students had to return for the retention test.

This test consisted of comparable items as were used in the pre-and post test.

Results

Reliability

The reliability of pre-, post-and retention test, as measured with Cronbach's α, was α = .49; α = .68; and α = .75 respectively. Deleting items from the test did not lead to significant higher reliabilities.

Pre-, Post-and Retention Tests

Table 2 shows the means and standard deviations on the tests for the three conditions GM, GN, SN. The maximum score was 11. Scores on the pre-, post-and retention tests increased significantly for condition GM (F(2, 42) = 3.85, p < .05), condition GN (F(2, 42) = 7.91, p < .01), and condition SN (F(2, 42) = 11.39, p < .001).

-

---------------------------------- Insert Table 2 about here -----------------------------------
To study the effect of the visualisation variable, the SN-and the GN-condition were compared. To study the effect of the manipulation variable, the GN-and GM-condition were compared. In both cases there was no difference between the conditions on the pre test (F(1, 42) = .27, p > .05 and F(1, 42) = .03, p > .05 respectively). The use of visualisation did not yield significant differences between the conditions SN and GN on the post test (F(1, 42) = .02, p > .05) and on the retention test (F(1, 42) = .00, p > .05). Neither did the manipulation of the objects in the world show a significant difference between the conditions GN and GM on the post test (F(1, 42) = .40, p > .05) and on the retention test (F(1, 42) = 1.29, p > .05).

Process Data

The students' actions, which were stored in the log files, are summarised in Table 3. As can be seen, condition GM distinguished from condition GN and SN on several aspects: (a) students in condition GM needed more time to complete the experimental problems than students in -----------------------------------

Insert Table 3 about here -----------------------------------All students were allowed to use scrap paper. It turned out that 59% of the students in the SN-condition made use of this possibility. In all cases, the paper was used to draw the given, sentential world. Students in the other two conditions did not use the scrap paper.

Questionnaire

At the end of the experiment, 82 % of the students stated that they enjoyed working with the programme. Furthermore, 72% of the students in the GM-and GN-condition stated that the concrete, visual representation made it easy to work with the logical formulae.

Discussion

Reliability

The fact that the reliability of the pre test was lower than the reliabilities of the post-and retention test is explained by the small number of correct answers on the pre test (M = 4.24, SD = 1.73). Apparently, the pre test was difficult, so that the students may have been guessing when answering the items, which has a negative influence on the reliability of the test.

Learning Results

The scores on the pre-, post-and retention tests clearly show that all students profited from the instructional conditions. The students were able to solve significantly more logic problems correctly on the post-and retention tests in comparison to the pre test. Because the items of the post-and retention test also comprised the Wason selection task, the knowledge and skills acquired in Tarski's World were transferred to very different problem situations. This is evidence that far transfer is possible. The results show that, although scores on the retention test are still rather low, even non-technical students are able to do better on the difficult items of the transfer tasks after instruction. The authors suppose the findings are fostered by the advantages of Tarski's World. In this learning environment, it is easy to construct logical expressions, as the programme automatically shows parentheses, commas and the number of arguments that go with a predicate. This allows the student to quickly concentrate on the conceptions and operations of logic. Also, the sentences are easily checked on syntactical correctness by using the inspector module. The worlds allow the student to visualise the objects and their relationships to one another and to test the truth of logical expressions in a given world. Learning occurs by successively testing logical formulae in worlds and by the immediate feedback that is given to the student.

The information on the pre-, post-and retention tests suggests, that in the interval between administering the post-and retention test, the students continued reflecting on the logic problems, which had a positive effect on the learning results. The scores on the retention tests were significantly higher than scores on the post tests. The students needed time for integrating their newly developed knowledge with their existing knowledge.

Though for all conditions the scores on the post-and retention tests increased, no differences were found between the three instructional conditions. In the experiment, the instructional variables, visualisation and manipulation, did not influence the test results. The authors assume that this result is probably due to the time spent on the instructional conditions. The students had to solve only six introductory problems. The number of steps in solving the problems was small and the students may have imagined a world and remembered the cognitive steps they made without the need for help from visualisation and/or manipulation. Also the use of scrap paper may have provided support that interfered with the visualisation variable. In a next experiment, the complexity of the problems will be increased to study the relevance of the variables for instruction.

Cognitive Processes

The process data of the students in the geometrical condition GN did not differ much from the process data of the students in the sentential condition SN. This might be due to the design of the SN-condition. By introducing scrap paper, the students in this condition were still able to use visualisations, if they needed it. In this condition, 59 % of the students used this opportunity, especially when the problems became more complex. This shows that most students need a geometrical representation in which their problem solving process can be made concrete and in which steps can be retained instead of keeping these into working memory. The other 41% of the participants did not use the scrap paper to solve the problems. They were apparently able to use mental objects for this situation instead of perceivable objects.

The process data of the students in the manipulation condition GM clearly differed in number of checks and amount of time used from the data of the students in the non-manipulation condition GN. Because of these differences, the working method of the students in condition GM was given a closer look. Three findings will be discussed here.

First, students in this condition had more freedom to explore the different combinations of sentences in worlds. The students were expected to manipulate the objects in the world and to infer the behaviour of the conditional by induction. However, this was not what happened. In the exercises, students first translated the Dutch sentences into first-order logic. Then, they constructed a world that matched the first sentence. However, if it was the case that the world had to be changed to have the second sentence match the world, the students only added objects to the world. They never removed objects or started all over, not even when the problem could then not be solved correctly.

Second, it appeared that students played around in such a way that they were confronted with the subject matter they already understood, but that they did not confront themselves with the subject matter put in a new situation. They were tempted to stay in familiar situations, even when given freedom in exploring. This can be concluded from the high amount of checks in situations in which both the antecedent and consequent were true (1 → 1). They kept on the safe side, repeatedly checking things they already knew, instead of trying out something new. They headed straight for the solution without straying from their path, even if this could have resulted in a better solution. Apparently, more guidance is needed to lead them to less familiar situations.

The last finding was that students in the GM-condition made a world on the basis of information from the sentences. If a sentence was about something, the students put these objects in the world. This is a way of working human beings use in everyday life. They work affirmatively, they do not start sentences using negations, and they do normally not reason about things that are not present. For instance, the sentence "all cubes are large" is a complete nonsense sentence in everyday life when no cubes are present. In logic, however, this sentence is true.

Differences in language between everyday life and logic are explained by the theory of Grice (for an overview, see [START_REF] Gamut | Logica, taal en betekenis 1: Inleiding in de logica[END_REF].

In addition to differences in working method, the students in the GM-condition had an extra difficulty in their instruction. As Tarski's World was not able to recognise Dutch sentences in natural language, the programme only checked whether the sentence in first-order language was correct. As a consequence, the programme did not check whether the logical sentence was the correct translation of the Dutch sentence. Therefore, it could happen that the students thought they correctly translated the sentences, whereas this was not true. If the students then checked the Dutch sentences in the world instead of the logical sentences, it was possible that they deduced the wrong principles. So, students in the GM-condition can have learnt wrong conceptions. The data in Table 3 support this assertion. Students in the GM-condition were more often not aware of making wrong worlds or sentences compared to students in the other two conditions. Table 1 Truth tables of the four connectives negation (¬), conjunction (∧), disjunction (∨), and conditional (→)

Elementary Propositions p ¬p 0 1 1 0 Complex Propositions p q p ∧ q p ∨ q p → q 0 0 0 0 1

0 1 0 1 1 1 0 0 1 0 1 1 1 1 1
Note. p and q denote arbitrary propositions. 0 = false; 1 = true. b mean number of checks per student over all sentences during the experimental exercises.

c mean number of checks per student made on the four possible checks of the conditional (p → q) during the experimental exercises (1= true; 0 = false).

d mean number of final errors students made during the experimental exercises.

e errors indicated by the feedback module of Tarski's World.

f errors not indicated by the feedback module of Tarski's World.

  stated that learners need to act in the environment if knowledge development is to ensue. Knowledge is constructed through actions on objects in the environment. He added to this that the development of knowledge of formal concepts is realised in a different way from the development of empirical knowledge. Piaget distinguished two kinds of experiences: (1) the physical experience that resembled learning in the experimental sciences and (2) the logic-mathematical experience that resembled learning in the formal sciences. The physical experience consisted of abstracting information from the object itself. For instance, a child picking up balls of different sizes experienced different weights and could infer certain general rules from this. The logic-mathematical experience, however, consisted of abstracting knowledge by operating on the objects and not from the objects themselves. In addition to characteristics already present, new characteristics were attributed to objects.

Figure 1 )

 1 Figure 1): (a) the world module in which students could place the objects of a certain size and

  shows a typical example of an item of the Wason Selection Task (1966), as used in the experiment. In the transfer test, one abstract Wason (card)task, two concrete, non-arbitrary Wason tasks and one near-transfer Wason task in a Tarski's World setting was used.

  subjects of several aspects of the conditional. Successively, the subjects received a verbal instruction in which they got an introductory course into first-order logic, as used in Tarski's World. This instruction gave the subjects an idea of what logic can be used for, what Tarski's World can do, what logic operators and quantifiers are available, how these operators and quantifiers can be used and what truth and falsity meant. This, together with some examples was the knowledge the students were equipped with. After the instruction, the students received a general training of about two hours depending on the condition to which they were assigned. During the training the students learnt to work with Tarski's World and with the logic operators.

  condition GN (F(1, 42) = 14.31, p < .001) and students in condition SN (F(1, 42) = 5.42, p < .05); (b) students in condition GM used more checks on all sentences than students in condition GN (F(1, 42) = 10.72, p < .05) and students in condition SN (F(1, 42) = 13.24, p < .001), especially on sentences in which both antecedent and consequent were true (1 → 1); and (c) students in condition GM deliberately ignored less 'indicated' errors, although this was not significant compared to students in the condition GN (F(1, 42) = 3.08, p > .05) and compared to students in the condition SN (F(1, 42) = 3.14, p > .05), and they made far more 'not-indicated' errors, that is errors of which Tarski's World did not indicate they were made, compared to students in the GN-condition (F(1, 42) = 7.78, p < .01) and compared to students in the GMcondition (F(1, 42) = 7.68, p < .01).

Figure 1 .

 1 Figure 1. Tarski's World: world module, sentence module, inspector module and keyboard

Figure 2 .

 2 Figure 2. A typical example of the Wason Selection Task[START_REF] Wason | Reasoning[END_REF].

Figure 3 .

 3 Figure 3. A typical example of a problem in the transfer tests.

a

  mean time students were working during resp. the general training, the experimental exercises, and the sum of these two.

Table 2

 2 Means and Standard Deviations for Each Condition on Pre-, Post-and Retention Tests

	Condition

Note. Maximum score = 11. SN = sentential, non-manipulation condition; GN = geometrical, non-manipulation condition; GM = geometrical, manipulation condition.

Table 3

 3 Summary of Process Data of Students Working in Tarski's World

			Condition
		SN	GN	GM
	time a
	General training	1:41:20	1:28:51	1:41:46
	Exp. Problems	0:24:39	0:21:32	0:30:27
	Total	2:05:59	1:50:23	2:12:13
	# checks
	all sentences b	31.5	29.7	47.4
	1 → 1 c	4.3	3.9	12.6
	1 → 0 c	3.7	3.5	4.2
	0 → 1 c	1.8	2.1	1.3
	0 → 0 c	3.8	4.0	2.1
	# final errors d
	Indicated e	29	27	15
	Non-indicated f	44	44	80
	Note. SN = sentential, non-manipulation condition; GN = geometrical, non-manipulation
	condition; GM = geometrical, manipulation condition.

(table continues)

abstracted from reality and these representations can be operated on mentally. This theory was extended to the acquisition of concepts and rules of logic.

Therefore, it was supposed that for solving logic problems the manipulation of objects would support the development of formal logic concepts and the use of logical operators. By adding and removing objects, by changing size or position, students could see what happened with the truth value of the logical expression they constructed.

To investigate whether operating on objects facilitated the development of formal concepts, two conditions were compared. In one condition, students were given a geometrical world in which they could manipulate concrete objects. In the other condition, students were given a geometrical world in which the objects could not be manipulated. Learning results were again measured by a transfer test. The authors assumed that students who were given the opportunity to manipulate objects would profit more from the environment and better understand the meaning of the logical expression than students who lacked this opportunity. Furthermore, differences in problem solving processes were expected, although the authors did not have specific expectations of the errors made.

Summary

In this study, the effect of two instructional variables was studied. The first variable concerned the extent of visualisation of the subject matter: a geometrically given world versus a sententially given world. The second variable concerned the extent in which the students could manipulate objects. As manipulating objects could only occur in a computer-based geometrical world, the two dimensions, visualisation and manipulation, partly overlapped. As a result, three conditions were administered: (a) the sentential, non-manipulation condition SN; (b) the geometrical, non-manipulation condition GN; and (c) the geometrical, manipulation condition

Affective Reception

The students in the GM-and GN-condition were positive about the use of a world. Of these students, 72% stated that the concrete, visual representation made it easy to work with the logical formulae. Of all students, 82% stated that they enjoyed working with the programme.

World module: the student can construct a world of geometric objects by mouse clicking on the objects on the left of the screen. The objects can vary in shape (Tet, Cube, Dodec), size (Small, Medium, Large) and position (LeftOf, RightOf, BackOf, FrontOf, Between).

Keyboard module:

the student can construct logical formulae by mouse clicking on the keys.

Sentence module: the sentences the student is constructing by using the keyboard module appear in this screen. The computer ignores the semicolons followed by sentences in English. By giving T(rue)s and F(alse)s, the computer gives feedback about the truth value of the sentences in the world given in the world module.

Inspector module: the student can check whether a logical sentence is syntactically correct (WFF?), whether all variables are bound to a quantifier (Sentence?) and whether the formula is true (True?) in the world given in the world module, by selecting a box. The computer gives immediate feedback (, ).

The following statement is given:

If I go to the city today, I will eat an ice cream.

I am going to the beach today and I am eating an ice-cream.

Is the above given statement true or not? yes no that depends


Jelke Van Der Pal

Tessa Eysink

Balancing Situativity

Formality

Balancing Situativity and Formality

published or not. The documents may come

Introduction

Any expectation that a certain training programme may facilitate abstract conditional reasoning is audacious, as it has proved to be extremely difficult to establish such understanding (for a review of the abundant literature, see [START_REF] Evans | Human reasoning: The psychology of deduction[END_REF]). Yet, to facilitate conditional reasoning is precisely one of the promises of Tarski's World [START_REF] Barwise | The language of first-order logic[END_REF], an Interactive Graphical Representation System (IGRS) intended as an introduction to formal logic, with a strong focus on semantical aspects of logic. As the logical theory behind situation semantics (Barwise & Perry, 1983) may have inspired Barwise and Etchemendy to develop courseware in which statements refer to concrete situations, it is not surprising to find [START_REF] Greeno | Productive learning environments[END_REF]

putting

Tarski's World into a situated cognition realm. Situated cognition as it was debated within the field of instructional science in the late eighties [START_REF] Brown | Situated cognition and the culture of learning[END_REF] provided a new escape route from formal education. Within purely formal education, knowledge is taken to be a static set of abstract concepts and procedures which are assumed to be taught by exposing their formal representations to the students in one way or another. In contrast, instruction based on situated cognition notions is designed in such a way that the student can learn in an appropriate social and real-life context, through which knowledge is not transmitted to the student, but constructed through authentic activity and social interaction. Knowledge, therefore, is a dynamic by-product of the individual -environment interaction. Although education based on situated cognition (for a recent overview see [START_REF] Choi | Situated cognition and learning environments: roles, Structures, and implications for design[END_REF] may be refreshing in itself, its novelty is questionable given that it might merely represent another repetition of history in which some formal education is counterbalanced with some naturalistic education. By pointing to Tarski's World, Greeno illustrated, perhaps unintentionally, that the formalist -naturalist debate may not be really important at the instructional level. Tarski's World has formal aspects and yet it elicits situated cognition from students.

A more severe problem is that the psychology behind situated cognition has been neglected. In the perception of instructional scientists, there can only be situated cognition when the learning situation has obvious authentic characteristics. Consequently, for educational purposes, situated cognition seems to be a function of external, nonpsychological variables. This is unfortunate, because tasks can be construed to facilitate situated cognition without being authentic, whereas authentic tasks may hardly seem to involve situated cognition (as in logic). Without a psychological grounding, situated cognition leads to inaccurate claims and mistaken educational implications [START_REF] Anderson | Situated learning and education[END_REF], while the determination of appropriate learning environments and tasks is intuitive. This recognition has led to the formulation of an adapted version of situated cognition, which is labelled the 'developmental situativity theory' ( [START_REF] Van Der Pal | The balance of situated action and formal instruction for learning conditional reasoning[END_REF]. This theory attempts to resolve the troublesome relationship between formal and situated elements of most situated cognition stances [START_REF] Bereiter | Beyond situated cognition[END_REF]. After introducing the developmental situativity theory, the main results of a study focusing on the theory's central research question will be described. Tarski's World will be used for that purpose.

The Developmental Situativity Theory

It is assumed that a situated cognition theory that aims at the relation of formal and situated elements within cognition, action, and learning cannot be accurate without clarifying the psychological processes behind them. A start for a psychological grounding for situated cognition can be found in a number of situated action theories (Greeno, 1989;[START_REF] Clancey | Situated action: A neuropsychological interpretation response to Vera and Simon[END_REF]Greeno & MSMTA Project Group, 1998), which are based on similar assumptions as situated cognition theories but try to provide a more complete model of Balancing Situativity and Formality human activity. The basic assumption is that cognition or action in most everyday life situations is unmediated by representation. In our understanding of the situated action stance, no distinction is to be made between a perception and an object, as the observer can only deal with perceived information. This implies that on a psychological level the perceived information does not refer to the object, it is the object. Activity towards the object is not only based upon but also intertwines with the perceived information (i.e., the object itself). That is why activity is called 'direct' and 'situated'. There is no symbolic layer between perception and action, between an object and an actor. When using the information processing metaphor, there may be several processors for perceiving and acting upon objects, but these "processors coconfigure each other" during "an ongoing perception -action coordination" [START_REF] Clancey | Situated action: A neuropsychological interpretation response to Vera and Simon[END_REF]. It is within reality-as-perceived (a psychological reality) that activity occurs, not through abstracted processing of symbols.

Connectionism has been recognised as a computational pendant of situated action theories [START_REF] Bereiter | Implications of connectionism for thinking about rules[END_REF]cf. Wallner & Peschl, 1993). However, most connectionistic models still need symbols as input (for a promising model see [START_REF] Brooks | How to build complex creatures rather than isolated cognitive simulators[END_REF].

At hand, we have a rough, rudimentary theory about situativity and a promise towards its computationability. However, in its present form it fails to describe human activity accurately as it is evident that we use symbols and representations in external as well as in internal forms. No situated action theorist ignores representations or symbols functionally, but present theories lack a satisfying psychological relation between situated action and representation. To obtain this relation, two sorts of representation should be clearly differentiated-as opposed to their conflation in cognitivism: (a) the first-order physical representation which equals the concept of psychological reality in situated action theory, and (b) the second-order physical representation which will be considered a psychological representation. We will use the developmental psychology of Vygotsky [START_REF] Vygotsky | Thinking and speech[END_REF][START_REF] Vygotsky | Tool and symbol in child development[END_REF] to fil the psychological gap between situated action and representation. Two important Vygotskian notions are of particular interest:

Balancing Situativity and Formality 6 (a) the relation between thought and symbols, and (b) the distinction between everyday concepts and formal concepts. A short outline will now be provided.

The concept of symbols as tools for thought is a compelling and powerful one in situated action theories, but it is important to realise that symbols for everyday concepts (as used in situated action) are not catalysts. During children's development, according to Vygotsky, the use of symbolic tools will change the information-processing (or activation) structure and its possibilities and consequently the meaning of the symbols themselves. During formal concept formation, however, it is assumed that the use of symbols is catalytic: a formal concept does not alter the information processing (or activation) structure.

In their early use of concepts, children try to situate formal concepts. Although this effort is in vain, it enhances children's ability of generalisation (a concrete, situated sort of generalisation, which differs from abstraction). Such generalisations are the basis for 'pseudo concepts', through which students can use the formal concept correctly, although understanding is inaccurate. For example, students may try to situate the formula 'y = ax + b' by substituting the constants with numbers and the variables with a range of numbers as is possible within the external representation technique used (e.g., a co-ordinate system on a blackboard). The effort will never be successful because the representation of a single instantiation is always inadequate: the line is finite and has a certain thickness. Students learn, but do not understand, that it does not matter how far and thick the line is drawn and that the materials on which the representation is drawn are insignificant. Students may thus acquire a pseudo concept of the formula.

The difference between pseudo concepts and real, formal concepts, Vygotsky argued, is merely a matter of understanding. Eventually, the adolescent will understand abstract, formal concepts. In our view, the crucial understanding does not concern formal concepts itself, instead it is a more general insight that formal concepts are not truly meant to be understood at all, at least not in the sense of the fully affective or situated understanding of everyday concepts. This insight opens the way for the adolescent to a pragmatism towards symbols or symbol systems, a tools perspective. The intention and control of formal thinking is well rooted in situated action. This provides formal thinkers with the feeling that they understand what they are doing, even when the meaning of the elements of the process is minimal. The meaning of the result of the thought, a representation, has to be reconstructed, i.e., situated. But this reconstruction is inherently partial.

This particular situated action theory, the developmental situativity theory, (for a complete description see [START_REF] Van Der Pal | The balance of situated action and formal instruction for learning conditional reasoning[END_REF] provides a psychological relation between situated action and formal symbol systems. For a theory in which perception, action, and cognition are not clearly distinguishable, 'situativity' (coined by [START_REF] Greeno | Situativity and symbols: Response to Vera and Simon[END_REF] is considered to be a more appropriate label than 'action' or 'cognition'. The term 'development' encompasses both the global structural changes during childhood as well as the specific learning processes.

Implications of the Developmental Situativity Theory for

Interactive Graphical Representation Systems for Instruction An important aspect of an interactive graphical representation is direct manipulation (e.g., [START_REF] Norman | Cognitive engineering[END_REF], which means that there is no layer of symbols between the representation to be manipulated and the student (as in old command-line operating systems), but that there is merely a physical tool (e.g., a mouse). Since the manipulator may act towards the representations as if they were real-life objects, the representations will belong to the psychological reality of the manipulator. Therefore, these objects are no longer considered to be representational in a psychological sense. They neither refer to objects in the real world nor to abstractions (formal concepts). The (natural) activity towards such objects is at the level of everyday concepts: it is situated. This is evident for graphical representations of natural objects (having colour, size, depth, etc.). For graphical representations of abstract relations or models, it is less obvious. However, a graph of some relation may become an object for a higher-order situated action (not worked out here) provided that the user already understands the relation of the graph to the formal concepts it represents and to other representations, such as a notated formula.

Situativity in such an environment is in itself not very interesting from an educational point of view (cf. Bereiter's argument that basic-level concepts do not need to be instructed, [START_REF] Bereiter | Referent-centred and problem-centred knowledge: Elements of an educational epistemology[END_REF]. The induction of knowledge is easy but will not transcend the psychological reality. This means that everyday concepts may be changed or created during situated activity, but these concepts lack the necessary abstractness for sufficient transfer to other tasks (cf. Norman's discussion about the important but limited use of the experiential mode of thinking, [START_REF] Norman | Things that make us smart[END_REF].

The developmental situativity theory predicts that situated action will benefit from using symbol systems as tools in situations in which everyday concepts are not sufficient for appropriate action. However, formal systems are merely catalytic: they do not work without appropriate situated action. The instructional feedback should apply to situated action that is enhanced by using a formal system. As in fact all activity is situated somehow, this type of situativity will be labelled here as Situativity-in-Domain (SiD).

From a developmental situativity perspective, at least two kinds of IGRSs can be distinguished: (a) systems in which graphics constitute a microworld affording SiD (e.g., Barwise & Etchemendy's, 1990, Tarski's World), and (b) systems in which graphics are intermediate abstract tools (e.g., systems using Law Encoding Diagrams, [START_REF] Cheng | Law Encoding Diagrams for instruction[END_REF].

The experiment reported below focuses on the IGRS of the first kind, Tarski's World.

- --------------------------------------Please insert Figure 1 about here ---------------------------------------An Experiment using Tarski's World Tarski's World is a computer-based introduction to formal logic. It uses a First-Order Language in which predicates have fixed interpretations, contrary to abstract predicates.

The language can be used to formalize statements in natural language about three types of polyhedrons (cubes, tetrahedrons and dodecahedrons), which have properties and relations to each other with respect to spatial positions. Besides this, the user has the possibility to visualize the first-order language in a world. Referring to the interactive graphical representation of the polyhedrons, the natural statements are about everyday concepts and for that matter they can be fully processed in a situated mode. Statements in natural language then are true or false in a specific world. Feedback facilities of Tarski's World can be seen as simulations of the feedback and explanations that logic teachers might provide.

The combination of the interactive graphical representation, the natural language behind the expressions, and the feedback facility allow for SiD. However, instruction facilitating only SiD would not be very powerful as students would not have the necessary tools to be able to work at a more abstract level than that of everyday-concepts.

In Tarski's World, students have to work at an intermediate abstract level because they can use the feedback facilities only by making use of the formal component of Tarski's World. Therefore, situative and formal aspects are combined in a tangled way in Tarski's World (see Figure 1).

In order to see whether both situative and formal aspects are necessary for a good understanding of logic, two impaired versions of Tarski's World will be compared to the complete Tarski's World. The impaired versions will be described below.

Balancing Situativity and Formality 10

The only-situative version of Tarski's World, TW SiD (in which feedback is based on invisible formal expressions, see Figure 2), would facilitate learning according to the induction framework [START_REF] Holland | Induction: Processes of inference, learning, and discovery[END_REF]. Although students receive feedback based on their own graphical representations, it is especially the feedback based on the given graphical representations that confronts students with the logical rules of the 'if -then' statements as opposed to the natural 'if -then' expressions (cf. pragmatic reasoning schemas, [START_REF] Cheng | Pragmatic reasoning schemas[END_REF]. This gives students space to discover the logical rules, a space that students using the only-formal version of Tarski's World, TW F (in which feedback is based on invisible graphical representations, see Figure 3), lack. TW SiD students have no means or tools other than natural language to abstract from it. It will be very difficult to detach and restructure the pragmatic meaning from the natural sentence without formal tools. Therefore, induction in such an only-SiD environment may require much more feedback than can be provided without harming students' motivation.

It is expected that working with formal tools in a situative environment clears the path to induction beyond the near-transfer level. Therefore, TW instruction is expected to facilitate more logical reasoning on Wason selection tasks than the other two instructions, which will not go much further than a control group will do. This stands in contrast to cognitivistic theories (e.g., the induction framework, [START_REF] Holland | Induction: Processes of inference, learning, and discovery[END_REF], which would not predict a difference between TW-and TW SiD -instructions, as one of the twoequivalent-linguistic representations is redundant.

- --------------------------------------Please insert Figure 2 about here ---------------------------------------

Method

Subjects

The subjects were 80 first-year technical science students of the University of Twente (63 male, 17 female; mean age 19.4 years, SD = 1.6) who volunteered for the experiment.

Most students reached high math-levels at secondary school, after which they attended some mathematics and computer programming courses.

Materials

Software. The software used in this experiment was Tarski's World 3.1 completed by an instructional shell and modified to provide an experimental design. The changes were as follows: (1) for the SiD-condition the sentence window was made invisible (see Figure 2) and for the F-condition the world window was made invisible (see figure 3), (2) the entire text was in Dutch instead of English, (3) menu options were disabled, (4) worlds and sentences were automatically loaded and saved when starting and finishing a task, (5) instructions and tasks were given in a window situated under Tarski's World, using Hypercard 2.1, (6) demonstrations of basic Tarski's World operations were given, using the programme Screenrecorder, (7) a help function was added that presented information about every screen element, using Hypercard 2.1, and (8) during the experiment itself the game was disabled.

General Training. The general training included an introduction in first order language and an introduction to Tarski's World with all the aspects that could be used.

Treatment. The subjects in the SiD&F, SiD and F condition received a text about the material conditional (see Appendix A). They also had to practise with six problems that consisted of 18 sentences together. The explanatory text about the material condition has partly been derived from the text that [START_REF] Cheng | Pragmatic versus syntactic approaches to training deductive reasoning[END_REF] used in the rule training condition of their experiment 1, and partly derived from the text about the conditional in the book that goes with Tarski's World [START_REF] Barwise | The language of first-order logic[END_REF]. The six exercises all focused on the material conditional. The subjects in the control condition received neither the text nor the problems.

Tests. There were three tests: a pretest, a posttest and a follow up. All three consisted of four arbitrary Wason selection tasks [START_REF] Wason | Reasoning[END_REF]. For an example, see Appendix B.

Procedure

The subjects were randomly assigned to one of four conditions: (1) SiD&F-condition in which there was situated instruction as well as formal instruction, ( 2 World problems containing 18 sentences in natural language. After this instruction, the subjects in the experimental groups had to do the same posttest as the control group. A week later, the follow-up assessment took place. In this session which lasted approximately five minutes, the subjects only had to solve the four Wason selection tasks.

- --------------------------------------Balancing Situativity and Formality Please insert Figure 3 about here ------------------------------------------------------------------------------Please insert Figure 4 about here ---------------------------------------

Results

Immediate effects (posttest -pretest)

The overall increase in PAC (Percentage All items Correct) is 16% (19% -3%, see Figure 4 or Table 1), which is significant, McNemar binomial test p < .001. However, for individual groups, differences were significant only for the SiD&F-group (30% improvement), McNemar test, p < .05, one-tailed. Differences in PAC increase between all groups were not significant, Kruskal-Wallis χ 2 = 4.63, p = .20. Comparing individual groups revealed only a significant difference between the SiD&F-group and the Control group in favour of the SiD&F-group (25% difference in gain), χ 2 = 4.22, p < .05.

- --------------------------------------Please insert Table 1 about here ---------------------------------------Parametric analysis (repeated measures with contrasts for comparing individual groups) can be done by using the logic index (Pollard & Evans, 1987), which indicates the number of falsifying selections (P or NOT-Q, when the rule has the form of "If P then Balancing Situativity and Formality 14 Q") relative to non-falsifying selections (NOT-P or Q). 1 Logic indices (see Table 2) showed the same pattern as PAC (using repeated measures with contrasts for comparing individual groups). The overall increase of the logic index was 0.31 points, F(1,76) = 19.48, p < .001. No significant gain-difference was found between all groups, F(3,76) = 1.11, p = .35. Comparing individual groups, only the 0.36 points gain-difference between the SiD&F-group and the C-group was found to be significant, F(1,76) = 3.28, p < .05.

- --------------------------------------Please insert Table 2 about here - --------------------------------------Delayed effects (follow-up -pretest)

The 25% overall gain in PAC (see figure 4 or Table 1) was significant, McNemar test, p < .001. Improvement for individual groups was significant for the SiD&F-group (50%), McNemar test, p < .001, one-tailed. The 20% improvement for the SiD-group was not significant, McNemar test, p = .06, one-tailed. PAC-increase differs between groups, χ 2 = 9.40, p < .05. This difference can be attributed to differences between the SiD&F-group and the other groups: SiD&F versus SiD and SiD&F versus F (resp. 30% and 35% more gain for the SiD&F-group) χ 2 > 3.85, p < .05; SiD&F versus C (40% more gain for the SiD&F-group) χ 2 = 7.43, p < .01. Differences in PAC increase between SiD, F, and C groups were not significant, χ 2 < .77.

The overall increase on the logic indices was 0.46 points (see Table 2), which was significant, F(1,76) = 51.75, p < .001. Comparing individual groups, it was found that SiD&F versus SiD and SiD&F versus F differences in gain (resp. .38 and .39 points more increase for the SiD&F-group) were significant at the 5% level, F(1,76) > 4.31, and that the SiD&F versus C difference (0.46 points more increase for the SiD&F-group) was significant at the 1% level, F(1,76) = 6.21. Differences in increase on the logic index between SiD, F and C groups were not significant, F(1,76) < .18.

Discussion

As hypothesised, the results revealed a facilitating effect of the complete Tarski's World instruction on the Wason selection tasks, immediately after the treatment and even stronger after a week (50% of the students in the SiD&F-group did all items correct against 15% of the control group). It is unclear why the delayed effect was stronger than the immediate effect. Conform the hypotheses, only-SiD instruction and only-formal instruction did not enhance logical reasoning, compared to the control group. These findings illustrate the importance of a combined SiD and formal instruction. An only-SiD instruction is not effective, because it lacks the tools to abstract from the experiences beyond the near-transfer level. This demonstrates the incompleteness of cognitivist theories like the induction framework of [START_REF] Holland | Induction: Processes of inference, learning, and discovery[END_REF]. An only-formal instruction is not effective as it only provides tools without the situations in which the tools can be used in a relevant way.

This study signifies the importance of Interactive Graphical Representations when used in combination with formal systems for logic instruction. As logic may not be prototypical due to its abstract subject, it may be objected that the conclusion -formal tools are needed during SiD-can be generalised to other domains. This criticism appears to find additional support in many instructional practices in which it is apparently sufficient only to act directly on the matter (situated learning) for which purpose IGRSs may be developed, or to act merely by using formal symbols (formal education).

Although these learning effects are not contradicted, they are considered to be uninteresting for educational purposes. Most learning processes in everyday life are situated, with concrete-symbolic, situative tools (everyday or basic-level concepts). [START_REF] Bereiter | Referent-centred and problem-centred knowledge: Elements of an educational epistemology[END_REF] argues that it is not necessary to teach basic-level concepts as they are learned without much effort in everyday life. Instruction that is totally based on situated learning, therefore, is a waste of time and effort. Formal symbols (referring to abstract concepts) are useful as tools just because they lack situativity. Even without dual stimulation (situated & formal), some bright students may be able to play with formal systems quite handsomely on near-transfer tasks, without really knowing what they do and why they do it; they act on the pseudo-conceptual level. For these students, formal education seems successful, but what is likely to happen is that they learn to operate tools without understanding. As these students are bright, they may eventually understand the use of these tools in further education or in their jobs, but such a learning process is not efficient and it is ineffective for other students.

Unfortunately, this argument does not make instructional design easier. For instance, it implies the importance of another Vygotskian concept, the 'zone of proximal development' in which instruction is only effective and efficient when the student may be able to understand a concept or to perform a task with some help.

The finding that a certain type of IGRS has a necessary but insufficient functionality for a certain type of learning is important in itself -if only because of the unprecedented learning effect for abstract conditional reasoning-but it fails to demonstrate a precise relation between instruction and learning. The instructional guidelines distiled from situated cognition notions may often be inaccurate or invalid [START_REF] Anderson | Situated learning and education[END_REF], but the developmental situativity theory needs a layer of more specific learning models before an alternative set of guidelines with sufficient specificity can be produced.

something is an orange, then it is also a citrus fruit." It is evident that this statement is equivalent to "If something is not a citrus fruit, then it is not an orange either." It will be equally evident that P → Q is not equivalent to Q → P ("If something is a citrus fruit, then it will also be an orange.") This also holds good for statements concerning overlapping sets. Consider the conditional "If something is a sports car (S), then its colour is red (R)." This statement is only false when elements exist in the white part of the diagram. Now, a black sports car x has been observed. Therefore, car x falls in the white part of the diagram. The conditional is false. Everybody knows this, but if someone would not know a thing about cars, then the truth value of the statement could be found by watching out for sports cars and for other than red cars.

  ) SiD-condition with only situated instruction, (3) F-instruction with only formal instruction, and (4) Ccondition (control group) without any instruction besides the general training. The experiment was held in two sessions. Sessions took place in groups of at most five students in which all subjects worked individually. The first session took about three to five hours. The students had to do the pretest consisting of four arbitrary Wason selection tasks. Then all subjects received the general training in which they learned to work with the programme and received the introduction in logic. After this training, subjects in the control group immediately had to do the posttest consisting of four similar Wason selection tasks. The other three groups received the instructional treatment which consisted of the explanatory text about the material conditional and the six Tarski's

Figure 1 .

 1 Figure 1. Example of a task in the TW SiD&F -condition. 2 The top three windows belong to the Dutch research version of Tarski's World 3.1. The lower window belongs to the instructional shell for the experiment.

Figure 2 .3

 2 Figure 2. Example of an experimental task in the TW SiD -condition. 3

Figure 3 .4Figure 4 .

 34 Figure 3. Example of a task in the TW F -condition. 4

Table 1

 1 Percentages All items Correct (PAC) for each condition Situativity-in-Domain, F = formal, C = Control. n = 20 for each group. Note. SiD = Situativity-in-Domain, F = formal, C = Control. n = 20 for each group. The logic index ranges from -2 to 2.

	Condition

The logic index specifically scores +1 for a P or NOT-Q selection and -1 for a NOT-P or a Q selection. This produces a 5-point scale from -2 to

The logic index can be seen as the extent to which the choices agree with logical choices, but as Pollard and Evans point out, it does not measure any personal logical ability, strategy or insight. The logic index is particularly useful for its potential to measure a change in response other than incorrect -correct. For instance, the improvement of a NOT-P and Q choice to a P choice is considerable (3 points on the 5-point scale), but PAC and the number of correct responses are insensitive to it.

Appendix A: The Explanatory Text about the Material Conditional (translated from Dutch)

The symbol for the material conditional (→) is used to combine two sentences P and Q to form a new sentence P → Q. Such a sentence is called a material conditional. The sentence P is called the antecedent of the conditional, and Q is called the consequent. The logical meaning of P → Q is more rigid than the meaning of conditionals in everyday Dutch. In natural language, P always has a sensible relation to Q. This is not necessary in logic. The truth value of a material conditional is determined only by the truth values of the antecedent and the consequent. This is why the conditional is called material. To put it differently: The logical meaning of a conditional is an abstraction of the natural meaning. A text follows which clarifies the material conditional. Try to understand the material conditional and try to apply it to the tasks that will be presented after the text.

We can come fairly close to an adequate Dutch rendering of the expression P → Q with the sentence If P then Q. Other Dutch sentences that may be translated to P → Q are: Q provided P, P only if Q, and Q when P. At any rate, it is clear that Dutch conditionals, like the material conditional is false if P is true and Q is false. A sentence as: If Wim is at home, then Linda is at the library can be translated into: Home(Wim) → Library(Linda).

Dutch expressions of the form All A's are B's and Every A is a B can be translated into: ∀x (A(x) → B(x)). In logic, this sentence is true when any object either fails to be an A or else be a B. Let P stand for "It is raining." and let Q stand for "The pavement is wet." Then P → Q says "If it is raining, then the pavement is wet." This expression is equivalent to ¬Q → ¬P, saying "If the pavement is not wet, then it is not raining." A way of understanding this equivalence is through the concept of sets. Sets can be represented by Euler diagrams. (Consult the experimenter when the diagrams are not clear.) Let P stand for "Something is an element in the set of oranges (S)." and let Q stand for "Something is an element in the set of citrus fruits (C)." Then P → Q means "If Below are four cards. On each card a letter is printed on one side of the card and a number on the other side. A card never contains two numbers or two letters. Hypothesis:

If there is an A on one side of the card, there is a 2 on the other side. Which of the cards below do you have to turn to decide whether the hypothesis is true or false? Click only on the card(s) that is (are) needed for your decision. Note that the clicked cards will become black. If you want to undo a choice, just click again on the card.