Academic achievement

The role of praise in motivating students

E F F I E M AC L E L L A N University of Strathclyde, UK

A B S T R AC T The motivation of students is an important issue in higher education, particularly in the context of the increasing diversity of student populations. A social-cognitive perspective assumes motivation to be dynamic, context-sensitive and changeable, thereby rendering it to be a much more differentiated construct than previously understood. This complexity may be perplexing to tutors who are keen to develop applications to improve academic achievement. One application that is within the control of the tutor, at least to some extent, is the use of praise. Using psychological literature, the article argues that in motivating students, the tutor is not well served by relying on simplistic and common sense understandings of the construct of praise and that effective applications of praise are mediated by students' goal orientations, which of themselves may be either additive or interactive composites of different objectives and different contexts.

K E Y WO R D S : motivation, person criticism, person praise, process criticism, process praise

Context

It is not surprising that the motivation of students is an important topic (Biggs, 1999;Knight, 2002;Knight and Trowler, 2001;Prosser and Trigwell, 1999) given the claims for its power to influence academic achievement (Brophy, 1999;Graham and Weiner, 1996;Hattie, 2003;Pajares, 2001). In the current context of increasing access to higher education, some students may have little in the way of formal entry qualifications, while others may have varying conceptions of what learning might mean (Dart, 1997;Eaton and Dembo, 1997;Purdie et al., 1996;Schuller et al., 1999). Within such diversity, we cannot assume that all students are well prepared for the particular demands of higher education or that they are necessarily going to learn or perform according to traditional academic mores. It therefore becomes important to understand, for example, why some students complete tasks despite enormous difficulty while others give up at the slightest provocation, or why some students set active learning in higher education such unrealistically high goals for themselves that failure is an inevitable consequence. The resolution of those issues and others like them has its knowledge base in the literature on motivation, which is commonly understood as the study of why people think and behave as they do.

In contemporary social-cognitive models of motivation (Pintrich and Schunk, 2002), motivation is assumed to be both dynamic and context sensitive. No longer is motivation some (quantifiable) characteristic between the endpoints of 'motivated' and 'not motivated' but rather is multifaceted. However, not only can students be motivated in multiple ways but these motivations are inherently changeable and domain specific, rather than being stable, personal traits. A further assumption is that students' motivation, learning and achievement are mediated by their selfregulatory activities. Taken together these assumptions have two corollaries. One is that it is a misconception that some students are motivated but others are not. Whilst there may be students who are not motivated to behave in the way that tutors would like them to behave, by definition, students are motivated when they choose goals and expend effort to achieve them (Wolters, 1998). The other is that it is a misconception that one person can directly motivate another. Motivation is a subjective experience (Brophy, 1999) for which there is no guaranteed equivalence between self-reports of the experience and behaviours that allegedly evidence the experience (Elliot and Hufton, 2003). That academic motivation is neither a unitary nor a static attribute which students either do or do not 'have' but is actively shaped by their perceptions of control over the learning environment, their metacognitive processes, their perceptions of ability and their beliefs about the utility of effort (Leo and Galloway, 1996;Prosser and Trigwell, 1999) means that understanding whether students are eager or disinclined to learn, whether they embrace or shy away from challenge and whether they persevere or give up in the face of a difficult task defies simple and straightforward explanation. While the sheer plethora of motivational constructs that have emerged in socialcognitive models may well be theoretically important to the psychologist, they may be confusing and less than helpful in developing applications to improve academic achievement. The need for tutors to be able to harness their understandings into practices that relate motivation to achievement, is a perfectly reasonable one and is one which this article attempts to satisfy. Notwithstanding the importance of self-efficacy (Bandura, 1997), intrinsic motivation/interest (Krapp, 2003), attributions (Weiner, 1986) and a myriad of other explanatory variables posited for motivation, this article will focus on the importance of praise, since it is an application that is within the control of the tutor, at least to some extent. Furthermore, given that it is what teachers know, do and care about (Hattie, 2003) that can be so powerful in academic achievement, it would seem important to be clear about the power of praise.

What is wrong with praising people?

Because both common sense and the literature tells us that feelings of helplessness and contingent self-worth can be debilitating if not downright damaging, we have come to believe that praising people is essentially 'good' and may also connote the complementary belief that criticism is unhelpful and makes people more vulnerable. Given the evidence for the use of praise (Koestner et al., 1987(Koestner et al., , 1989;;Schunk, 1994) and for the phenomenon of learned helplessness (Dweck, 1975;Dweck and Reppucci, 1973;Seligman et al., 1995), it is not surprising that tutors believe that giving praise may be necessary for academic achievement (Mueller and Dweck, 1998). The extensive literature in behavioural psychology on the use of praise to shape desirable/appropriate behaviour largely describes success-only procedures and eliminates the experience of error for the student. The documented procedures are premised on the view that the experience of errors renders the context/materials/event aversive and elicits negative emotions on the part of the student. Although there is ample evidence that errors per se do indeed have adverse effects on the performance of some students, and that success motivates them, there is also evidence that 'positive reinforcement' may actually result in students avoiding intellectual tasks or not persisting in the face of difficulties (Ames and Ames, 1984;Deci and Ryan, 1985;Dweck, 2000). Further, extrinsic reward systems are associated with a decline in interest or liking of academic work, a marked anxiety about cognitive outcomes and a perception of self as being externally rather than internally controlled (Ryan et al., 1985). The issue thus turns on whether the most effective way of overcoming helpless or negative reactions to failure is to eliminate failure from the students' experience or to teach students how to deal with it since simplistic attempts to empower students can result in their fearing failure, avoiding risks and coping badly with setbacks (Kamins and Dweck, 1999;Mueller and Dweck, 1998). Moreover, the literature on the psychological construct of resilience (such as Howard and Johnson, 2000;Howard et al., 1999;Luthar et al., 2000;Masten, 2001) suggests that students learn to deal effectively with academic setbacks, stress and study pressure by receiving specific help with the particular learning difficulties that they are experiencing, rather than from social and emotional support/comfort. In other words, the errors that give rise to academic setbacks, stress and study pressure should not be viewed as pathological symptoms from which students have to be shielded but as the focus for subsequent academic improvement and achievement. This then suggests that while our common sense views about praise leading to academic achievement are intuitively appealing, they are not altogether helpful to the tutor who needs a much finer-grained understanding of how to help students to embrace learning and to be resilient in the face of obstacles/challenge.

Praise in a social-cognitive perspective

The social-cognitive perspective on motivation includes general approaches to, and evaluations of, engagement in tasks in academic learning contexts, otherwise known as goal orientations. Although these psychological dispositions towards achieving one's objectives have been described using a variety of constructs, each carrying different nuances of meaning, Pintrich (2003) summarizes the evidence in terms of the goal (mastery or performance) and the focus (approach or avoidance) that students might have in mind, to yield four different goal orientations: mastery-approach, masteryavoidance, performance-approach, performance-avoidance as indicated in Table 1.

This schematic prototype should not be taken to mean that there is comprehensive and incontrovertible evidence for each of the cells. Indeed Pintrich (2003) makes clear that the mastery-approach orientation is the one for which there is most evidence. Notwithstanding the debates that attend the veracity of the approach-avoidance focus or the doubts about the , 1997;Miller et al., 1993;Stipek and Gralinski, 1996).

M A C L E L L A N : A C A D E M I C A C H I E V E M E N T 197

Different views of ability and their implications

Briefly, Dweck (2000) reports two views of ability: one as the fixed entity theory of intelligence which assumes that intelligence is unevenly distributed throughout the population and is fixed and finite; and the other as the incremental theory of intelligence which assumes that ability is an ever increasing repertoire of skills and knowledge effected through one's instrumental behaviour. These different theories of ability have ramifications for how students (and tutors) behave. Students with a fixed, entity theory of ability value intelligence (perhaps because it is a socially esteemed commodity) but believe it ultimately limits achievement. The contradiction in this belief (of valuing that which they themselves allegedly may not amply possess) results in their desire to avoid taking risks, refrain from accepting challenges and conceal their ignorance because they believe that to do any of these and fail is to make very public one's lack of intelligence.

Rather, students with a fixed, entity theory of ability choose 'safe', selected tasks at which they will succeed (because there is no risk of failure). These students are thus concerned to demonstrate how smart or bright or clever they appear to be, through exemplary performance, whilst possibly sacrificing valuable learning opportunities because they never really find out if they could do more. The significance of flawless performance is in the perception that the performance is a direct reflection of intelligence. Because less than exemplary performance is deemed to reflect poor intellect (without any regard for the nature of the task, the individual's inclination to attend to the task or whether the level of performance was typical/atypical), any academic setbacks are viewed very negatively, as decisive judgements about the person(s). These views are further corroborated in the fixed, entity view of intelligence that effort is unnecessary (because if you're smart, or clever, or brainy you shouldn't have to work hard) and/or ineffective (because hard work does not compensate for low ability). This view of intelligence as a fixed entity is not without cost, however. Because of the importance of flawless performance and the need to avoid tasks when this cannot be assured, students handicap themselves through leaving tasks till the last minute, through finding displacement activity, or through withdrawing from the task. Such self-defeating behaviour has the cumulative effect of lower levels of achievement than might have obtained had students accepted, and engaged in, the challenges offered.

The alternative view of ability, as malleable and incremental, also values intelligence but sees it as a potential to be developed. Students with this view give primacy to learning (rather than to performance) even if there is a risk of making errors. These students do not perceive failure as either a personal judgement or a negative statement about people but rather as an indication of insufficient effort or inappropriate strategy choice. The recognition that errors are an inevitable part of learning which can lead to strategy diagnosis and remediation can allow students to make progress because they are not shackled with the worrying doubt of whether they have or do not have enough ability. Since ability can always be improved, through the power of effort, task difficulty is not viewed as an insurmountable obstacle but rather as an opportunity for increased and improved learning. Such a view of intelligence, unlike the previous view, means that students need not be concerned with face-saving, self-defeating strategies and since they are willing to expend effort (because they see effort as the necessary mechanism through which learning actually happens, and setbacks as the opportunities for further learning) their achievements can outperform those of students with a fixed view of intelligence.

From the account given here it is rational to conclude that a malleable, incremental view is more conducive to promoting academic achievement and that the alternative, a fixed entity view of ability, binds its proponents into unhelpful motivational patterns of behaviour. However, the complexity of psychological functioning does not necessarily mean that what is rational is what really informs our beliefs and behaviour. While young learners hold unrealistically high ability beliefs (Nicholls, 1979) in that they neither modify their beliefs following failure (Parsons and Ruble, 1977) nor consider their performance in relation to that of their peers (Ruble et al., 1976), older learners become more realistic about their ability beliefs as they draw on the evaluations of self, peers, tutors and parents (Nicholls, 1978(Nicholls, , 1979;;Rosenholtz and Rosenholtz, 1981). It would seem that in the course of development people learn to subscribe to the fixed entity view of intelligence, even although this is a limiting and debilitating view. In other words, in the process of moving towards the normative conception of difficulty, concerns with the performance of others lead students to become increasingly concerned with the reasons for, and presentation of, their own performance relative to that of others. While students' views of intelligence may remain stable if 'uncontaminated' by other influences, they are susceptible to intervention (Robins and Pals, 2002). One type of intervention documented to effect some change in viewpoint is the type of feedback (praise or criticism) given to students (Dweck, 2003).

What Dweck and her colleagues found

In a series of studies (such as Kamins and Dweck, 1999;Mueller and Dweck, 1998;Robins and Pals, 2002) students experienced different forms of praise and criticism for their achievements. Summarizing and extrapolating from the various studies, the following can be reported.

1. There are differences between the effects of person praise and process praise. In person praise the students are told that they are good or smart or wonderful. In other words, the praise is directed at the students globally as when told, 'You're a good student', 'I'm very proud of you' or 'You're very good at this'. In process praise, the feedback is directed at the effort or strategy used by the students as when told, 'You tried really hard' or 'You found a good way to do this. Can you think of other ways that would also work?' 2. There are differences between the effects of person-oriented criticism and strategy, or process criticism. As for person praise, person-oriented criticism expresses a global evaluation of the student's performance as when told 'I'm very disappointed in you' or 'You didn't do that very well' after some task has been incompletely carried out. In process criticism, like process praise, students' attention is drawn to the specifics of what is incomplete about the task as in, 'This piece of work has no conclusion and the references are inconsistent', but this is immediately followed up with, 'Maybe you could consult the library help sheets to find the different ways of referencing and come to a clear decision about which one you are using and why'. So this form of criticism contains two essential features: drawing attention to the error/mistake and asking the student to think of an alternative solution strategy. 3. Students who experience person-oriented criticism give significantly lower ratings than do those who experience process criticism to the quality of their products, the satisfaction with their performance in making the product, how capable they consider themselves to be in making the product and their willingness to resolve setbacks. 4. The authors draw a number of conclusions from the findings:

• having received person criticism in the past increases the likelihood that current mistakes are seen as failures whilst having received process criticism leaves the individual able to generate constructive solutions to errors;

• not only does person criticism encourage one to view one's performance less positively but such criticism negatively influences self-perceptions, causing feelings of being 'not good', 'not smart', 'not nice'; • the type of criticism experienced influences not only one's affect and self-perception but also influences subsequent behaviour in terms of persisting with or desisting from the setback.

5. The findings on the effects of praise mirror those from the studies on the effects of criticism leading the authors to conclude:

• students experiencing person praise when they succeed are the ones least likely to attribute their achievement to low effort so person praise teaches students to make inferences about their ability from their performance whilst process praise seems not to trigger this inference; • person praise does not appear to lead to obvious motivational deficits immediately after a successful and well-received task performance but it would appear to leave students extremely vulnerable to subsequent setbacks; • in response to setbacks, person praise encourages students to find out how their performance compares with that of others while process praise encourages students to seek information/problemsolving strategies on how to remedy failure; • students experiencing person praise appear to overstate their actual achievements when reporting their performance to peers whilst those experiencing process praise do not need to artificially inflate reports of their achievements.

In other words, person praise generates in students a fixed view of intelligence, whilst process praise generates a malleable view of intelligence (with all the connotations that each of the views carry). It is perhaps worth noting that the studies had included a third type of praise/criticism: that of outcome praise or outcome criticism. This type of praise targeted the behaviour and was included because of the folklore which advocates that one should focus on the behaviour to depersonalize critical feedback. However, being told that one's behaviour was on or off the mark yielded ratings that fell between person and process. This suggests that while outcome feedback may be preferable to person feedback, it is not as effective as process feedback. These findings caution us against over-simplistic interpretations of the use of praise and criticism because they give the lie to the belief that you can help students to be resilient and withstand difficulty through indiscriminate use of praise and through protecting them from exposure to criticism. Rather, what the evidence suggests is that if praise and criticism are in terms of task analysis, strategy formulation and challenging, ipsative standards instead of global, whole-person evaluations, then it will serve to motivate further endeavour. Person-oriented praise, however, while positively and enjoyably experienced by recipients in the immediacy of the successfully completed task, leaves students vulnerable in the face of subsequent difficulty because they interpret such praise to be deep-seated, intractable and all important. Person-oriented praise is therefore a very fragile motivator because its frequent use will encourage students to protect positive feedback by avoiding challenging tasks, thereby orienting them to performance goals.

Implications for tutors' practices

These complex but subtle differences between different types of praise and criticism tell a consistent story. Feedback that centres students as people confirms a belief in fixed intelligence with all of its vulnerabilities while feedback that focuses students on effort, challenge or strategy promotes a belief in malleable intelligence with all of its benefits. This implies that some tutor/classroom practices may be helpful and others unhelpful.

• The dynamic, situationally determined nature of motivation means that there can be no guaranteed prescriptions for giving effective feedback.

The attention that students pay to feedback, the sense they make of it and the relevance/appropriateness of the feedback for subsequent learning can be understood only in the context of the specific problems and specific students. • In the social-cognitive perspective, motivation is understood as being within the student's own self-management of learning as well as being within the influence of tutors. Thus in situations where the material is regarded as boring but important to learn, as interesting but unimportant to learn or as difficult/effortful to learn, motivation to persist is clearly a complex phenomenon requiring detailed definition of the possible motivational processes involved. • While a malleable view of intelligence is preferable, it should not be assumed that students who are achieving academically will continue to do so in the absence of feedback. Without appropriate feedback to promote increasing academic autonomy, students may well get stuck in, or even slide from, existing learning trajectories. • When praise is given, it is most helpful when it is process praise. Praise for low-challenge, low-effort, error-free success tells students that they are praiseworthy only when they carry out tasks quickly, easily and perfectly. This does not enable students to embrace learning and challenge. Indeed Dweck goes so far as to advise that the tutor might apologize to a student -'sorry I wasted your time with a task that is too simple for you' -when tasks have underestimated the challenge level. • Students need to learn to tolerate ambiguity and confusion as integral stages in learning so the tutor can helpfully model how to confront difficulty -'well that strategy didn't work','what can we try now?','that tells us we used the wrong strategy'. Such feedback makes clear that both tutor and student use mistakes as the platform from which to launch an alternative strategy. • Process praise is helpful following success (to indicate that success can be repeated) and failure (to overcome mistakes). But praise following mistakes and errors has to be carefully delivered. If the praise is of the variety, 'Well done, you did your best' the message conveyed is one of pity, thereby confirming to the student that the mistakes were due to fixed ability and unavoidable, and not the responsibility of the student. Equally, it is not helpful merely to tell students to try harder because this conveys no information about how effort might be expended, and is tantamount to person praise. This in turn implies that the process praise is demanding of the tutor: both to steer students towards the malleable intelligence view and to enable students to develop the strategies which will support them. • The importance of strategy choice and effort on the part of students can be effectively communicated to students by careful tutor judgement of how much help to give in the enactment of class tasks and learning activities. While this is not to suggest that the tutor should desist completely from providing help, it has to be recognized that the nonjudicious provision of tutor help, particularly if unsolicited by students, can imply that student difficulty is due to low, fixed ability. On the other hand, by requiring students to engage in the task, and make their own sense of it, they are learning that sufficient effort might be needed from them. This then suggests that as tutors we be clear about what we are meaning and doing when engaging in such ubiquitous but ill defined tutor tasks of 'monitoring', 'helping' or 'providing support'.

Conclusion

For tutors in higher education the motivation of students is an extremely important issue. No sane educator would deny this assertion. However, there is room for debate about how to effect motivation that drives high quality learning and achievement. The social-cognitive perspective characterizes motivation as dynamic, contextually sensitive and personally variable. Thus the historical and prevalent belief that motivation is achieved and allegedly enhanced by the excessive and indiscriminate use of praise is an incomplete account of how feedback might be used to promote academic achievement. Essentially, motivation is a complex construct which is regulated by students as well as being influenced by tutors. Within such strictures, tutors may be able to influence students' motivations through the feedback they offer. Such feedback can take the form of praise or criticism but the evidence suggests that the value of either is within quite specific limitations.

Table 1

 1 Goal orientationssuperiority of mastery goals, most of the literature acknowledges the contribution of Dweck and her colleagues on ability-outcome linkages in which alternative views of ability 'explain' academic achievement. Within a social-cognitive perspective, therefore, praise cannot be understood as a stand-alone application to be enacted solely through a set of procedures but has to recognize that the effects of praise are mediated by students' goal orientations, which of themselves may be either additive or interactive composites of different objectives and different contexts (Kaplan and Midgley

		Approach focus	Avoidance focus
	Mastery goal(s)	Mastering the task,	Avoiding misunderstanding,
		learning or understanding not learning or not mastering
			the task
		Gauged against standards Gauged against being correct
		of self-improvement	in relation to task
		progress
	Performance goal(s)	Being superior, smart or	Avoiding inferiority or
		'the best'	appearing stupid
		Gauged against normative Gauged against normative
		standards of getting	standards of not getting worst
		highest grades or giving	grades or giving the poorest
		the best performance	performance

Source. Adapted from Pintrich, 2003.

M A C L E L L A N : A C A D E M I C A C H I E V E M E N T

Biographical Note

E F F I E M AC L E L L A N is a Reader in Education at the University of Strathclyde and is Vice-Dean (Research) in the Faculty of Education. A Chartered Psychologist, she has research interests in learning, teaching and assessment and is engaged in a number of projects which seek to clarify what 'good practice' in higher education might mean. Address: Department of Educational Studies, University of Strathclyde (Jordanhill Campus), Southbrae Drive, Glasgow G13 1PP, UK. [email: e.maclellan@strath.ac.uk]


Christopher Laing

Alan Robinson

Veronique Johnston

C H R I S To P H E R L A I N G

V E Ro N I Qu E J O H N S To N

Managing the transition into higher education An on-line Spiral Induction Programme

come

Introduction

Studies undertaken by Lowe and Cook (2003), Cook and Leckey (1999) and Ozga and Sukhnandan (1998) suggest that UK student expectations and skill-sets may not be appropriate for the requirements of successful study in higher education. Student expectations of the teaching and learning environment in higher education are partly driven by their previous educational experiences, their life experiences (Ozga and Sukhnandan, 1998), and their level of pre-university preparation (Lowe and Cook, 2003). In addition, Cook and Leckey (1999) point out that students will have developed study skills and learning strategies matched to the teaching and learning styles experienced in secondary education, and that these skill-sets are at odds with independent learning styles expected in higher education.

Consequently, the mismatch in expectations and lack of preparation may mean that many prospective students may find the transition from secondary education to higher education difficult, in that they neither have the skills necessary to become independent learners, nor the means of acquiring these skills (Lowe and Cook, 2003;Cook and Leckey, 1999). Retention and progression rates may suffer unless the reality of the student experience can be aligned with their prior expectations and perceptions. As Yorke (1999) points out, for many students it is important that this transition period is managed -and managed well; students must be successfully inducted into the expectations and requirements of higher education -a view echoed by the Select Committee on Education and Employment (2001), who suggested that support at the beginning of a student's academic life was critical to their success. This is especially important, given the diverse range of students now entering higher education, and that nearly two-thirds of those students who withdraw do so during the first year of their course of study (Yorke, 1999).

Does a typical 'Freshers Week' solve this problem?

In the UK, induction is commonly known as 'Freshers Week', and while elements may vary between universities, they share common themes, with presentations on course structure, library and IT systems, Student Associations, and central Student Services. Unfortunately, 'Freshers Week' is often a short period of intensive information exchange, and much of the information is 'dull', particularly when the information exchange is 'passive' (Edward, 2003). Consequently, the effectiveness of these sessions in helping students adjust to higher education may be limited. However, what is perhaps of more importance is that there is also insufficient time to develop social and peer support groups in a more structured manner. Compared to the UK, the majority of US induction (or orientation) programmes are often timetabled into the first semester. The purpose of these orientation programmes is to help students adjust to, and hence participate in, the university environment (Perez, 1998). Perigo and Upcraft (1989) report that such programmes can aid in the retention of students, while Perez (1998) concludes that participation is central to the student 'connecting' with the institution.

Within the UK, various educational commentators (Yorke and Thomas, 2003;Thomas, 2002;Select Committee on Education and Employment, 2001) have suggested that institutions should have a more proactive induction framework, and given that some students are lacking a 'cultural' understanding of higher education (Yorke and Thomas, 2003), this is even more of an imperative. It should be noted that those students whose previous educational experiences have not given them a sufficient understanding of higher education must not be 'blamed'; instead they must be helped to develop the necessary academic study skills (Yorke and Thomas, 2003;Thomas, 2002). Furthermore, induction should be seen not as an event, but as a process that has a more student-centred approach, promoting peer group interaction and academic preparation (Lowe and Cook, 2003), and encouraging social and academic integration (Tinto, 1993).

Providing a more student-centred approach to induction

Southampton Institute, in an attempt to provide a more student-centred approach to induction, initiated a Spiral Induction Programme (Laing and Robinson, 2003a), of which the on-line Spiral Induction Programme (or onSIP) is a particular implementation. The primary purpose of the Programme is to provide an opportunity for students to work on collaborative activities in an informal manner with as many of their fellow students as possible. Robson (1998) suggests that collaborative activities may facilitate levels of peer support and peer learning not offered by teacher-centred approaches. The informal and relaxed manner of the sessions helps students to adjust to the requirements of the university's teaching and learning environment, to deal with change, and to develop those generic communicative and interpersonal skills (e.g. listening, discussing and group working) that are essential in any working environment.

The initial programme runs for the first six weeks, but is extended throughout each academic year with activities at key times to ensure that students receive timely information and support. For example, sessions on: (i) regulations and methods of assessment; (ii) procedures relating to cheating and plagiarism; (iii) mitigating circumstances procedures; and (iv) report writing, standard referencing, etc. are scheduled before assignment hand-in dates. Other sessions on: (i) team working; (ii) study and time management; (iii) personal development; and (iv) skills acquisition are held during the initial six weeks and followed up as required. In some cases, formal sessions may not be needed. The student cohort also has an opportunity to develop additional learning activities for future student intakes. These could be about anything the student cohort considers important to their learning, as long as the learning activities are designed to help students understand more about their own learning process.

The onSIP provides students with a series of web-based interactive learning activities, designed to encourage the student to take responsibility for his or her own learning, and is primarily concerned with the process of learning rather than content. The activities provide the students with an appropriate challenge from the start of their course, and are developed on the basis that it is 'what the student does' (Biggs, 2003) (or doesn't do) and 'how they perceive' what they are doing that is important.

The onSIP records the student's attempts, which forms part of the evidence that they have participated in the learning activities. If a student does not attempt an activity then they are considered 'at risk'. The interactive nature of the activities provides both staff and student with feedback on if and what additional support may be required.

For each activity students should monitor and record: (i) preparation for the session activity; (ii) attendance and participation in the session activity tasks; (iii) completion of the post-session tasks; and (iv) a rating, on a scale from 0 to 5, of their perception of how well the session outcomes matched their expectations. The academic facilitating the sessions also rates the students from 0 to 5 as follows:

A rating of 3 to 5 (with evidence of the student completing all activities)

indicates that the student should be successful if they continue their efforts in their academic subjects. A rating of 1 to 3 (with evidence of attempting all activities and completion of a majority) indicates that the student may need further assistance in order to achieve success. The two-way feedback provided via the monitoring and rating process, by both the student and tutor, initiates a dialogue centred on the evidence generated from attempting the activities. All other cases (i.e. lack of evidence of attempting activities, or a difference between the student's and tutor's ratings) may indicate a lack of participation and engagement in the course, or a mismatch between student and Institute expectations. These conditions initiate a more structured intervention strategy.

An important dimension of the onSIP framework is the attempt to bring a student's perceptions and expectations into line with the reality of academic study. The onSIP attempts to initiate the student into a process of learning to learn, in which self-assessment and self-diagnosis are important elements within the whole process, the underlying message being, 'if you recognise the need to obtain new skills, undertake activities needed to acquire those skills, assess your learning needs, and clarify areas of support needed, then the probability of completing a successful academic career is enhanced' (Robinson and Udall, 2003). By allowing students to take control of their learning, students become empowered and autonomous, from which a sense of learning ownership and hence learning independence can be developed.

In essence, students are encouraged to take responsibility for their own learning within a network of structured support. The onSIP provides a realtime analysis and indication of those students who may be 'at risk'. This early identification of problems (by both staff and student) enables the appropriate targeting of timely support, allowing for rapid implementation of action plans. The automated analysis technique enables the whole cohort to be monitored, reducing the workload overhead in initially identifying those students who may be 'at risk'. The means by which onSIP identifies such students are presented in the following section.

Implementing onSIP

The study cohort consisted of 80 1st year computer science and business information systems students. They were divided into four 20-cohort groups, each with timetabled sessions for the first six weeks. The timetabled sessions were voluntary. The sessions were informal, with no formal teaching structure. During each session, onSIP provided the students with a series on learning activities. These activities included a learning style inventory, skills audit, etc. The student undertook the various activities, and onSIP recorded their attempts. After each attempt, onSIP provided each student with automated feedback. Apart from recording the students' attempts, onSIP also used a Napier questionnaire (Johnston, 2001) and attendance at onSIP sessions to identify those students who may have been 'at risk'.

A simple example will illustrate how the onSIP uses a Bayesian method to incorporate information, and thereby identify those students who may be 'at risk'. Readers should note the following assumptions: Student B has some academic ability x, this ability can be represented as a distribution, and this distribution reflects the current 'belief' in their possible academic success. This prior distribution can have the form (Hays, 1980):

p(x ∈ A j ) (1)
Additional information (relevant to student B) will have some bearing on their possible academic success.

For example, let this new evidence be the results y from the Napier questionnaire. Results from previous Napier questionnaires can be used to define a distribution of y scores, given possible levels of x, and this conditional distribution of y|x can be of the form p(y|x ∈ A j ) (Hays, 1980).

Applying Bayes' theorem we arrive at a new distribution (Hays, 1980):

p(x|y ∈ A j ) = (2)
This new distribution reflects an updated (based on information from the questionnaire) belief about student B's academic ability, and represents a posterior distribution. If even more evidence is obtained -for example, information from the attendance at onSIP sessions -then this can also be added in a similar manner. Applying Bayes' theorem again will result in another distribution, reflecting a further updated belief about student B's possible academic performance. In this way onSIP continuously reflects the possible academic performance of all those students who underwent the on-line induction programme.

Feedback from onSIP is based on the three regions of academic performance identified by Robinson and Udall (2003), namely:

Region 1 students are participating fully in the teaching and learning process; they should progress into the following year in good standing if they keep this level of work up in their academic studies. Region 2 students are participating in the teaching and learning process, but may need additional support to progress into the following year. Region 3 students are not participating fully in the teaching and learning process, and need support in understanding the requirements of academic study in their chosen course.

This feedback (generated from a Bayesian approach) may be viewed as a conditional statement, which allows students and staff to make judgements about whether additional support is needed. An example of a conditional statement resulting from the real-time analysis is presented in Table 1.

Predicting student achievement: the results

Results from the initial implementation indicate a positive reaction by the students to onSIP, with greater student involvement and collaborative learning. In addition, the predictive feedback from onSIP has a good correlation with the end-of-year outcomes for the cohort of computer science and business information systems students who took part in this study.

In the context of this study, student achievement is defined as progressing into the following year. In the majority of cases, students normally pass p y x p x p y x A p x A j j j

( | ) ( ) ( | ) ( ) ' ' ∈ ∈ = ∑ 1 A C T I V E L E A R N I N G I N H I G H E R E D U C A T I O N 6(3)
and proceed into the following year without a deficit (Region 1 students). However, some students who proceed, do so carrying a deficit (Region 2 students). This deficit (according to the Institute's assessment regulations), must be cleared at the earliest opportunity, which usually means attempting the referral during the September re-sit period. Some students will clear their deficit and proceed as Region 1 students, while others will clear sufficient units to proceed with the remaining deficit as Region 2 students. Region 3 students are those who are unable to proceed to the next year.

During this longitudinal study the system identified at week 6 of the academic session: 83% of Region 1 (PASS -Proceed) and Region 2 (REFER -Proceed with Deficit) students, i.e. those students who progressed with no or some deficit into the following year. 69% of Region 3 (REFER -Cannot Proceed until Deficit reduced) students, i.e. those students who were unable to progress into the following year.

It should be noted that this identification is based on the voluntary onSIP sessions and not the academic units of study.

Discussion

Feedback from the students indicated a positive attitude to onSIP, expressing a view that the sessions provided an opportunity for students to work in a collaborative and informal manner. Students also felt more integrated into the teaching and learning environment of Southampton Institute, with the majority of students being classified as active participants. The focused activities allowed for a rapid pace -students undertaking individual on-line activities, with the immediate feedback provided by onSIP being used in subsequent group activities and discussions. Overall, this immediate feedback led to higher levels of student participation. The students appreciated the use of a structured framework for the session activities, supported by onSIP.

As onSIP activities were undertaken on-line, students did not need to make any hardcopies. However, students were expected to maintain a logbook (portfolio) to record their reflections on the activities and the discussion thereafter. However, many of the logbooks only contained a minimum of reflective writing. The use of logbooks was not well supported, and the reason for this needs further investigation.

Napier questionnaire

The Napier questionnaire is a diagnostic test, developed by Johnston (2001). This questionnaire is based on a paper survey of first year students at Napier University. It was distributed from November 1996 until January 1997, and completed by students during class time. An additional telephone survey of 43 non-responders to the paper survey was undertaken. This was used to identify the characteristics of these non-responders, and to ascertain whether they were critically different from the responders. A postal survey of 77 students who withdrew during the 96/97 academic year was also undertaken.

In order to provide a 'richer' student-specific data set, each survey response was linked to the appropriate student record. A 'hot-decking' technique was used to 'generate' any missing data in the survey returns. This technique uses a range of known characteristics to group similar records; from which a likely value (given those characteristics) for the missing data can be identified.

The derived questionnaire was based on 756 survey returns from a possible total of 1456 students -a response of 52%. An initial analysis indicated that those students who ultimately progressed were also more likely to undertake the student survey. To compensate for these differences in the response rates, a logistic regression procedure re-weighted the data to resemble the original population. This logistic regression identified 20 student characteristics that were found to be jointly influential over the likelihood of academic success in the first year. The results of this research indicate that early identification of students at risk is possible, and that the diagnostic questionnaire could aid in this identification.

Attendance monitoring

Evidence (Devadoss and Foltz, 1996;Durden and Ellis, 1995;Park and Kerr, 1990;Romer, 1993;Schmidt, 1983) would appear to suggest a link between attendance and academic performance, while Eaton and Bean (1995) point out that those students who are 'at risk' may be those very same students who are the 'least visible'. Indeed, the question arises: do students leave 'quietly', testing the institution through non-attendance to see if the institution notices or cares?

The authors accept that studies by Devadoss and Foltz (1996), Durden and Ellis (1995), Park and Kerr (1990), Romer (1993) and Schmidt (1983), do not imply a causal relationship. Moreover, the majority of such studies are from a US perspective, and the cultural and educational differences may negate the findings. However, the evidence appears to indicate that some form of correlation exists, supporting the hypothesis that students benefit from attending teaching sessions, and that more attendance is better than less attendance. From a UK perspective, this view is shared by Gatherer and Manning (1998), their studies suggest the existence of a weak but positive correlation between lecture attendance and exam performance, while for ethnic minority students there is a high correlation between lecture attendance and exam performance. In light of such studies, the authors believe that the influence attendance plays in academic performance should be included within the Bayesian model.

A Bayesian approach

Bayesian methods allow for the disciplined assimilation of information from differing sources and may be viewed as an extension of classical probability (Hays, 1980). However, the idea of prior and posterior distributions may present difficulties for some conventional statisticians, and probably some educationalists. This is especially so, where these prior and posterior distributions represent individual beliefs. These beliefs will be personal and subjective, and will influence the determination of the posterior distribution. In this study, the initial prior distribution is derived from the academic performance of past Napier University students. The problem with this approach is the implication of similarity; is the past academic performance of Napier students the same as the current academic performance of Southampton students? However, since onSIP is an attempt at continuous monitoring (from which the identification of students at risk will be undertaken), the data mass will continue to grow, and the original distribution will become irrelevant; this initial prior will be swamped by the accumulated data (Hays, 1980). It should be noted that the biggest advantage of using a Bayesian methodology is that the inferences try to include all the available information (MacKay, 2003). With this mind it is the intention of the authors to extend onSIP by capturing additional characteristics of the teaching and learning environment.

One aspect that is missing from onSIP is the student's perceptions of the teaching and learning process. Prosser and Trigwell (1999) have argued that the quality of a student's learning is related to the student's perception of the teaching and learning environment. The authors believe that a student's perceptions are also central when trying to understand, and hence explain a student's decision to withdraw. Laing and Robinson (2002;2003b) have previously argued that, in attending to the explanations of withdrawal, consideration must be given to discovering the manner in which a student's perceptions of the teaching and learning environment may influence their decision to withdraw. Laing and Robinson (2003b) provide an explanation of the teaching and learning environment as it relates to the student's beliefs, student/staff actions and the intentions of the institution. There are indications that the use of this additional information may improve the predictive properties of the model, and future work will attempt to incorporate this 'grounded' study into onSIP.

It should be noted that onSIP, while having a high success rate at predicting Region 1 (PASS -Proceed) and Region 2 (REFER -Proceed with Deficit) students, has a slightly lower success rate at predicting Region 3 (REFER -Cannot Proceed until Deficit reduced) students. Region 3 students are by definition not fully participating, resulting in fewer data on which to base the prediction. However, does such a tool have to be completely accurate? How good a predictor does a tool need to be before it is fair to use it on students? If the tool is so good at identifying Region 1 and Region 2 students, is it appropriate to assume that the remainder are Region 3 students, and plan accordingly?

Conclusions

The aim of the onSIP is to provide a real-time analysis and indication of those students who may be 'at risk'. The use of onSIP enables a measure of risk to be generated for each student automatically. However, making the assessment is merely the first part of a process of negotiation -a process that will help in identifying the tensions that sometimes exist between the expectations the student has of the institution, and the expectations the institution has of the student. Consequently, this negotiation becomes part of the educational team building, in which both parties (student and institution) share common goals. The automatic nature of this assessment reduces the staff workload, and provides timely analysis and information, thereby allowing staff to formulate an appropriate intervention strategy before the student's performance is irretrievably affected.

These findings appear to suggest that the Bayesian feedback from the online Spiral Induction Programme enables an improved prediction of the end-of-year outcomes for a group of technology students at Southampton Institute. This prediction was made at only week 6 in the academic year, and in a series of voluntary activities. However, there are some caveats, which require further consideration. Firstly, just how 'universal' is this model? This approach may be applicable to the Technology Faculty at Southampton Institute, but what of other institutions? This approach needs to be further validated in other higher education establishments. Secondly, the number of students in this initial study was low (when compared to the overall first year student cohort). Therefore the level of confidence that can be assigned to the dependability of the predictions made by onSIP requires careful consideration.

With an ever-increasingly diverse student population entering higher education, the 'one size fits all' model of induction and subsequent support is no longer satisfactory. This initial study has shown that automated techniques can provide an efficient predictive system to identify 'at risk' students. This early warning allows for more targeted and personalized support in a proactive rather than a reactive manner.

L

  A I N G E T A L . : T R A N S I T I O N I N T O H I G H E R E D U C A T I O N 249

Table 1 Conditional statement of academic performance

 1 Your attendance is less than 41%, which is low, and could hinder your studies.Region 2, you are participating in the teaching and learning process, but may need additional support to progress into the following year.What can I do? With your Support Tutor, have another look at the questionnaire to identify where you scored low and in which areas the Institute does not match your expectations. Some of these things can be changed and some can't. Are there any you could reasonably change? What would you like the Institute to change? Your Support Tutor will help you put together an Action Plan and where necessary point you to other sources of help. Being proactive in seeking out appropriate help and support is an important part of a successful first year.

	Tutor A. Non

Biographical notes

C H R I S TO P H E R L A I N G is a Senior Lecturer at Northumbria University. He is currently developing the supporting website for the School of Informatics 1st year Study Skills Programme. He is also Module Tutor for the Object Oriented Design 2nd year undergraduate module. Address: Quality Enhancement Services, Bevan Villa, Napier University, Craighouse Road, Edinburgh EH10 5LG, UK. [email: v.johnston@napier.ac.uk]


Retention and Student Success in Higher Education

The chapters in Part Three (Enhancing Student Progression) address reflective practices and how they can influence a student's progression in HE. Jenny Moon explores learning as represented in level descriptors, Jenny Phillips describes the development of the Keynote Guide to PDP, and Kathryn McFarlane's chapter on Learning about Employability completes the set of three topics that are a priority for many tutors in HE. All the authors offer excellent advice to readers wishing to incorporate the strategies discussed in the chapters into their courses or research projects.

The chapters in Part Four (Supporting and Developing Staff) shift the focus from students to the support and development of staff. John Shaw claims that it is important for those who support learning to understand how they learn in order to improve their teaching. He challenges the reader to instigate an action research project addressing an educational hypothesis, engaging in reflection and acting on the findings. Vivien Sieber's chapter on Learning to Teach with Technology is excellent, highlighting the important skills required and some strategies that promote the development of these skills. Each section ends with key questions and challenges for the tutor, to aid the development of their teaching with technology. The final two chapters address the impact of new legislation and policy changes on inclusive learning and equality of opportunity. Both authors make it very clear that the new legislation involves everyone in HE and offer excellent guidance for implementation.

This book does what it sets out to do in that it considers at the operational level a range of new approaches for improving students' learning and will be useful and challenging for a wide range of tutors in HE. In particular, the examples of pedagogic research will be extremely helpful to tutors who are participating in Higher Education Academy Accredited Postgraduate Certificate courses and will complement the core texts that already exist. This book is concerned with many aspects of retention and student success in higher education systems, covering: retention and student success from a public policy perspective; the theoretical approaches and enhancement measures for retention.

All ten chapters of the book have been written in a professional manner, with its presentation, analysis and discussion based upon a wide range of sources of information. A welcome feature is the international perspective, particularly from the UK, Australia, South Africa and the United States.

The book comprises two main parts. The first five chapters are primarily on the subject of higher education systems and their performance. The latter five chapters focus on how institutions can help to increase the chances of student success. The main benefit of this book is in the information gathered and reviewed, which would be useful to both researchers into higher education and educators who want to enhance retention and student success practices in higher education. I particularly enjoyed reading all the chapters in the book, but those who are seeking ideas and innovative approaches that they could implement in their institutions would probably want to focus on the latter three chapters.

Chapters 6 and 7 discuss and draw together the theory relating to retention and student success. These two chapters, together with the high level of relevant citations, should be a useful resource for researchers in the field of 'student departure and retention in higher education', as they allow for reflection on how to refine and extend the theoretical approaches. The authors advanced the argument that, 'whilst much of the theorizing about retention reflects sociological concerns, students' decision-making whether to persist or withdraw is fundamentally a matter of psychology'. It is arguable that students' perception of their experience in higher education is influenced by many factors, including sociological and psychological factors. The authors suggest, quite rightly in my experience, that some of the factors are beyond the powers of an institution of learning to influence -hence the suggestion by the authors that existing theoretical approaches relating to retention and student success need refinement and extension. The authors acknowledge that while the suggestions made in the final chapter have a broad applicability within higher education, some are likely to need adjustment to suit individual institutions and particular national systems.

The book contains a large number of excellent illustrations, which have been used to great effect in assisting the understanding of the texts in the book. The citations, list of references, endnotes, author index and subject index are sufficiently comprehensive to help the reader. One list of references is provided at the end of the book, and I would have preferred to see a list of references at the end of each chapter.

The book is well structured and the fonts used in the book should find favour with the reader. The use of case studies and research evidence is particularly welcomed.

In summary, this is a high-quality publication. It is an enjoyable read and a valuable information source. The book should serve as a very good resource for practitioners in higher education institutions, including lecturers, support staff and senior managers. The book is also likely to provide an interesting read for politicians and civil servants with wider policy interest in higher education matters. The publication achieves its objective and has many references for reader to obtain more detail if required.

  D E R RY C O R E Y College of St Mark & St John, Plymouth Retention and Student Success in Higher Education by Mantz Yorke and Bernard Longden. London: The Society for Research into Higher Education & Open University Press, 2004. ISBN 0-3352-1274-3. 171 pp. £23.99 (pbk).

K

  E H I N D E O D U Y E M I Senior Lecturer, University of Abertay, Dundee


A tool to explore the initial expectations of PhD students and supervisors

Keywords: expectations, postgraduate supervision, service quality, SERVQUAL

A B S T R AC T In recent years there has been growing concern to improve standards in postgraduate research. Much of this has focused on generic research skills training. However, there are other, equally important, supervisory aspects such as welfare, mentoring and support arrangements. This article focuses on the initial expectations, of both students and supervisors, of their roles and duties at the start of the supervisory process. Using concepts from the service quality literature, an instrument 'Superqual' is developed aimed at exposing any gaps in expectations that, if not resolved, could generate problems during the life of a doctorate. The use of Superqual in a number of case studies is reported. The results are mostly positive, and suggestions are made for its improvement and adoption.

Introduction

In recent years, there has been growing pressure to improve standards in postgraduate research programmes. The recent joint consultation process, initiated by the four UK higher education funding bodies (JFC, 2003), invites higher education institutes to develop threshold standards across a range of practices.

Much of the focus has been on generic skills training, and the 'NewRoutePhD' along the North American 3+1 framework is one product of this thinking. However, there is another aspect that is equally important -the supervisory relationship itself, 'a good relationship between supervisor and student is one of the most powerful and lasting relationships you can have. Get it right and the bond will be sealed for life ' (Bassnett, 2003: 23). The joint funding bodies' consultative paper suggests that 'there should be regular structured interactions between student and supervisor' (JFC, 2003: 9) but gives little guidance on the nature of the interaction. The literature on studying for a PhD, however, stresses the desirability of a negotiated model of student supervision where expectations are made explicit as early as possible. For example Wisker talks about setting up a learning contract that clarifies the 'ground rules of expectations and behaviours' (Wisker, 2001: 54). Yet Phillips and Pugh (2000) acknowledge a common problem, that meetings between supervisor and student tend to be dominated by technical and logistic elements whereas issues concerning expectations and behaviours can become taboo topics. So it is perhaps unreasonable to expect the student to initiate discussion on these taboo subjects. Supervisors too may have problems broaching these subjects. They may have poor interpersonal skills, or poor listening skills, or they simply may not realize that there are any differences between their expectations and those of the student.

Although it is commonly agreed that 'a good supervisor-student relationship can only thrive if both parties share mutual expectations and have established ground rules about the regularity, type and focus of supervisions' (Wisker, 2003: 24), it is not so easy to create these conditions. Hence a tool in the form of an inventory of expectations may prove useful in initiating discussion especially if it is seen as being independent of both the supervisor and student. In this article the development and testing of a tool named Superqual is reported. The goals are to make explicit the initial expectations of PhD student and supervisor, to promote discussion and to expose problems at an early stage. It is hoped that this simple tool will help ease and structure this process.

Development of Superqual

The instrument used in these case studies is based on the SERVQUAL instrument, developed by Zeithaml et al. (1990) to measure customer service quality.

SERVQUAL is the best known example of the disconfirmation approach, wherein service quality is measured by how well the service delivered 'disconfirms' the customer's expectations. Hence Parasuraman et al. (1988) define service quality in terms of the gap between customers' expectations of service and their perceptions of the service experience. There are a number of critiques of the instrument and of the underlying model (see, for example, Cronin and Taylor, 1992); however, it has been widely used in measuring service quality in a vast array of contexts including higher education.

The SERVQUAL instrument measures five underlying dimensions of customer service expectations and perceptions: Tangibles:

Facilities, equipment, materials and employee appearance.

Reliability:

On time, first time.

Responsiveness: Prompt, willing service. Assurance: Knowledge, courtesy, trust and confidence. Empathy:

Tailored, caring personal attention.

Are these dimensions of customer service appropriate in the postgraduate setting? Pole and Sprokkereef (1997) used in-depth interviews to examine PhD students' expectations of supervisors and found that among the desirable qualities for a supervisor were:

Organization and structure (Reliability). Availability, uninterrupted meetings (Responsiveness).

Expertise, knowledge, reassurance (Assurance).

Interest, passion (Empathy).

With the possible exception of 'tangibles', the dimensions covered by SERVQUAL appear to be relevant to PhD students. Indeed, SERVQUAL has already been used in higher education with some success. Cuthbert (1996aCuthbert ( , 1996b) ) used it to measure undergraduate perceptions in Manchester. The instrument was quite heavily modified, especially by use of negative statements. Cuthbert found that his version had poor reliability and, more worryingly, different dimensions emerged other than the five Zeithaml et al. had suggested. Brown (1998) used it to measure postgraduate perceptions on an MBA course in England. He used only slight modifications of SERVQUAL but did not test the reliability of the instrument. Slade et al. (2000) used it on Australian undergraduates looking, in particular, at differences in expectations between students who left and those who completed their studies. They made only slight changes and reported good reliability. Darlaston-Jones et al. (2003) also used it on Australian undergraduates, concentrating on differing student perceptions of administrative and academic staff. They used slight modifications but reported no test of reliability. Finally, Sherry et al. (2003) used the instrument to look at differences between the expectations and perceptions of home and foreign students among New Zealand undergraduates. Their study, based on work carried out by East (2001), reported good reliability using only a slight modification of the classic SERVQUAL instrument.

The work carried out so far using SERVQUAL in a higher education context would seem to suggest that the instrument can be used successfully, as long as the modifications are kept to a minimum. However, the author could find no work specifically using SERVQUAL on PhD students or on supervisors and there may be good reasons for this.

PhD supervision differs from both undergraduate and even taught postgraduate courses in terms of the complex and subtle relationship between supervisor and student. This makes the simple view of PhD students as customers problematic. In the standard company-customer relationship the service provider provides a service to the customer. This may include an element of training but service providers do not generally train the customer to replace them! Yet this is precisely what the supervisor is aiming to do. Gurr (2001) argues that the goal of supervision is 'competent autonomy' so that the student is able to become an autonomous but competent researcher. Phillips and Pugh (2000) argue that supervisors are training postgraduate students to become their own supervisor while Pole and Sprokkereef (1997) say that the supervisor acts as a kind of gatekeeper for entry to the academic world.

In order to develop the instrument, Superqual, to explore initial supervisor-student expectations SERVQUAL has been amended in two ways. Firstly there have been some minor changes to the actual statements used. Secondly there have been some major changes in the way the instrument is implemented.

Superqual comprises 22 statements covering all five of the service quality dimensions (see Appendix 1). Institutional functions (tangibles) are covered as well as supervisory functions (reliability, responsiveness, assurance and empathy). Tangibles are included to help establish some commonality and to make the use of the tool less confrontational. The response format was a seven-point Likert scale, couched in terms of importance rather than agreement. A caveat to be realistic and to prioritize was added, as it would be easy to give maximum importance to the majority of the statements.

The evidence of the research cited earlier is that changes to the statements should be kept to a minimum and so the wording was kept as close as possible to that used in previous research using SERVQUAL in a higher education setting. In particular all the statements were positively phrased, as the evidence was that these were more reliable. Nevertheless it is important that the instrument covers all the important issues and in, particular, acknowledges the specificity of the PhD experience. This was reflected in a number of non-standard statements, for example statements about constructive criticism (Q21), competent autonomy (Q19) and the concept of a research culture (Q18).

Although there were few changes to the actual statements, there were two major changes to the way Superqual was implemented. Firstly the classic use of SERVQUAL asks customers to respond to the statements twice, once about 'excellent companies' (their expectations) and once about the actual company (perceptions). Analysis centres on the gaps between customers' expectations and perceptions. This is inappropriate to the early stages of the supervisor-student relationship as perceptions have not yet formed. However, Zeithaml et al. (1990) note that the gap between customer expectation and perception may be due to a number of smaller gaps and they suggest ways that SERVQUAL can be adapted to measure these. One of these sub-gaps is between customer expectation and management perceptions of customer expectation and SERVQUAL can be used to ascertain the expectations of both. Superqual is used in this fashion, to compare the expectations of supervisor and student.

Secondly instead of using the SERVQUAL statements in a standardized structured questionnaire, Superqual uses them as a template for a semistructured qualitative discussion in a one-to-one setting. The introductory rubric emphasizes that both supervisor and student should complete the questions in private making explicit their own expectations. The concluding rubric explains that they should then come together to discuss their expectations, uncover any gaps and hopefully resolve their differences (see Appendix 1).

Used in this way, as a basis for discussion and negotiated agreement, the statements themselves are really of secondary importance, what matters is the process of asking, leading to reflection, discussion, negotiation and resolution.

Methodology

The purpose of the case studies reported here was to evaluate Superqual. There are a number of possible approaches to evaluation, formative evaluation, summative evaluation, illuminative evaluation and integrative evaluation (Jackson, 2003). The approach taken here is largely illuminative evaluation, trying to discover those aspects of Superqual that work best by identifying the issues that are most pertinent to the participants. However, the four categories of evaluation often overlap and there are also elements of formative and integrative evaluation with the aim of modifying either the content or the implementation of Superqual. The method adopted was semi-structured interviews of a small number of student-supervisor pairings which is particularly suitable for this type of evaluation.

Twelve students were selected from a cohort of 24 who were all in the first few months of their PhDs. While the sample was small and clearly limited, a great deal of care was taken to ensure that key variables such as gender, ethnicity, subject area and supervisor experience were adequately covered. The supervisor and student were both asked to complete Superqual and then meet to discuss the results. Seven out of the 12 did so giving a response rate of 58 per cent. All seven of the supervisors were British; two were female and five male. There was a range of experience with two of the supervisors being very new and three very experienced. Two of the seven students were UK nationals, four were European and one Asian. There were four males and three females. They ranged from mature students returning to full-time education after a period of employment to those coming directly from undergraduate courses. Three of the students were registered in the Business School, one in Education, one in Social Sciences and two in Science.

Respondents were contacted in mid November and the meetings between supervisor and student were supposed to take place before Christmas. In fact there was a much wider range of timing with the first pair meeting in November, three in December, one in January and two in March. The discussion that follows is based on semi-structured interviews with the supervisors. The views of the students will be the subject of another paper that compares the experiences later in the postdoctoral process between those who have and those who have not used Superqual.

Results

There was a good deal of variation in supervisors' overall perception of Superqual. Two were very positive, two were reasonably positive, two were neutral and one was negative. A typical neutral comment was: I didn't really learn anything about the student that I didn't already know. (S6) However, even here there was some positive feedback: Overall perceptions were related to the length of the meeting with students, which varied from 20 minutes up to an hour; to when the meeting took place; and to level of experience of the supervisor. It was noticeable that the later the meeting, the shorter and less useful it proved to be; for example the feedback from those who met in March was at best neutral. It was also noticeable that the longer and most productive meetings came with the least experienced supervisors.

Analysis of the completed forms from both supervisor and student showed that there was little overall disagreement, the overall mean gap was just over 1. However, even when there was agreement this could be an important factor. For example many supervisors reported that question 19: 'Supervisors . . . will ensure research students eventually become competent autonomous researchers' was a key question. The fact that there was little disagreement on this was openly welcomed:

I was very pleased that we were at one on this . . . I hadn't had time to discuss this with the student and I was pleased that the question had been aired and the result was harmonious. (S2)

The answer was not a bolt from the blue but I was very pleased about it. This was one question that I wasn't so sure about. (S7)

The student was more independent than I thought she might be . . . and I was pleased about that. (S6) Supervisors also welcomed the opportunity to discuss the full range of topics even if there was little disagreement:

It put all the issues clearly on the table . . . in future we will both be happy to raise those issues again. (S2)

Although there was little disagreement in terms of mean scores -only five of the 22 questions (Nos 7, 10, 11, 20 and 21) elicited no disagreement from anybody -most questions led to some disagreement between at least one pair of respondents (disagreement is defined as a gap of at least two in the scores). There were also a number of questions that fairly consistently caused some discussion (in at least three separate cases). These will be considered under the five dimensions.

Tangibles

There was one question that caused discussion, Q4: 'Excellent universities should help provide a good living environment for research students'. Foreign students tended to rate this higher than their indigenous supervisors, interpreting the question more widely, meaning help in coming to terms with a new environment. With indigenous students the opposite often occurred, with supervisors scoring more highly. Comments included:

He mentioned things like opening a bank account while I just took it to mean finding a place to live. (S7) She's a local girl, she didn't need any help but I was answering this more generally, about all students. (S4)

Reliability

There was disagreement on two questions, Q8 'When a research student has a problem, supervisors at an excellent university will show a sincere interest in helping to solve it' and Q9 'supervisors at an excellent university will ensure that research students plan an adequate programme of work and keep records'. In fact the disagreement in both questions is related to the level of independence that a student should exhibit. Sometimes there was disagreement because the supervisor felt that the student was in danger of becoming too autonomous too early. In these cases the supervisors felt that this should be flagged as a potential concern: He (the student) felt that when any problem arose within his research he should first try and solve it himself and only seek help when he failed but I was worried that he would waste a lot of effort going down blind avenues. (S3) There's a real danger that she will go off on a tangent if I don't keep some check over what she does. (S4) However, in other cases the problem seemed to be that the supervisor was anxious to encourage independence and not to solve all problems too quickly: I didn't want her to feel that I was pushing her in a particular direction but she felt that I wasn't taking an interest. (S5)

The disagreement over the keeping of records was also related to levels of autonomy. Students often felt that record keeping should be their duty while supervisors felt it was their responsibility to ensure that students had some structure to their work. One supervisor, from the science faculty, explained that keeping accurate records was critical as often experiments may not be reproducible and this should not be left entirely to the student.

Responsiveness

There were two questions that generated disagreement, Q12 'Supervisors at an excellent university will be willing to help research students' and Q14 'Supervisors at an excellent university will never be too busy to respond to research students' requests'. Both these questions are very similar and in all cases students rated them higher than supervisors. In the former there was often a difference in interpretation; supervisors would emphasize that they may not always be able to 'help' while students would emphasize the word 'willing'. In the latter, supervisors argued that they could not realistically be expected to answer questions immediately whereas students tended to view 'respond' as meaning at least an initial acknowledgement. They recognized that supervisors could not always respond quickly; however, receiving no response or a vague response was frustrating, and an acknowledgement of the problem should be received quickly, together with an indication of the likely timeframe for addressing the problem.

Assurance

Q17 'An excellent university will offer research students a range of support and training services' generated some disagreement. Students tended to rate this more highly than supervisors. There was also some disagreement over Q16 'Supervisors . . . will always be courteous and polite' although, curiously, supervisors always rated this more highly than the students. This may be related to the general consensus around Q21 'Supervisors . . . sometimes need to be constructively critical in the best interests of the student'. Empathy Q22 'Supervisors . . . should . . understand the needs of both international and domestic students' also generated some disagreement. Foreign students in particular valued empathic understanding of their wider needs, especially with respect to their integration into Scottish society.

A number of themes seem to emerge from this analysis. Firstly there were few serious disagreements, a great deal of the disagreement occurred because of differences in interpretation of the questions, in the meanings ascribed to them, and could be resolved fairly easily. Secondly the issue of the relative autonomy of the student came up in discussions over a number of questions, although, interestingly, not in response to the direct question on the goal of 'competent autonomy' (Q19). It would seem that the pertinent issues were to do with timing rather than the eventual goal. Finally issues surrounding the wider needs of foreign students came up a number of times.

Conclusion

The case studies reported here have looked at Superqual very much from the supervisors' perspective. A complete evaluation of Superqual also needs to consider the longer-term student perspective and that work is on-going. However, supervisors' consent to the process is clearly crucial if the use of Superqual is to be accepted and these case studies can help shed light on this. In this regard a number of positive factors have emerged about the use of Superqual.

Firstly, Superqual was generally felt to be useful. Most supervisors felt that the meetings with students were productive. Some practical issues were sorted out and a number of concerns were flagged. As well as these practical benefits, the relationship between supervisor and student was subtly strengthened. Even when there was little actual disagreement many still felt the exercise had been useful if only in a confirmatory sense.

It may be argued that Superqual is unnecessary. The same ends could be met by a friendly chat over drinks in the local pub. This is true and the author would not want to discourage this. However, there may be factors such as personality, gender or ethnicity that mitigate against such a cosy relationship. It was notable that some the most positive assessments of Superqual came from supervisors with foreign students. However, it also found favour among the two supervisors who had known their students for some time prior to the start of supervision because it enabled the issues to be presented in a neutral fashion as emanating from a third party: I've known [the student] for years but we've never really talked about some of this stuff. (S5)

It was strange to meet and not talk about work . . . the discussion was useful because it explored a range of issues in a non-threatening way. (S4) Secondly, despite the small size of sample, there is no evidence to suggest that the dimensions and statements used in Superqual are inappropriate. A number of statements across all five dimensions generated discussion; no one dimension seemed more important than any other. In fact, despite the small sample, all but five of the 22 statements generated some discussion among some respondents indicating that there is little obvious scope for cutting out statements. Also the new statements covering autonomy, constructive criticism and research culture were all well received and initiated important dialogue. Indeed autonomy seemed to be a key issue that came up in a number of different questions. Finally there was no evidence that any important elements had been left out. Supervisors were expressly asked if they felt there were any aspects that had been omitted but none were volunteered.

Thirdly, there was general support for the way Superqual was implemented. It may be argued that the approach is overly technicist. As one supervisor stressed,

The student-supervisor relationship is not generic but individual, some students go their own way, others require constant help. (S1) However, the way that Superqual was used seemed to allow for individual tailoring. As emphasized in the concluding rubric (Appendix 1), the statements are to be taken as a starting point to a more general discussion. Although a number of supervisors claimed that, when talking with the student, many of the disagreements turned out to be 'only a matter of interpretation, of meaning' this is somewhat ingenuous. Conversations, as writers such as Grice (1975) and Bateson (1984) remind us, are essentially a transfer of meaning through the medium of words. Hence sorting out differences in meaning, of subtleties in interpretation, in face to face discussion, is actually quite a crucial aspect of communicating.

There were also a number of more negative factors that emerged from the case studies. Firstly it was clear that Superqual was found to be more beneficial the earlier it was used. The last two case studies, which took place when the students had been in place over six months, did not find Superqual to be as useful as those that took place in the first three to four months.

Secondly Superqual was taken up more enthusiastically by newer supervisors and it is possible that the tool will not be used by more experienced supervisors. Certainly the response rate of 58 per cent indicates that this may be an important factor. At the institution where the case studies were carried out it is clear that supervisors, especially experienced supervisors, are very busy people. At least two of the non-responders were promoted to senior management positions during the time of the study. This meant firstly that the project was quite low down on their list of priorities and secondly that it was difficult for the author to successfully badger them to complete the study.

What lessons have been learned from this evaluation? Firstly, it seems that it is more useful the earlier in the candidature it is used. It could, for example, be built into the general student induction process. Secondly, it is also likely to be of more immediate benefit to relatively new supervisors rather than established supervisors. Indeed it may be usefully employed in the training programmes for doctoral supervisors, which are now being developed within the British academic system. Thirdly, if it were to be administered universally, there are issues of non-compliance and of late compliance that would need to be addressed.

Finally there is the potential of widening the scope of its use. In this article only a single relationship between supervisor and student has been considered but increasingly PhD supervision is being undertaken by teams of supervisors. The use of supervisory teams has clear advantages, but is not without its problems, especially in terms of the group dynamics between colleagues of differing seniority. The tool, or something similar, may be useful within supervisory teams to make explicit different approaches from different supervisors.

As discussed earlier, a complete evaluation of Superqual requires us to consider if its use is of benefit to students. In particular are those who have experienced Superqual better equipped to deal with the doctoral process as it unfolds than those who have not? This question can only be evaluated later after a suitable passage of time and that work is on-going. However, in the meantime, the evidence in this study of the supervisors' perspective indicates that Superqual has some merit and has been found to be useful in a number of cases. It is also not very onerous. As a practising teaching academic and supervisor, the author is extremely reluctant to suggest to colleagues that they fill in yet another form. However, in the current climate within Higher Education, with the emphasis on enhancement and on audit trails, it is only a matter of time before something similar to Superqual is foisted onto supervisors. If such an instrument is to be used surely it is better to have one that has been developed with some thought and properly tested rather than one that has been devised by administrators. Q4. Excellent universities should help to provide a good living environment for research students. Q5. When supervisors at an excellent university promise to do something by a certain time, they should do so. Q6. When research students at an excellent university promise to do something by a certain time, they should do so. Q7. Supervisors at an excellent university will assess research students' work carefully and accurately. Q8. When a research student has a problem, supervisors at an excellent university will show a sincere interest in helping to solve it. Q9. Supervisors at an excellent university will ensure that research students plan an adequate programme of work and will keep accurate records. Q10. Supervisors at an excellent university will be concerned about the educational progress of their research students. Q11. Supervisors at an excellent university will respond promptly to research students' needs. Q12. Supervisors at an excellent university will always be willing to help research students. Q13. Supervisors at an excellent university will have extensive knowledge in their field of training. Q14. Supervisors at an excellent university will never be too busy to respond to research students' requests. Q15. Supervisors at an excellent university should keep convenient office hours for all research students. Q16. Supervisors at an excellent university will always be courteous and polite. Q17. An excellent university will offer research students a range of support and training services. Q18. Excellent universities should ensure new research students contribute to and benefit from the research culture of the university. Q19. Supervisors at an excellent university will ensure research students eventually become competent autonomous researchers. Q20. When research students have problems, supervisors at an excellent university should be sympathetic and reassuring. Q21. Supervisors at an excellent university sometimes need to be constructively critical in the best interests of the research student. Q22. Supervisors at an excellent university should understand the needs of both international and domestic students.

Concluding rubric:

Thank you for completing this questionnaire.

Remember that the aim of Superqual is to help diagnose potential problems and to allow the relationship between supervisor and student to develop.

Both the supervisor and the student should now bring their answers to a meeting arranged solely to compare the responses to the statements. In particular you should address the following:

1. In what specific areas is there a lack of agreement between student and supervisor? 2. Is there any more general lack of agreement between the overall views of the student and the supervisor?

The meeting should provide an opportunity to discuss these differences. It is suggested that a record of the discussion is made and any negotiated recommendations be recorded.

Note: the following did not form part of the Superqual instrument but is included for information:

Questions 1 to 4 cover the Tangibles dimension, questions 5 to 9 cover Reliability, questions 10-12 and 14 cover Responsiveness, questions 13, 16, 17 and 19 cover Assurance, and questions 15, 18 and 20-22 cover Empathy. UK. [email: mario.hair@paisley.ac.uk]

Biographical note

  I didn't really learn anything new but it did firm up a number of ideas (about the student) I already had in my head. (S4)On the other hand there was more positive feedback, for example: He is quite a serious student and it is difficult to know what he really feels . . . I learnt a lot about how he thinks of research. (S2) I really felt that I began to understand his problems, coming here to study in a strange country. (S3) Only one of the supervisors was antagonistic but admitted to an aversion to training for supervisors which predates receipt of this form. (S1)

  She [the student] seemed to welcome bluntness and honesty. (S5)

  M A R I O H A I R has been a statistics lecturer at the University of Paisley for the last 17 years. He is also an active researcher within the Statistics Consultancy Unit based at the university. His research interests include postgraduate training and survey methodology. Address: Department of Physical Sciences, University of Paisley, High St, Paisley PA1 2BE,

Acknowledgements

I would like gratefully to acknowledge the support of the Higher Education Academy in helping to fund this research. I would also like to thank all the supervisors and students who gave up their time to help me with this work.

Appendix 1: Superqual

Introductory rubric: Thank you for agreeing to take part in this evaluation of Superqual which is being funded by the Academy. The aim of Superqual is to help diagnose potential problems that may otherwise emerge over the course of candidature and to allow the relationship between supervisor and student to develop.

Both the Student and the Supervisor are asked to complete separate copies of this questionnaire.

Please think about the kind of university that would deliver an excellent quality of research environment. Then read the statements below that present features that an excellent university might possess.

There are no right or wrong answers, just select a number that truly reflects your feelings.

Please try to be realistic in your assessment -even excellent universities cannot provide everything so try to prioritize what really matters to you.

If you feel a feature is not very important for an excellent university please circle the number 1.

If you feel a feature is absolutely essential for an excellent university please circle the number 7.

If your feelings are less strong circle one of the numbers in the middle.