Simulations and games Use and barriers in higher education

Keywords: barriers, games, role play, simulations

This article explores the use of simulations and games in tertiary education. It examines the extent to which academics use different simulation-based teaching approaches and how they perceive the barriers to adopting such techniques. Following a review of the extant literature, a typology of simulations is constructed. A staff survey within a UK higher education (HE) institution is conducted to investigate the use of the different approaches identified within the typology. The findings show significant levels of use of both computer and noncomputer-based simulations and games. The main barrier to teaching with simulations, as perceived by the respondents, is the availability of resources. However, further analysis indicates that use of simulations is not associated with perceptions of resource issues, but rather is influenced by views on the suitability of, and risk attached to, such learning methods. The study concludes by recommending improved promotion of simulation-based teaching through enhanced information provision on the various techniques available and their application across subject areas.

Introduction

As HE provision has expanded, so too has the variety of methods used for teaching and learning. There has been a movement towards more active and experientially based learning. Amongst those techniques that have gained acceptance are those that might be embraced within the term simulation. However, to date there have been relatively few published studies exploring the range of simulation techniques that are employed for teaching within HE. The purpose of this paper is to examine the use of simulation and gaming approaches for teaching and learning in HE and to analyse the perceived barriers to the employment of such techniques. The study is based upon the findings of a 2005 survey of academic staff working across a range of subject disciplines in a UK university.

Literature

A number of studies support the utility and effectiveness of simulations and games for teaching and learning (see Feinstein, 2001;Keys and Biggs, 1990;Wolfe and Crookall, 1998). Ruben (1999), tracing the evolution of such experiential learning approaches from the late 1960s, argues that these methods address many of the limitations of more traditional teaching. Specifically, he identifies their value in addressing cognitive and affective learning issues and in facilitating interactivity, collaboration, peer learning and active learning.

Simulation-based learning approaches aim to imitate a system, entity, phenomenon, or process. They attempt to represent or predict aspects of the behaviour of the problem or issue being studied. Simulation can allow experiments to be conducted within a fictitious situation to show the real behaviours and outcomes of possible conditions. However, simulations cannot simply be regarded as a homogenous collection of approaches. Whilst overlaps between activities exist (Yorke, 1981), previous studies have identified three specific types of simulation-based learning: role play, gaming and computer simulation (Feinstein et al., 2002;Hsu, 1989). Each type is different in its composition and utility. In a role playing exercise, participants act out the role of a character in a particular situation following a set of rules and interacting with other role players. Role play might also be non-interactive, for example in the case of presentations where students adopt a particular character role. With gaming, the key elements entail interaction within a predetermined context, often involving forms of competition, cooperation, conflict or collusion. These interactions are constrained by set rules and procedures. Finally, computer simulations aim to replicate system characteristics using mathematics or simple object representations (Feinstein et al., 2002). Ellington (2001) also considers differences in format as a means of distinguishing types of simulation. He presents a hierarchical classification of simulations, within which a key distinction is made between manual exercises and electronic exercises. Examples of manual exercises identified by Ellington include card, board and field games. Maier and Größler (2000) elaborate further the particular differences in types of computer-based simulations. In their typology, they develop the principal categories of modelling oriented simulation tools (i.e. those used to model particular issues) and gaming oriented simulation tools. The latter is further divided into simulators and planning games, the former identified as single-user applications (played against a computer model) and the latter as multi-user applications (played against other users through a computer application). Hays and Singer (1989) refer to simulators specifically as training devices used to develop or maintain defined skills. Feinstein et al. (2002) note that such applications can be distinguished by the lack of any interpersonal element.

Figure 1 presents a typology of simulation types derived from the findings of previous studies. This typology forms the basis of the research presented within the current study.

Use of simulations, games and role play

Evidence on the extent to which simulation approaches are used for teaching in HE is patchy. There is an abundance of studies relating to the development, use and evaluation of specific simulation activities. Furthermore, largely descriptive studies exist which explore the use of simulation within specific subject areas. However, there is limited statistical evidence on using simulations (in all their different forms) across subject areas. A review of published research in specific subject areas or relating to specific simulation approaches nevertheless provides a useful indication of the Burgess (1991) provides the most significant evidence relating to business simulation game use in the UK. His study, which predates the reclassification of UK polytechnics into universities in 1992, finds that there is a difference in the adoption of business games between polytechnics and universities, with the former having a higher level of use. This could be attributed to the more vocational nature of educational provision in polytechnics. Since 1992, these differences may have become less apparent owing to the significant increase in business education provision in socalled 'old' (pre-1992) universities. Nevertheless, this difference is important to note given that the host for this current study is a former polytechnic. Overall, Burgess finds that the level of usage of business games in UK polytechnics (at 92% school uptake) is broadly similar to levels of adoption found by Faria (1987) in US business schools (at 95%).

There appear to be no substantive surveys on using computerized simulation games in non-business subjects. Similarly, there is limited statistical evidence relating to modelling-based and training-based simulations. However, individual descriptive or evaluative studies of specific computerbased simulation activities indicate that such techniques are employed across a range of disciplines. In relation to simulation games, these include sociology (Hofstede and Pedersen, 1999), architecture (Sipes, 2004), media studies (Rettburg, 2004) and engineering (Kumar and Labib, 2004). Modelling-based simulation is widely used to teach engineering but is also used to teach other disciplines (see, for example, Chwif et al., 2001;Jan and Jan, 2000). Training-based simulation is common in disciplines such as medicine, safety, shipping and aeronautics. Examples include Yang et al. (2001) and Rolfe and Hampson (2003).

Much of the literature on role play is rather descriptive and there is little empirical evidence on the extent of use in HE (Armstrong, 2003). Nevertheless, a review suggests that the approach is employed across disciplines.

These include geography (Oberle, 2004), foreign languages (Ladousse, 1987), health studies (Goldenberg et al., 2005), social work (Moss, 2000), religious studies (Couture, 1999) and science (Sleigh, 2004).

Evidence on the extent to which non-computerized educational games are used in HE is again limited. However, a review of studies reporting on specific instances of use confirms that games are employed across degree programmes. Most studies relate to board or card games and examples of their use span subjects including economics (Stanley, 2001), chemistry (Russell, 1999), psychology (Weisskirch, 2003), biology (Taylor and Jackson, 1996), mathematics (Baker, 1999) and creative writing (Morice, 1992).

Barriers to using simulations, games and role play

A number of studies evidence the barriers to using simulation. Most of these relate to either gaming simulations or role play within specific learning environments. These studies help to identify a number of common issues that face educators. Faria and Wellington (2004), examining the use of gaming simulations in business schools, ask academics who have never used simulation games what has prevented them from adopting the teaching method. The most important factor is preparation time (35.7% of respondents). This constraint is also highlighted as the most significant barrier in Chang's (1997) study. The next barrier elicited by the Faria and Wellington survey is a poor fit with the courses being taught (34.4%). A lack of information on simulations is also reported by 29.9 per cent of respondents. The problem of information is further highlighted in that only 26.2 per cent of 'never-users' are 'very familiar' or 'somewhat familiar' with simulation games. As the authors note,'it is difficult to be a business game user if one is not familiar with business games' (Faria and Wellington, 2004: 198). Other less important barriers reported are a preference for alternative pedagogical approaches, the amount of class time required for simulations, funding, administrative issues and technical issues. In the same study, the authors also ask former users why they stopped. The top two responses to some extent mirror those identified by non-users, with 32.8 per cent citing changed teaching responsibilities (i.e. simulation games were no longer appropriate for courses taught) and 28 per cent indicating the time involved in using games against the learning benefits achieved. Chang's study indicates that the number of academics who stop using simulation games is high, with only 10 out of 35 past users continuing with the approach.

Examining the use of simulation games in agribusiness and agricultural economics, O'Rourke (2001) elicits responses from game users on the problems associated with employing the method. The most significant problem is found to be updating and maintaining software, followed by taking time away from other activities.

Time demands also emerge as key barriers associated with role play. Francis and Byrne (1999) estimate that the preparation workload for role play exercises is 50 per cent more than is required for a conventional lecture. Similarly, Moss (2000) concludes that role play 'is one of the most "labour-intensive" ways of working as far as the teacher is concerned ' (2000: 481). Like other authors (Alden, 1999;Armstrong, 2003) he highlights the demands of the approach during the implementation stage as well as the preparation stage. Armstrong ( 2003) also discusses how role-play may be regarded as a challenging or risky teaching approach in that it is often unfamiliar to students and staff, and may therefore result in anxiety amongst students. In addition to having to manage the possible negative reactions of students, lecturers must also deal with outcomes that are often unpredictable and difficult to control (see also Jones, 1988). Armstrong finally identifies resource issues (both human and non-human) as potential challenges.

Grisoni ( 2002) presents qualitative research findings relating to experiential learning practice in HE. In particular, she highlights the uncertainty felt by many academics considering experiential learning approaches. Some lecturers appear not to be confident about how students will react to such approaches and how successful these techniques will be. The study also reveals that some academics feel experiential approaches to be 'nonacademic' and less suited to university-level education than more theoretically based approaches. The study also identifies significant cost and resource challenges. Grisoni does however highlight the possible role of technology-based approaches as mechanisms for reducing the high costs associated with staff time.

Overall, the literature presents a fairly uncertain and incomplete picture about the extent to which different simulation approaches are used in HE. Furthermore, whilst there is some understanding of the possible issues that might act as barriers to using simulation, there is little evidence relating to the relative importance of these issues across the full range of simulation methods. Needless to say, there also appears to be no empirical exploration of the possible associations between the barriers to simulation use as perceived by staff, and their actual use by academics.

Aims and methodology

The purpose of this study is to examine the use and perceived barriers to use of simulation and gaming approaches for teaching and learning in HE.

To achieve the study's purpose, a typology of simulations was first developed from the preceding literature review (see Figure 1). The study aims to address the following research questions:

What proportion of staff in the host institution are 'current', 'former' and 'never' users of each of the forms of simulation identified in the typology? How important do staff perceive the different barriers to the use of simulation approaches to be? Do any associations exist between staff perceptions of barriers and their use of the various simulation approaches?

During the first phase of primary data collection, a small number of grounding interviews were conducted with academics. These interviews assisted in the identification of issues relating to using games and simulations, and in particular their implementation in the host university. In providing a clear picture of the institutional context for using these methods, the interviews played an important role in informing the development of a questionnaire form.

The research sampling frame for the questionnaire survey comprised all academics across six faculties of the institution. Hence results would represent the characteristics and perceptions of staff teaching across many subject domains covering all years of study and a variety of class sizes. Initially, in this second phase, a pilot survey was conducted using a small sub-sample of academic staff. Feedback resulted in minor changes to the design and content of the questionnaire. Subsequently, 963 of the revised forms were distributed across the faculties. The sections significant to this study were:

Use of simulations and games: role play, non-computerized educational games, computer-based gaming simulations, computer-based training simulations, computer-based modelling simulations; Perceived barriers to the use of games and simulations: subjects were asked to gauge their level of agreement or disagreement with 10 items. Each item was scored on a five point scale from strongly agree (1) to strongly disagree (5). The items were developed from the grounding interviews held in phase one and the literature reviewed.

158 forms were returned, giving a response rate of 16.4 per cent. Whilst this response rate is relatively low, it is not atypical for surveys of this nature.

For instance, the largest study to date of the use of simulation games by academics achieved an effective response rate of 8.4 per cent (Faria and Wellington, 2004). Nevertheless, the limited number of responses and the possibility of response bias should be considered in interpreting the survey results.

Analysis of results

Use of simulations and games

Table 1 below shows results on the use of the different forms of simulation.

The table presents the proportion of staff who described themselves as current users, former users and never users. The last category is disaggregated to show those staff who would consider using the method in the future.

Role play is most prevalent amongst the learning techniques listed in the table, with 58.3 per cent of staff being either current or former users. Also, non-computerized games have fairly significant application amongst the academics surveyed. Levels of use of computerized simulation approaches are generally low in comparison to non-computerized approaches. Of those that have never used the various approaches listed, it is worth noting that a large proportion would consider using such methods in the future. This latent interest suggests that there may be real or perceived barriers to the adoption of simulation and gaming for teaching.

Perceived barriers to use

Table 2 below shows the extent to which the responding academics agreed with a series of ten statements relating to possible barriers to using simulation and gaming approaches.

The levels of agreement with the statements varied across the respondent group. A number of assertions did encourage more skewed responses in both directions (see responses to statements 1, 3, 5, 8, 9, 10). The findings show particularly strong concurrence with the assertion that limited time was available for teaching development. Nearly 82 per cent agree or strongly agree with this statement. The limited availability of resources to allow new methods of teaching can also be cited as a point upon which academics agree, with 64 per cent of respondents strongly agreeing or agreeing with this statement. Sixty-four per cent of respondents strongly agreed or agreed that limited technical or administrative support was available. The lack of awareness of available methods and products also resonated amongst the staff, with 60 per cent strongly agreeing or agreeing with this proposition. Encouragingly two statements generated responses with which staff largely disagreed; 66 per cent of respondents disagreed with the assertion that students would not react well to these learning methods, whilst 60 per cent disagreed with the statement that teaching innovation is a relatively low priority in their school.

A C T I V E L E A R N I N G I N H I G H E R E D U C A T I O N 7(3) 234

Associations between use and perceptions of barriers

To simplify the data for further analysis, scores corresponding to the 10 items on barriers to simulation and gaming use were subjected to a principal components (factor) analysis. The purpose of this analysis was to identify whether there was a smaller number of underlying variables (factors) which explain variance in the responses to the items in the survey. A more meaningful analysis of the associations between simulation and gaming use and staff perceptions of the barriers to using these approaches could then be undertaken.

L E A N E T A L . : S I M U L A T I O N S A N D G A M E S 235
The suitability of the data was confirmed by Bartlett's test of sphericity ( 2 = 250.098, df = 45, p = 0.000), and Kaiser-Meyer-Olkin's (KMO) test (KMO = 0.620, all diagonal elements of the anti-image correlation matrix > 0.48). There were four factors with eigenvalues greater than 1. However an examination of the scree plot output from the analysis suggested three main factors which represented 55.4 per cent of the total variance. These factors were extracted. An examination of the content of items loading at or above 0.5 (and using the higher or highest loading in instances where items cross-loaded at greater than this value) gave rise to the following factor labels:

Factor 1 (F1) -Suitability (25.0% of variance explained) Factor 2 (F2) -Resources (19.4% of variance explained) Factor 3 (F3) -Risk of unknown (11.0% of variance explained) Table 3 lists the three factors, their associated items and factor loadings after Varimax rotation with Kaiser normalization.

Factor 1 represents items relating to the perceived suitability of simulation and gaming for teaching and learning activities. Factor 2 contains items associated with resource issues. The final factor represents items concerned with risk. For the purpose of subsequent analysis, factor scores were calculated using the regression method (Tabachnick and Fidell, 2000: 597-600, 627). To assist in the analysis, the following null hypothesis was proposed: H 0 : There are no significant differences in staff perceptions of the barriers to simulation and gaming use between user and non-user groups.

A C T I V E L E A R N I N G I N H I G H E R E D U C A T I O N 7(3) 236
A multivariate analysis of variance (MANOVA) was conducted using the three extracted factors as dependent variables and the following independent variables: use of role play, use of non-computerized educational games, use of computer-based simulations. For the independent variables, respondents were either users (current or former) or non-users of the approach.

A marginally significant difference was observed in relation to the combined dependent variables and the use of non-computerized educational games (Pillai's Trace significance = 0.059, see Table 4). The interaction between the use of role play and the use of computer-based simulations was also significant for the combined dependent variables. A univariate analysis for each of these main effects is presented in Table 5. This indicates that the associations are with Factor 1 (Suitability) and Factor 3 (Risk).

Table 5 also reveals further significant differences relating to Factor 1 (suitability) and both the use of role play and the use of computer-based simulations. These results indicate a clear association between academics' use of different simulation and gaming approaches and their perceptions concerning suitability. There are no associations relating to perceived resource barriers. Overall, the results suggest the rejection of null hypothesis H 0.

Discussion

Within this host institution the study findings indicate that simulations, games and role play are used for a variety of teaching and learning purposes. An encouraging level of use of a variety of simulation approaches is evident. However, role play and other non-computerized games represent the most frequently used techniques. This on the face of it is quite surprising given the technical-and science-based heritage of the university. However, the results reflect the shift in focus to a broader academic portfolio of subjects over recent years, in particular a relative growth in academic programmes in the arts, humanities and social sciences.

The fact that a majority of those staff who have never used simulations would not discount their use in the future suggests that there is scope for higher levels of adoption of such approaches in the future. Such growth however is likely to be moderated by staff perceptions of the barriers to using simulations and games. Results from the survey show that resource related issues are perceived to represent significant barriers. Limited time, resources and support are the three principal obstacles to the employment of these learning techniques, though a lack of awareness of simulation approaches is also of importance. The findings relating to time and resources are understandable given the pressures placed on staff as a result of the competing demands of teaching and research (particularly the expectations and potential rewards associated with the UK Research Assessment Exercise). Of course, it is not unusual in any organizational context for time and resource limitations to be cited as factors encumbering progress and higher achievement. It is useful therefore to look at the associations between perceived barriers and actual levels of use of the various simulation techniques. Factor analysis and MANOVA results indicate homogeneity in staff perceptions pertaining to resource barriers, with no significant variations between users or non-users of the simulation approaches examined. This indicates that whilst most staff highlight resource related limitations as constraints, this does not prevent many of them from going ahead and using simulation-based approaches in their teaching. The study results suggest that in fact, the use of simulations is associated primarily with issues concerning suitability. In other words, decisions are made mainly on the basis of academic judgements regarding the appropriateness of different teaching approaches.

Other results from the survey indicate that with computer-based simulations and role play, the unknown risk of their use as perceived by academics may be associated with levels of adoption. Consideration of the factor items for Factor 3 (Risk) may lead to the suggestion that insufficient accessible information exists on the use of these approaches to provide some staff with the level of confidence required to employ them. It might also be that the outcomes of using such approaches (in terms of both possible practical difficulties and learning benefits) are regarded by staff as being less predictable than is the case for other teaching techniques. Armstrong's 2003 study certainly indicates that this may well be true of role play. These results suggest that improved promotion of simulationbased methods within HE institutions is desirable. Better information about available approaches may result in a clearer view amongst academics of the risks involved in their use, and in turn, encourage higher levels of adoption. In addition, some of the risks associated with using simulations could be reduced by employing external organizations to deliver learning inputs. The growth in the number of training organizations marketing outsourcing services to HE organizations perhaps presents opportunities to limit the risks involved in delivery. However, the costs involved are often high, and integration with other learning inputs may be challenging.

Conclusion

Overall, the findings of this study suggest that increasing resources for supporting simulation use per se may not have a significant impact on the uptake and maintenance of simulations, games and role plays as active learning approaches. The results suggest that academics make the decision to use these techniques based upon their professional judgement of benefit and risk, rather than on the resources available. If academics are motivated to use simulations, resource limitations will generally not stop them. In the light of the influence that suitability and risk seem to have on the propensity of staff to use simulations, there may be a role for awareness building activities and improved information about these approaches. This would assist academics with an interest in such approaches to make more informed judgements as to their suitability within a given learning context, the possible risks that might be involved and how these can be minimized. Further, such information could stimulate an interest in simulations amongst staff not familiar with the method.

It should be noted that the findings of this study relate to one specific HE institution and that further research across institutional boundaries would be desirable to add further validity to, or refute the findings of this study. In particular, further research might usefully investigate how universal the findings are across universities with different traditions, missions and constituencies. In addition, there would be value in assessing the association between simulation and gaming adoption and factors such as class size, student year of study, academic subject and staff characteristics.

Figure 1

 1 Figure 1 Typology of simulations

  L E A N E T A L . : S I M U L A T I O N S A N D G A M E S 237

  intensity with which these techniques are employed. The most substantive studies to date relate to the use of computer-based simulation games in business education.Faria and Wellington (2004) review several studies on the use of simulation games across US business schools.

	These show that the
	proportion of schools using simulation games has grown from 71.1 per cent
	in the early 1960s (Dale and Klasson, 1962) to 97.5 per cent of schools by
	1998 (Faria, 1998). Faria and Wellington's 2004 investigation focuses on
	the use of simulation games by individual academics. The findings show that
	30.6 per cent of the 1085 respondents are current simulation game users,
	17.1 per cent are former users and 52.3 per cent have never used simulation
	games for teaching business. These figures are higher than Chang's (1997)
	findings in Hong Kong business schools. In this study, some 24.5 per cent
	of staff are either current or former users of simulation games.

Table 1 Respondent use of different types of simulations (%)

 1

		Current	Former	Never used,	Never used,
		user	user	would	would not
				consider	consider
				using	using
	Role play	35.9	22.4	22.4	19.2
	Non-computerized games	20.5	13.9	47.0	18.5
	Gaming simulations	5.4	9.4	64.4	20.8
	Training simulations	6.5	4.5	66.5	22.6
	Modelling simulations	13.8	7.2	54.6	24.3

Table 2 Staff perceptions of the barriers to the use of simulation and gaming (%)

 2

	Strongly Agree	Neither Disagree Strongly
		agree				disagree
	1 I have limited time	32.6	48.6	9.0	6.3	3.5
	available for teaching
	development
	2 I feel that using new	4.1	13.1	23.4	40.0	19.3
	methods is risky
	3 I am not aware of available 17.2	43.4	21.4	11.0	6.9
	methods and products
	4 I am satisfied with current	5.4	30.6	33.3	29.3	1.4
	teaching methods used
	5 There is limited availability 16.4	47.9	21.9	11.6	2.1
	of resources to allow the
	use of new methods
	6 These methods are not	5.5	18.5	29.5	31.5	15.1
	suited to my subject
	7 There are no	5.0	15.7	42.1	27.1	10.0
	simulations/games
	available for my subject
	8 Students won't react well	2.8	4.2	27.1	45.8	20.1
	to these methods
	9 Teaching innovation is a	5.5	15.1	19.2	40.4	19.9
	relatively low priority in
	my school
	10 There is limited support	22.1	42.1	20.0	13.8	2.1
	available (e.g. technical
	or admin.) for new methods

Table 3 Factor loadings for barriers to simulation and gaming

 3

	Suitability
	I am satisfied with current teaching methods used	0.635
	These methods are not suited to my subject	0.782
	There are no simulations/games available for my subject	0.730
	Students won't react well to these methods	0.760
	Resources
	I have limited time available for teaching development	0.499
	There is limited availability of resources to allow the use of new methods	0.751
	Teaching innovation is a relatively low priority in my school	0.668
	There is limited support available (e.g. technical or admin.) for new methods 0.761
	Risk of unknown
	I feel that using new methods is risky	0.684
	I am not aware of available methods and products	0.813

Table 4 Significant associations between simulation and game use and combined dependent variables

 4

	Source of variance	Pillai's Hypothesis	Error	Multivariate	Sig.
		Trace	df	df	F
	Use of non-computerized 0.062	3	116	2.548	0.059
	educational games
	Use of computer-based	0.073	3	116	3.061	0.031
	simulations by use of
	role play

Table 5 Significant associations between simulation and gaming use and perceptions of barriers to simulation and gaming

 5

	Source of variance	DV	df	Mean	F	Sig.
				square
	Use of non-computerized	Suitability 1	5.092 7.034 0.009
	educational games
	Use of computer-based simulations Risk	1	8.612 9.258 0.003
	by use of role play
	Use of computer-based simulations Suitability 1	3.310 4.572 0.035
	Use of role play	Suitability 1	2.774 3.832 0.053

Acknowledgement

This study was funded by a University of Plymouth Teaching Fellowship Award.


Keywords: management, objective tests, survey, textbook publishers' student

This article presents the findings of a survey of students' usage of the objective question bank section of an academic publisher's textbook website. The findings are based on a survey of 239 business and management undergraduates conducted using a quantitative research methodology. The results suggest that increased use of the objective question bank improves students' examination performance only where it matches exactly the course assessment format. Other factors, such as usage cross-tabulated with gender and off-campus Internet access were also examined. On the basis of this survey, the authors consider the use of textbook publishers' objective question banks in teaching business and management subjects at university.

Textbook publishers' website objective question banks

Does their use improve students' examination performance?

S C OT T PAU L J O H N S TO N Civil Service, UK

A N D R Z E J H U C Z Y N S K I University of Glasgow, UK

Introduction

Textbook publishers' student websites represent an increasingly important aspect of higher education that has, to date, been neglected. These websites have been developed by, and are maintained for student use by academic publishers, specifically to accompany their individual textbooks. All the major academic publishers now host multiple textbook websites. Past research into publishers' uses of their websites is limited, and focuses upon their role in online selling directly to customers (Kipling and Wilson, 2000); their use as sales, adoption or public relations tools (Posey, 2000); how publishers put tables of content, abstracts and the full-text of their print journals onto the Web (Scheschy, 1999;Peek et al., 1998); and publishers' e-books and electronic publishing processes (Shiratuddin, 2005).

None of this literature gives any indication that publishers possess textbook websites. Tookey's (2002) review of two such websites was amongst the first in an educational academic journal, and drew attention to their existence. There are several reasons for investigating them. First, having a textbook website is now considered essential by publishers. In the last decade, the major publishing houses have all invested money into online learning resources to support their textbooks. In 1998, it was estimated that 83 per cent of UK-based publishers had a presence on the Internet (Stone, 1998), a figure that must now have reached 100 per cent. Second, probably the simplest, cheapest, fastest and most effortless way in which academics can augment their courses with a web-based, student learning resource, is to select a course textbook that has a website associated with it. Third, the increasing use of virtual learning environments (VLEs) within universities is encouraging the provision of interactive resources for students. Most VLEs contain facilities for instructors to develop their own banks of objective questions. All VLEs allow linkages to textbook publishers' question banks. Thus, if one were looking for an example of a computer-related resource that was most easily introduced into universities, then the publishers' textbook website would offer itself as an obvious candidate. Johnston and Huczynski (2004) investigated the contents of these websites. They examined 175 different textbook websites, maintained by 6 different publishing companies relating to business and management subjects. They identified 698 items which were classified into 11 distinguishable categories of learning resource, and one large, residual one. They found that, in terms of the frequency of occurrence, objective test questions (multiple choice, true-false, fill-in-the-blank, matching) represented the single largest resource available to students (see Table 1). The publishers' textbook website's objective question bank (OQB), its use and impact on student examination performance, is the focus of this study. The large residual category, containing 112 items, included a collection of items which, it appears, sought to attract students to the site. Among the items found were games, study advice, career advice, fashion hints, free music and contests to win money. The diversity of these items precluded further useful classification.

Literature review

Do students studying courses which have an associated core textbook that has an objective question bank use it? If they do, does it improve their performance in examinations? The literature review did not reveal any empirical investigations into students' use of publishers' OQBs, or into the relationship between their use and the students' examination performance in any subjects. Within the authors' own field of business and management, most past research has focused either on the more general question of the feasibility of using computer assisted learning (CAL) in business and management subjects (Bigelow, 1999;Meisel and Marx, 1999), or else has investigated student attitudes to CAL in business and management courses (Arbaugh, 2000;Human et al., 1999). There have been some studies into students' use of CAL in general. Authors like Davies andCrowther (1995), Spellman (2000) and Goffe and Soskin (2005) argue that CAL can offer a more enriching learning experience for the student, increasing their motivation to learn, and facilitating active learning. However, how often this happens in practice needs to be examined in detail as CAL materials vary considerably. Apart from publishers' textbook websites, other Internet sites provide multimedia lectures, with features such as audio and video (Bland, 2002). Haworth and Davidson (2000) argued that the greater the diversity of resources on a CAL website, the more frequently the website would be used by students, as students possessed a variety of different learning styles.

Bigelow's study of a transition from a conventional classroom-based organizational behaviour course to an Internet-based one, showed that the latter could offer learning benefits for students. These included greater selfmanaged learning, and the possibility of instant feedback for students who participated in examinations (Bigelow, 1999). Ward and Newlands (1998 Source: Johnston and Huczynski (2004: 51) also found that CAL applications gave students greater choice of when to study. Despite these advantages, Bigelow stressed that CAL may have a negative impact on activities that emphasise face-to-face interaction, though this may be reduced as computers and CAL technologies develop in the future. Merisotis and Phipps (1999) reported research which revealed that the learning outcomes of students using technology at a distance are similar to those who participate in convention classroom instruction. The authors go on to highlight the shortcomings of the comparative research on this topic.

) J O H N S T O N & H U C Z Y N S K I : P U B L I S H E R S ' Q U E S T I O N B A N K S 259
There is also a research literature that takes a more managerial approach, looking at the benefits for universities of implementing CAL applications. McInnes et al. (1995) argued that there had been increased pressure on academic staff to publish as a result of regular, institutional research assessments. Coupled with increasing student numbers and budget constraints, they suggested that an investment in CAL should only take place if it reduced costs and freed up time for academic staffs' research. Indeed,Ward and Newlands (1998) noted that CAL had a great potential to benefit student learning, but involved a considerable investment of time for lecturers who had to prepare CAL materials, if ready-to-use packages were not available commercially. McInnes et al. (1995) also argued that the use of CAL should be justified in terms of monetary cost.

What studies have been conducted of potential CAL users? Our literature search indicated that there were few studies profiling the users of CALbased business and management-related courses. Consequently, we had to draw upon studies from other social science fields. With regard to economics courses, there appeared to be no gender-specific characteristics determining student usage of CAL (Haworth and Davidson, 2000). This finding was also replicated for psychology courses (Grabe and Sigler, 2002;Wang and Newlin, 2000), and for those on an education course (Barrett and Lally, 1999). Haworth and Davidson suggest those who underperformed in class examinations may be inclined to start using CAL to correct their performance in time for the final examination. Grabe and Sigler suggested that the level of students' computer literacy was not a crucial factor in determining their use of CAL. Therefore the literature suggests that various factors can determine the use of CAL by students.

What is the relationship between using CAL and examination performance? Sangster (1992) suggested that CAL applications could take two forms: supplantive, the substitution of conventionally delivered lectures; and supportive, reinforcing traditionally delivered lectures and tutorials. The literature search indicated that there were few studies comparing the success of CAL-based business and management-related courses vis-à-vis those delivered by conventional classroom means. Consequently, studies from other social and information sciences were considered. Supportive CAL applications, such as publishers' textbook OQBs are those which reinforce traditionally delivered lectures and tutorials. The literature did not reveal any studies examining the use of supportive CAL applications within business and management subjects. In psychology, the literature suggests unanimously that supportive CAL applications do facilitate student performance. In a study by Worthington et al. (1996), students who used a supportive CAL application for an introductory psychology class attained higher examination scores. Higher examination scores were also found among students who had used supportive CAL applications for introductory psychology courses in research studies conducted by Buchanan (2000), Grabe and Sigler (2002) and Maki and Maki (2001).

With regard to economics, MacDonald and Shields (1998) showed that students' use of the WinEcon CAL software to supplement their studies improved their examination performance. Kennedy et al. (1998) showed that students who used CAL to support their introductory economics course achieved better grades than non-users. However, a study of an introductory accountancy course by McInnes et al. (1995) found no difference in examination performance between users and non-users of a CAL application. As with the literature review of McInnes et al. (1995), no students who used supportive CAL applications received inferior grades to non-users. In information science courses, McIntyre and Wolff (1998) found that students who used a supportive CAL application for an introductory computer programming course achieved higher examination grades than those who did not. Consequently, on this basis of this literature, it would appear that students who made use of supportive CAL applications, generally tended to achieve higher examination grades. If using CAL increased their examination grades, then one would expect that they would be more motivated to use it (Davies and Crowther, 1995;Spellman, 2000). The authors' expectations therefore were that students who had used the question bank, would attain higher examination grades than those who had not.

Despite such research, the authors had to take into consideration that students' entry qualifications into university as a confounding factor on this research. Superior examination and test performance may be due to the fact that a student was generally of high academic calibre, and could have nothing to do with their use or non-use of CAL. That is, those who were high achievers in the past would continue to be so in the future. However, Tuckman (2000) suggests that increased test practice will lead to increased performance, with the possibility of school under-performers gaining higher scores than those with previously higher performance at school. Consequently, respondents were also divided on the basis of their university entry grades.

The literature review revealed a total neglect of publishers' textbook websites, including their objective question bank sections. There have been studies into students' use of supportive CAL applications, but not specifically into computerized objective test banks. Those studies of CAL applications that have been carried out, suggest that their use by students does improve their examination scores. In the light of these omissions and gaps in the literature, this study had three aims. First, to identify the extent of use made by students of the publishers' textbook objective question bank that accompanied their course's core textbook. It sought to identify factors that promoted or lessened their use; and the value that students gained from it, in terms of perceived benefits and actual benefits in midterm and final examination performance. The literature suggests that CAL in general offers a wide range of advantages for the student user. Consequently, a number of hypotheses were proposed for testing with respect to the publishers' textbook objective question bank. They sought to answer three related questions. Whether those students who used the publishers' textbook objective question bank, would feel there was a good range of resources available (1a); whether they felt that open-access use allowed them greater flexibility in their study activities (1b); and whether the summary feedback facility was useful (1c).

Second, with regard to the characteristics of student users, the study sought to examine if the factors found in the literature review concerning propensity to use CAL applications, were relevant to first year, business and management students. The hypotheses sought to determine whether the degree of usage of the publishers' textbook objective question bank was influenced by students' gender or by their off-campus accessibility to the Internet. Third, with regard to possible links between students' use of the publishers' textbook objective question bank and their examination performance, the hypotheses sought to investigate whether the degree of usage of the publishers' textbook objective question bank influenced students' class test results; whether it influenced their final examination results; whether there was a correlation between university entry grades and class test performance; and finally, whether there was correlation between students' university entry grades and their examination performance.

Methodology

The first year, foundation Business and Management course at the authors' university focuses on organizational behaviour and management topics. Its course textbook (Buchanan and Huczynski, 2004) has a companion website containing a bank of objective questions. That bank currently contains over 2500 objective questions (http://www.booksites.net/buchuc). This website and its associated question bank is maintained by Pearson Education. Organised by chapter, the question bank allows students to revise the textbook material. It includes different question formats: multiple choice (at the knowledge and comprehension levels), fill-in-the-blank, matching and true-or-false. Users obtain instantaneous feedback on their performance under timed conditions. This includes their answers being displayed alongside the correct answers, as well as the time that they have taken to complete each test. The site is available continuously; can be accessed from anywhere in the world; and test results are known only to the student users (staff do not keep a record). All academic publishers promote their textbooks to potential lecturers using the argument that the associated companion website, containing objective tests, will encourage students' revision, enhance their learning and improve their overall examination performance in the course. By implication, they suggest that student performance in nonobjective question exam formats, will also be enhanced.

Like many similar courses, the present first year, foundation course currently has a multiple assessment structure, and mixes objective testing with other assessment formats, so as to assess students' achievement as widely as possible. As shown below, pencil-and-paper objective tests count for 45 per cent of the total student assessment on this course. These tests match exactly those found on the website just described:

• a class test consisting solely of objective questions, worth 20 per cent of the total course grade; • a written, 2000 word assignment, worth 30 per cent of the total course grade; • a final, degree examination consisting a set of (compulsory) objective test questions and one (chosen) written essay answer, each worth 25 per cent of the total course grade.

A sample plan was used to define the study population, ensuring the data obtained was as accurate as possible. Within this study, convenience sampling methods were used, whereby respondents were selected on the basis of convenience to the researcher. Respondents were selected for both the pilot testing and main data collection stages in this way. This was done because a sufficient sampling frame was unavailable owing to confidentiality issues related to students' personal data. Furthermore, the population was readily available, and concentrated in only two locations. Sample size affects the accuracy of the results. Owing to the fact that convenience sampling was used, the sample size reflected all the students present on the day of data collection, 284 respondents. Although the sample was biased to the extent that it reflected members of the class that were present on that day, questionnaires were distributed to everyone and all forms were returned, thus eliminating any socio-demographic biases which would have otherwise rendered the study invalid.

If further information regarding the composition of the sample had been available, the researchers could have adopted a probability sampling method. Only 86 respondents would have been needed to attain a statistical confidence level of 95 per cent from a class roll of 442.

For the collection of primary data, the quantitative research approach was favoured, owing to the large number of respondents. A questionnaire was chosen to collect the empirical data. To test the hypotheses stated earlier, a 60-item questionnaire was developed (after piloting), using variables drawn from the studies identified in the literature review. This questionnaire also included variables for additional academic research and course evaluation purposes. The majority of the questions included were response questions. They consisted of a statement, for example,'I found the question bank easy to use', followed by a five-point modified Likert interval scale, in which each descriptor in the scale is exactly one unit away from the preceding one. The exceptions to this were questions such as 'Do you have off-campus access to a personal computer?', which are nominal-scaled questions. In addition to these questions, each respondent's matriculation number was collected so that the respondents' survey responses could be anonymously linked to their class test, final examination results and university entry grades. Of the 284 questionnaires distributed, 45 questionnaires were returned incomplete or completed incorrectly. These were discarded as many of these questionnaires had boxes scored that were not an option provided in the multiple choice question, suggesting that the respondent was not interested in giving accurate answers. The final sample size of n = 239 reflects an 84.1 per cent response rate. From Table 2, it can be observed that the majority of the sample, 54.8 per cent, was female, and the majority of the sample, 47.7 per cent, was aged 18.

From Table 3, it can also be observed that the majority of respondents, 51.5 per cent, were from the Social Science faculty, and that 89.1 per cent of respondents were in their first year of university study.

Results

The results of the 239 questionnaires returned were analysed using the software package SPSS 11.5 for Windows. The data was re-coded from alphabetical to numerical form. The first stage of the statistical analysis involved calculating the percentage response for each variable for a subjective impression of the data. The mean response and standard deviation of each variable were also calculated. Next, the statistical tests for the hypotheses stated earlier were completed. For each test, assumptions underlying their use such as large sample size and independent observations were verified before they were employed. In all tests requiring comparisons between groups, utilizing a Bonferroni, multiple comparisons procedure, there was a statistically significant difference between groups if the 'p-value' of the test was less than 0.05, a benchmark widely accepted in academia and professional statistical analysis.

The results obtained from that research are presented in two steps. First, usage patterns of the question bank by students are considered. These are followed by an analysis of the results of the statistical techniques employed to test the hypotheses presented earlier. With respect to question bank usage patterns, Table 4 indicates that two-thirds of students used the bank with the majority indicating that they used it sometimes or frequently. Table 5 also indicates that the majority of students, 73.3 per cent, used it alone, with only 7.6 per cent working in a group of three or more. The following subsections will now present an analysis and discussion of the results of statistical tests employed for each individual hypothesis.

Why use the question bank?

As hypotheses 1a to 1c involve interval scales (coded from 5 = strongly agree, to 1 = strongly disagree), relevant questionnaire variables were compared in a one-sample t-test against the value 3, representing the neutral scale-item on the questionnaire, to indicate the direction of agreement or disagreement to a variable (Table 6). As all the p-values are less than 0.05, this indicates a significant difference greater or lesser than 3 on the agreement-disagreement scale, with the strength of the difference illustrated by the 95 per cent confidence interval. Consequently, in line with hypothesis 1a, students who used the question bank felt there was a good range of resources available. Furthermore, concurrent with hypothesis 1b, students who used it felt that open-access use allowed them greater flexibility in their study activities. Finally, corresponding to hypothesis 1c, students who used the question bank found the summary feedback facility useful.

J O H N S T O N & H U C Z Y N S K I : P U B L I S H E R S ' Q U E S T I O N B A N K S 265

Characteristics of question bank users

To formally test hypothesis 2a, which stated that the degree of usage of the bank is not influenced by students' gender, a two-sample chi-square test was employed. The Pearson chi-square statistic was 0.75 with a significance (p) value of 0.95. Consequently, there was no significant difference in usage of the bank in terms of gender. To formally test hypothesis 2b, which stated that the degree of usage of the bank is influenced by off-campus accessibility to the Internet, a two-sample chi-square test was employed. The Pearson chi-square statistic was 1.52 with a significance (p) value of 0.82. Consequently, there was no significant difference in usage of the bank in terms of off-campus access to the Internet.

A C T I V E L E A R N I N G I N H I G H E R E D U C A T I O N 7(3) 266

Use of question bank and examination performance

From Table 7, it appears that those who always used the bank had the highest mean class test grades, with those rarely using the bank having the lowest mean class test grades. Similarly, it appears that those who always used it had the highest mean final examination grades, with those rarely using it had the lowest mean class test grades.

To formally test hypotheses 3a, which stated that the degree of usage of the bank will affect mean class test results, and 3b, which stated that the degree of usage of the bank will affect mean final examination results, two separate One-Way ANOVAs were employed. Statistically significant differences were found in class test grades for the five degrees of usage [F = 9.24, p = 0.00]. Post-hoc comparisons indicated where those differences lie, illustrated in Table 8, with the underlined degree of usage having a higher mean. However, there were no statistically significant differences found in final examination grades for the five degrees of usage [F = 1.59, p = 0.18].

To formally test whether or not university entrance grades were correlated with class test performance, a Pearson correlation was computed for university entry qualifications data against class test performance. It was found that there was a significant correlation [r = 0.319, p = 0.00] at the p = 0.05 level of confidence. Thus, there is a low, positive relationship

Discussion

The results obtained indicate that, overall, students gave a positive assessment of the range of question bank questions and the summary feedback provided, and valued the flexibility in study activities that the bank offered.

The question bank level of usage was not influenced by either gender or off-campus accessibility to the Internet. Its increased use did appear to influence students' performance in the objective class test, between several degrees of usage. However, perhaps of greatest interest was that increased use of the objective question bank did not significantly influence students' final examination performance between several degrees of usage. That is, if you include students' performance in their degree essays conducted under examination conditions, the increased use of the question bank did not improve this.

Since the question bank focuses only on objective question testing, it may be deduced that its benefit to students is therefore only limited to objective test forms of assessments. It does not appear to improve their performance in written exam essays. This is despite the hopes and expectations of academic staff that students' familiarization with the concepts, theories, authors, models and research, through the use of objective testing, will translate into higher quality written answers. Many lecturers who use objective testing, draw upon the foreign language learning analogy. They see it as helping students to develop the 'vocabulary' of a discipline, so that they can go on to use it in 'conversations' and 'prose', equivalent to assignment writing and essay degree writing. The findings of this research suggest that this is not happening.

Future research

First, this study has considered only one class of students in one academic subject, using one question bank associated with one textbook. However, all publishers are now providing websites containing objective question banks to accompany their top-selling introductory textbooks in all subjects.

It would be useful to confirm or reject the lack of link discovered here in other subjects. Second, student performance in tests that mirrored the objective test format of the question bank did appear to be influenced by a greater degree of usage of the bank. However, further studies could explore the underlying reasons why final examination performance (which included non-objective testing) was not significantly influenced by greater usage. What exactly prevents students incorporating what they have learned for the objective test in their essay answers? Third, this study concentrated on one year-group within a single university. Wider research can determine the true extent of business and management students' usage of the publishers' textbook question banks and their academic performance. It may well be that students at different universities and in different year-groups will be affected by question bank usage in different ways. However, the findings of this study have been consistent with the literature on CAL in other academic fields.

  A C T I V E L E A R N I N G I N H I G H E R E D U C A T I O N 7(3) 268

Table 1 Frequency of resources appearing on publishers' textbook websites

	Ranking Student learning resource	No.	% of total
	1st	Other	112	64.0
	2	Web links	108	61.7
	3	Objective test questions (multiple choice)	102	58.3
	4	Updates	68	38.9
	5	Internet activities	65	37.1
	6	Objective test questions (true-false)	61	34.9
	7	Learning objectives	61	34.9
	8	Glossary	46	26.2
	9	Written answer questions	38	21.7
	10	Case studies	31	17.7
	11	Objective test questions (fill-in-the-blank)	3	1.7
	12	Objective test questions (matching)	3	1.7
	Total		698

Table 3 Faculty and year of study of the sample

 3

	Faculty	Frequency Percentage	Year	Frequency Percentage
	Arts	48	20.1	1st	213	89.1
	Law & Financial	7	2.9	2nd	18	7.5
	Studies
	Science	55	23.0	3rd	7	2.9
	Social Science	123	51.5	4th	1	0.4
	Other	6	2.5

Table 2 Gender and age of the sample

 2

	Gender	Frequency	Percentage	Age	Frequency	Percentage
	Female	131	54.8	21+	20	8.4
	Male	108	45.2	20+	19	7.9
				19+	44	18.4
				18+	114	47.7
				17+	42	17.6

Table 4 Usage of question bank

 4

		Frequency	(%)
	Always	29	12.1
	Frequently	48	20.1
	Sometimes	55	23.0
	Rarely	26	10.9
	Never	81	33.9
	Table 5 Number of users
	Number of users	Frequency	Proportion (%)
	In a group of 3 or more	12	7.6
	With a partner	30	19.1
	Alone	115	73.3

Table 6 One-sample t-tests for hypotheses 1a-1c

 6

	95% Confidence interval

* Significant at p = 0.05

Table 7 Average examination performance by usage of question bank

 7

	Degree of usage	Class test	Class test	Final	Final
		mean	std. deviation	examination	examination
		(%)	(%)	mean (%) std. deviation (%)
	Always	77.1	10.4	67.2	16.4
	Frequently	73.5	12.3	62.0	19.6
	Sometimes	67.1	12.9	61.3	19.9
	Rarely	60.7	17.0	54.4	22.8
	Never	64.4	13.1	61.3	16.2

Table 8 Multiple comparisons of class test performance

 8

	95% Confidence interval

* Significant at p = 0.05


A N T H O N Y

P B Ro M L E Y

S R B O R A N

W I L L I A M A . M Y D D E Lt O N

Investigating the baseline skills of research students using a competency-based self-assessment method

Recent government-led initiatives are changing the nature of the UK PhD to support the greater development of transferable skills. There are similar initiatives internationally. A key requirement and challenge is to effectively assess the 'baseline' skills of a cohort on entry to a research programme and then monitor their progress in personal development. This article describes an innovative methodology that combines competence model and training needs analysis theory to create an effective self-assessment tool: the Development Needs Analysis (DNA), for collection of baseline data. The DNA provides a means for effective selfassessment of skills, and is capable of highlighting particular needs of students grouped by, for example, date of birth and home vs. overseas status. The methodology is broadly applicable in determination of the baseline skills of students and allows practitioners to tailor learning and teaching to the requirements of a cohort.

Introduction

In a modern economy, based increasingly on intellectual capital, greater skills and knowledge are required of the workforce. (Sir Martin Harris, Review of Postgraduate Education, 1996) The nature of a research degree is changing from the pursuit of a scholarly piece of research focused on specialization within a discipline to a broaderbased training ground for skilled researchers who will be able to function in the modern, dynamic workplace. Within the UK, the educational funding bodies began to acknowledge this change during the 1990s based upon feedback from private sector businesses about the work skills of PhD postgraduates (Association of Graduate Recruiters, cited in [START_REF] Harris | Review of Postgraduate Education HEFCE[END_REF]. Although postgraduates were highly competent in technical expertise and problem solving, broader skills, such as communication and teamwork, were often found to be lacking.

In the UK this expanded skills agenda developed significantly in relation to research degrees, culminating in the 2002 report of Sir Gareth Roberts [START_REF] Roberts | SET for Success -The Supply of People with Science, Technology, Engineering and Mathematics Skills[END_REF]. Roberts recommended each research student should have two weeks per year of transferable skills training. Roberts' recommendations were supported with a new UK government funding scheme, acknowledged in a review of research degree programmes [START_REF] Metcalfe | Improving Standards in Postgraduate Research Degree Programmes, Higher Education Funding Councils of England[END_REF] and formalized in a revised Quality Assurance Agency Code of Practice for Research Degrees (QAA, 2004). The agenda is not restricted to the UK and is indeed international, illustrated by the Bergen Communiqué [START_REF] Bergen Communiqué | The European Higher Education Area -Achieving the Goals[END_REF] and for example the review similar to the 'Roberts' review carried out in Australia (Australian Government, 2006).

In order to clarify what skills PhD postgraduates might require, the UK Research Councils (RCUK) had produced a Joint Statement on Skills listing 36 critical skills ranging from research management to career development (RCUK, 2001). A number of universities developed skills training schemes, some integrated within degree programmes and others offering an add-on smorgasbord of training courses. However, these skills training initiatives were often implemented without a clear understanding of the effectiveness of different approaches.

A current key debate [START_REF] Team | Evaluation of Skills Development of Early Career Researchers -A Strategy Paper from the Rugby Team[END_REF]) is how to assess and monitor the effectiveness of these skills development initiatives. In order to monitor progress related to skills development it is critical to know the 'baseline' level in transferable skills of a diverse research student cohort. The questions driving the current study were:

• Can an efficient methodology be developed to obtain a valid 'baseline' assessment of transferable skills in a research student cohort? • Can differing development needs be identified in the cohort? • Can the assessment inform the university's development programme?

• Can the methodology be used to monitor progress in personal and professional development?

Building the methodology

These questions are similar to the questions an employer faces when assessing the current skills of their workforce to identify company strengths and to consider additional training needs to improve company performance. Also for consideration is the body of research on assessment in higher education [START_REF] Dochy | The Use of Self-, Peer and Co-assessment in Higher Education: A Review[END_REF][START_REF] Mcdowell | Changing Assessment Practice in Engineering: How Can Understanding Lecturer Perspectives Help?[END_REF][START_REF] Gijbels | Integrating Assessment Tasks in a Problem-based Learning Environment[END_REF]. However, assessment studies are most generally based on assessment methods in taught programmes rather than the personal and professional development of the individual.The unique approach of this study has been to review literature from the private sector reporting industrial practice in competence models and training needs analysis and apply the best practice to evaluating the effectiveness of skills development for PhD postgraduates in higher education.

There are two main approaches to workforce assessment and development: training needs analysis and competence models. Training needs analysis is seen as having a role in organizational change [START_REF] Reed | What Role Can a Training Needs Analysis Play in Organizational Change?[END_REF] and can vary in scale from a simple survey to a detailed structured process including interviews, observation and focus groups [START_REF] Mcclelland | Training Needs Assessment: An "Open-systems" Application[END_REF]. Training Needs Analysis is commonly described as a three-stage process: organizational analysis, operations analysis and personnel analysis [START_REF] Moore | Training Needs Analysis: Review and Critique[END_REF]. A whole-company approach is beyond the scope of this study as there is little opportunity for organizational or operations analysis of the university. However, personnel analysis, which looks at how well each employee is performing the tasks that make up their job, appears to be applicable. Techniques used for determining training needs include: observation -work sampling, interviews, questionnaires and tests [START_REF] Moore | Training Needs Analysis: Review and Critique[END_REF]. Within the constraints of this study the selfassessment questionnaire was considered the most viable option.

Historically, in industry, there has been an ethos that the outcome of training needs analysis is training and that performance problems are signs of deficiency. Indeed [START_REF] Gilbert | Praxeonomy: A Systematic Approach to Identifying Training Needs[END_REF] offers an equation: Deficiency ϭ Mastery -Initial Skill Repertory However, this assumption has changed with [START_REF] Wright | Needs Analysis Theory and the Effectiveness of Large-scale Government-sponsored Training Programmes: A Case Study[END_REF] pronouncing the demise of the 'needs-analysis-tied-exclusively-totraining' concept. This approach is beneficial in the case of postgraduate skills development because it can account for the wide diversity of the PhD postgraduate population. An experienced, mature PhD candidate may simply need to identify the full range of skills necessary for PhD-quality research, compare their current skills levels against these target skills and plan their own self-directed development towards improving any deficient skills. This process requires the second method of workforce assessment and development: the competence model.

Competence models are widely used in industry (e.g. in personnel appraisal and development as well as in assessment centres) to improve corporate performance through analysis of the behaviour of individuals [START_REF] Boak | Competencies for Retail Leadership: Accurate, Acceptable, Affordable[END_REF][START_REF] Boyatzis | The Competent Manager -A Model for Effective Performance[END_REF][START_REF] Dalton | Are Competency Models a Waste?[END_REF]. The starting point is for a researcher to carry out a number of 'behavioural event interviews' with individuals [START_REF] Boyatzis | The Competent Manager -A Model for Effective Performance[END_REF] and identify specific behaviours that can be related to definitions of 'more effective' and 'less effective' performing individuals. Behaviours are categorized under thematic headings which define the general 'competencies' with associated 'behavioural indicators' which clearly identify the 'level' of the competency required [START_REF] Boyatzis | The Competent Manager -A Model for Effective Performance[END_REF]. For example, a competency of 'IT proficiency' is open to interpretation: does it mean familiarity with word processing or motherboard reconstruction? Behavioural indicators can clearly specify this. There has also been work that considers 'structural' rather than 'functional' components of a competence, with competencies being considered as consisting of four elements and their relations: technology, people, organizational structure and organizational culture [START_REF] Drejer | How Can We Define and Understand Competencies and their Development?[END_REF].

In summary, with respect to a methodology for this study, the main elements of personnel assessment and development in industry which could be applicable to postgraduate skills development are competence models and training needs analysis. The present study will propose a unique methodology for dealing with the assessment of skills development and present results from the first year of implementation at the University of Manchester Faculty of Engineering and Physical Sciences (EPS), which has had a skills training programme in place since 1999 [START_REF] Barber | The Development of an Institute-wide Graduate Personal Skills Training Programme that Integrates Centrally Provided Core Skills Training with the Requirements of Individual Academic Departments, Research Councils and Professional Bodies[END_REF][START_REF] Barber | Developments in Graduate Personal Skills Training at UMIST[END_REF][START_REF] Goodison | Enhancing the Quality of Doctoral Theses[END_REF][START_REF] Bromley | An Innovative "Introduction to Research" Workshop for 1st year PhD Students[END_REF][START_REF] Brunn | Postgraduate Skills Training in Teamwork and Project Planning -A New Model Case Study on the UKGRAD TM Model[END_REF]Graduate Development Scheme, 2006).

Methods

The development needs analysis

The initial task was to develop a competence model for postgraduates. There was no need to identify the skills needed by PhD researchers from scratch through interviews because the RCUK Joint Skills Statement (JSS) provides a valid list of competencies and had the benefit of already being a recognized standard in postgraduate development. However, more detailed behavioural indicators needed to be developed for each competency. It was considered important to make the behavioural indicators clearly evidence-based, with specific examples of what type of work would have to be done to illustrate a competence in a given skill. In the interests of simplicity, it was decided to only define one level of each competency: the level that would describe the skill level expected when the researchers had completed their PhD.This level was termed that of 'an experienced PhD student'. Draft statements were generated by the authors and circulated to a panel of academics from the nine disciplines in the EPS Faculty: chemical engineering, chemistry, computer science, earth sciences, electrical engineering, materials, maths, mechanical engineering and physics. The challenge was to create behavioural indicators that were applicable to all disciplines. Of course, an additional benefit of this system could be that specific disciplines could tailor the content of the 'behavioural indicator' to the needs of their discipline, whilst still allowing generic cross-disciplinary data to be collected against the JSS competencies.

A 'skills audit' questionnaire was then developed for all new postgraduates as a form of training needs analysis. This was designed as an online tool using WebCT and asked new researchers to assess their current skills against the competencies and behavioural indicators of 'an experienced PhD student' mentioned above. Students could then rank their own skill level against the indicator on a four point scale: (1) 'good first degree graduate standard'; (2) 'a PhD student with some experience'; (3) 'an experienced PhD student'; and (4) 'a particularly able PhD student'. Thus if the new researcher felt they already had PhD-quality skills in a certain competency, they would give themselves a 'level 3' ranking (or 'Level 4' if they thought they were exceptionally skilled). If they felt they did not meet the Level 3 skill descriptor, they would have to decide, in their own interpretation, whether they were slightly skilled (Level 2) or not skilled at all (Level 1).

In light of the fact that not all training needs analysis need lead to training, the system was termed the Development Needs Analysis. This terminology reflected the variety of learning preferences in a diverse group such as this and acknowledged the importance of researchers designing their own personal development plan, which may or may not include training.

Data analysis

Data were analysed in a number of ways.The overall average score for each question was calculated for comparative purposes. The chi-square ( 2 ) goodness of fit test [START_REF] Zar | Biostatistical Analysis[END_REF] was applied to examine the distribution of scores for each question. The intrinsic hypothesis was that the overall distribution of all scores for all questions represented an average expected level of scoring for new research students which could then be used to generate expected values for the various samples being examined. These were compared to the specific observed number of each level and the chisquare statistic was calculated using the equation:

Resulting chi-square values were tested for significance for kϪ1 degrees of freedom (where kϭthe number of score levels) to determine the probability of being able to reject the intrinsic null hypothesis that the distribution of scores for any given subset of data did not vary significantly from the distribution of scores in the overall dataset. Significant differences were identified by minimum probability levels of pϽ0.05, although many differences were significant at pϽ0.001 [START_REF] Zar | Biostatistical Analysis[END_REF].

The data were also subdivided on the basis of date of birth and home vs. overseas status. The subdivisions were based on the available data recorded in the university student records system. The analyses mentioned above were then carried out on the various subdivisions of the overall dataset.

Results and discussion

Overall

The final Development Needs Analysis (DNA) is presented in Appendix 1.The 36 Research Councils' skills, or competencies, are listed in the left column and the level 3 behavioural indicators associated with each competency are listed in the right column. Each competency is referred to by the letter of the general skills area (e.g. A ϭ Research Skills and Techniques) and a number within that category. A sample screen capture from the online WebCT version of the questionnaire is presented in Figure 1.

The DNA was completed by 201 postgraduate researchers within the first three months of their research programme between October and April 2005. All 36 questions were answered by 187 students (83% of the total number of respondents).The average number of questions answered was 35.2 questions.

The overall dataset of 7067 scores (a range of 195-198 answers per question) is presented in Figure 2. The average for all scores (disregarding blanks) was 1.86, with individual student averages ranging from 1.1 to 4.0 (one student). Figure 2 shows that 64 per cent of students rated themselves above a 'good first degree graduate standard' (i.e. scores of '2','3' or '4').This could be due to the fact that many students achieve a Masters degree before beginning a PhD. However, the fact that 81 per cent of all answers were either '1's or '2's (Figure 2) shows a generally low level of scoring suggesting that students understood the basic process of the DNA in that they should not expect themselves to be expert at all PhD skills in the first stages of their research degree.

Figure 1 An example screen shot from the online Development Needs Analysis

Chi-square analysis found 12 competencies whose rating scores differed significantly from the overall score distribution (Table 1). There were six competencies in which students rated themselves significantly higher than expected from the overall dataset. It is perhaps reassuring that the area students rated themselves most highly in is Health and Safety (JSS B4). In general, students rated themselves most highly in Personal Effectiveness skills (category D: three entries) and in Research Management skills (Category C: two entries). This suggests that PhD students feel they are aware of the personal skills needed, are naturally confident that they have those skills and are highly motivated. There is an absence in the top six skills of categories A (Research Skills and Techniques), E (Communication Skills) or G (Career Management), suggesting that these are the areas in which development programmes should initially focus.

There were six competencies in which students rated themselves significantly lower than expected from the overall dataset (Table 1).They rated themselves weakest in the Communication Skill E3 (defend research outcomes) which is understandable for students at the start of their research programme.There are low rankings for understanding the Research Environment (category B: three entries) and Research Management (category A: one entry), indicating a lack of awareness from students about the context of their research nationally and internationally.

Date of birth

Date of birth was known for 194 students (Table 2). The group with the highest average rating was one of the older age classes (d.o.b.: 1970-1974; 35-39 yrs old). In fact, the three oldest groups all rated themselves significantly above the average of 1.86. The group with the lowest average rating was one of the younger age classes (d.o.b.: 1981-1982; 24-25 yrs old). This suggests that the older the student the higher their self-assessment rating. This might be expected as older students will have had more opportunity for life experience to develop their transferable skills and perhaps are generally more confident of their skills abilities.

Age does not appear to have a major effect on the rankings of specific skills (Table 2).There were only nine unique skills listed in the 30 top skills (that is, the other 21 skills were duplicated across the six age classes). Only three of [1970][1971][1972][1973][1974], A6 (progress summaries) and F2 (team-working) were unique.This suggests that the greater experience of older researchers has given them more confidence in writing and teamworking than the younger researchers, who appear to feel more creative.

There was greater variability for the bottom five self-assessed skills grouped by student date of birth (Table 2).There were 12 unique skills in the 30 skills listed by age in Table 2 and six of these were different from the overall bottomrated six skills in Table 1. The youngest age class (1982/83) had the greatest differences, with only E3 (research presentation skills) in their bottom-rated skills matching the list for the overall dataset. E3 is also the only skill from Table 1 that appears for all age classes.This is understandable since all students, regardless of age, are just starting out on their research programme and will have had little opportunity for previous experience of this competency.

Two of the unique skills listed by the youngest cohort as needing development are A4 (discipline methodologies) and C2 (library skills). A4 was also listed as a unique bottom-rated skill for the next older cohort, students born in 1981/82. These technical skills are obviously a special concern of these less experienced students. The youngest cohort (1982/83) also was the only cohort to list any skills from the Career Management section (Section G: Appendix 1). This shows a special concern of these younger students for their future careers. Again it may well be expected that more mature students have a better grasp of their career paths.

The final unique skill which appears in the bottom rankings of these age-related self-assessments is A2 (original critical thinking) which is only listed by the older students born in 1970-74. This suggests that, although having greater experience with a team-based, hierarchical workplace, they may not have had the opportunity to develop an awareness of the unique demands for highly independent PhD research.

The specific ranking order of the bottom-rated skills common to all students also varied between some age classes. B5 (understanding funding and evaluation of research) appears in the bottom five for all but the youngest group of students and is the absolutely lowest-ranked skill for two age classes (those born between 1970 and 1979). It is therefore quite clear that few research students have a good understanding of how research is funded at the start of their research programme.

Home and overseas

'Home' or 'overseas' status was only known for 52 of the respondents (Table 3). Overseas students rated themselves, on average, significantly higher than home students.

A comparison of the five top-rated skills of home and overseas students (Table 3) with those for the overall dataset (Table 1) shows little difference. Nine of the 10 skills top-rated for these students occur in the overall listing, suggesting there are no major differences between home and overseas students. The only unique skill listed was for overseas students, who felt more confident of their skills for D7 (show initiative, work independently/be self-reliant).

However, there were differences in the specific rank order of top-rated skills between home and overseas students. Home students ranked D3 (flexibility/open-mindedness) and D5 (self-discipline) in their top-rated skills, whilst overseas students ranked C3 (bibliographic skills) in their top five.

An analysis of the bottom five skills for overseas students (Table 3) is more clearly polarized than any of the other lists discussed previously. Only six of the 10 cells appeared in the overall list of bottom-rated skills (Table 1). Overseas students seemed uniquely concerned about their Career Management (G2, G4) and their knowledge of the Research Environment (B2, and uniquely, B1 and B5 in common with the overall bottom-rated skills). The listing for home students had four of the five bottom-rated skills in common with the overall dataset with the exception of G1 (professional development). This may suggest that CPD activity had not been a priority for this group in previous occupations or periods of study.

127 B R O M L E Y E T A L . : S E L F -A S S E S S M E N T O F R E S E A R C H
The ranking order of the bottom-rated skills also varied between home and overseas students. Home students ranked themselves lowest in two skills from category E, Communication Skills (E3 and E4). Interestingly, overseas students gave themselves an average of 2.04 for skill E4 and 1.78 for skill E3, and thus do not appear to consider themselves deficient in communication skills in spite of potential language differences.

Conclusions

The application of competence model theory from private sector workforce assessment and development has proven useful in the context of postgraduate skills development for PhD researchers.The importance of linking behavioural indicators to the competency statements of the UK Research Councils has provided clear, evidence-based guides towards defining the skill requirements for PhD researchers. The design of the Development Needs Analysis appears to give new researchers a starting point in identifying their strengths whilst also providing an insight to skill areas in need of further development to meet the PhD standard.

The data generated by the DNA also demonstrate the efficiency of the methodology in providing an interesting insight into the baseline competence for research students on entry to a PhD programme. The data illustrates that the DNA provides an effective methodology for self-assessment and is able to highlight differences between specific groupings within a student cohort. The strengths and weaknesses of the cohort can be defined and this information can be used to inform subsequent development programmes planned by the institution. By requiring the student cohort to complete the DNA at regular intervals it should be possible to monitor progress in development against the JSS. Some further exploration is required in the verification of these selfassessments in view of known differences in how people perceive their own abilities and their actual ability levels [START_REF] Kruger | Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-assessments[END_REF][START_REF] Ackerman | What We Really Know About Our Abilities and Our Knowledge[END_REF][START_REF] Dunning | Why People Fail to Recognize their Own Incompetence[END_REF][START_REF] Eva | How Can I Know What I Don't Know? Poor Self Assessment in a Well-defined Domain[END_REF]. It would be of interest in future studies to qualify data through interviewing students [START_REF] Boyatzis | The Competent Manager -A Model for Effective Performance[END_REF], observing skills such as competence in presentation and applying other skills assessment techniques such as Executive Skills Profile (ESP) [START_REF] Boyatzis | From Learning Styles to Learning Skills: The Executive Skills Profile[END_REF][START_REF] Mcclelland | Training Needs Assessment: An "Open-systems" Application[END_REF]. It would also be of interest to explore links between training programmes, learning styles and personal development [START_REF] Mainemelis | Learning Styles and Adaptive Flexibility: Testing the Experiential Learning Theory[END_REF]. The Development Needs Analysis was based on generic competencies for all PhDs (RCUK, 2001). Now that the methodology has been established, it would be interesting to look at competencies created for specific disciplines and locally, working with students and academics in a given university. Within this process the effect on organizational change linked to a specific university could be considered [START_REF] Reed | What Role Can a Training Needs Analysis Play in Organizational Change?[END_REF] and more in-depth information about individuals completing the DNA could be obtained.

For postgraduate development, self-assessment is only the start of a process which should support student interactions with their supervisor. The supervisor role requires corroboration, feedback and guidance to students in planning their development needs. However, the DNA places the rightful responsibility of initially considering their own competencies with the research students themselves in order to take charge of their own development.

The methodology of combining aspects of training needs analysis with competence models to produce the Development Needs Analysis is applicable broadly to learning and teaching programmes. It is suggested that a well-designed DNA completed by a student cohort at the beginning of any study programme with a significant transferable skills element would be an extremely valuable tool in informing both the students and the 'teacher' about learning needs in comparison to the expected learning outcomes of a given programme. For practitioners, any specified learning outcomes for a learning and teaching activity may be easily expandable to form a Development Needs Analysis. Taking skills-based learning outcomes as desired competencies, with the addition of illustrative 'behavioural indicators' for each competency, may provide a simple self-assessment analysis tool in the first instance. This would be particularly useful as a method for gaining a rapid overview of the skills base of a large cohort when the opportunity for significant one-to-one interaction is limited.

Figure 2

 2 Figure 2 Distribution of answers to all 36 skills questions from 201 research students

Table 1 Summary of self-assessed skills for the overall data set: (A) top six skills; (B) bottom six skills

 1

	JSS skill	Average score	Chi-square	Significance (dfϭ3)
	(A) Top-rated skills
	B4	2.31	73.06	pϽ 0.001
	D1	2.24	45.85	pϽ 0.001
	C4	2.19	35.88	pϽ 0.001
	D3	2.09	20.06	pϽ 0.001
	D5	2.08	16.95	pϽ 0.001
	C3	2.04	12.68	pϽ 0.01
	(B) Bottom-rated skills
	E3	1.50	41.96	pϽ 0.001
	B5	1.55	31.46	pϽ 0.001
	B1	1.57	28.10	pϽ 0.001
	B7	1.58	28.27	pϽ 0.001
	E4	1.64	14.22	pϽ 0.01
	A1	1.69	12.47	pϽ 0.01

JSS skill references are defined in Appendix 1

Table 2 Summary of self-assessed skills by date of birth: (A) top six skills; (B) bottom five skills

 2 these skills were different from the list of overall top six skills in Table1: for the youngest age group (1982/83), D2 (creativity/originality) was unique and for the second to the oldest group (

	Cohort	1982/83 1981/82 1980/81 1975-79 1970-74 1969 or earlier
	Avg. score	1.87	1.66 **	1.89	1.90 **	2.20 **	1.89 *
	Number in	15	47	50	54	18	10
	Sample
	(A) Top-rated skills
	1st	C4	B4	B4	D1	D5	C4
	2nd	B4	D1	D1	C4	B4	B4
	3rd	D1	D3	C4	B4	A6	D1
	4th	D3	C4	C3	D5	D1	C3
	5th	D2	C3	D3	C3	F2	D3
	(B) Bottom-rated skills
	1st	E3	E3	B1	B5	B5	E3
	2nd	A4	B7	B5	B1	B7	B1
	3rd	G2	E4	E3	E3	E3	E2
	4th	G4	B5	B7	B7	B1	E4
	5th	C2	A4	E4	A1	A2	B5

Chi-square significance values for average score values are shown as * ϭpϽ 0.05 and ** ϭpϽ0.001. JSS skill references are defined in Appendix 1

Table 3 Summary of self-assessed skills by date of birth: (A) top six skills; (B) bottom five skills

 3

	Cohort	Home	Overseas
	Number in sample	28	24
	Avg. score	1.68 **	2.05 **
	(A) Top-rated skills
	1	B 4	C 4
	2	D 1	B 4
	3	D 5	D 1
	4	C 4	C 3
	5	D 3	D 7
	(B) Bottom-rated skills
	1	E 4	B 5
	2	E 3	G 4
	3	B 5	B 2
	4	G 1	B 1
	5	B 7	G 2
	Chi-square significance values for average score values are
	shown as * ϭpϽ0.05 and ** ϭpϽ0.001. JSS skill references are
	defined in Appendix 1

B R O M L E Y E T A L . : S E L F -A S S E S S M E N T O F R E S E A R C H

ϭ (observed Ϫ expected) 2 expected

Appendix 1

The Development Needs Analysis (DNA) used in this study. The section titles and the reference codes (e.g. A1) refer to the JSS (RCUK, 2001) and the abbreviated skills are shortened forms of the full JSS skills. The behavioural indicator describes level 3, an experienced PhD student, on the four point scale described in the methods WILLIAM MYDDELTON is an Electronic Resources Officer in the Graduate Development Team. He is experienced in web design, the delivery of online learning materials and quality assurance. Currently, his main work involves the interpretation of traditional paper-based methods in an electronic environment. He is currently working as the Virtual Learning Environment Support Officer at the University of London.

Ref