Marta Crispino

Elja Arjas

Valeria Vitelli

Natasha Barrett

Arnoldo Frigessi

A BAYESIAN MALLOWS APPROACH TO NON-TRANSITIVE PAIR COMPARISON DATA: HOW HUMAN ARE SOUNDS?

Keywords: Non-transitive pairwise comparisons, ranking, Mallows model, Bayesian preference learning, recommender systems, musicology, acousmatic experiment

We are interested in learning how listeners perceive sounds as having human origins. An experiment was performed with a series of electronically synthesized sounds, and listeners were asked to compare them in pairs. We propose a Bayesian probabilistic method to learn individual preferences from non-transitive pairwise comparison data, as happens when one (or more) individual preferences in the data contradicts what is implied by the others. We build a Bayesian Mallows model in order to handle non-transitive data, with a latent layer of uncertainty which captures the generation of preference misreporting. We then develop a mixture extension of the Mallows model, able to learn individual preferences in a heterogeneous population. The results of our analysis of the musicology experiment are of interest to electroacoustic composers and sound designers, and to the audio industry in general, whose aim is to understand how computer generated sounds can be produced in order to sound more human.

1. Introduction. We consider experiments involving a set of assessors (experts, judges, users) who express preferences about a set of items. Each assessor is shown a predetermined sequence of pairs of items, one pair at a time, and chooses from every pair the item that she prefers. Preference is here interpreted in a broad sense as an order relation. The assessors act independently, and typically di↵erent sets of pairs are presented to di↵erent assessors, varying also their order. An assessor does not have the possibility to go back and check the answers she gave previously, let alone change any answer later. Under such circumstances, often some answers given by an individual assessor, when considered afterwards jointly, do not satisfy logical transitivity of preferences [START_REF] Tversky | Intransitivity of Preferences. Preference, Belief, and Similarity[END_REF], that is, they may contain a pattern of the form x y , y z but z x. On the other hand, neither are the answers given by an individual assessor independent, because conscientious assessors will generally try to follow some logic in their expressed preferences.

Pair comparisons are preferred to ratings or full rankings of a set of items when there are many items to be compared, or when the relative di↵erences between them are small: in both these cases assessors are unlikely to be able to inspect and compare all items jointly in order to perform a full ranking. A pairwise comparison test is then often preferred, and sometimes it is the only possible experimental procedure [START_REF] Agresti | Categorical data analysis[END_REF].

In this paper we consider pairwise comparison data coming from an experiment where each assessor was asked to hear a series of two di↵erent abstract sounds, and to tell which one was perceived as more human. Each subject only performed a limited number of comparisons, leading to sparse data, where not all pairs of sounds were compared by each assessor. The results of this test are relevant for musicologists, composers and sound designers, whose aim is to understand how human performance expression can be communicated through spatial audio, leading to computer generated sounds appearing more life-like. Although every sound can be regarded as 'spatial' in that sound waves propagate through space, the term 'spatial audio' is here used to describe the way sound captures the physical movement in 3-D needed to produce it. The cohort of listeners who took part in the experiment had varying backgrounds, ranging from musicologists to nonspecialized university students. Therefore we expected listeners to cluster into groups, sharing di↵erent opinions about the degree of human causation behind sounds. In addition to the grouping of the listeners around a shared consensus ranking of the "humanness of sounds", we were interested in studying the association between individual listeners' rankings and their own musical experience or musical background. This application is described in detail in Section 2.

Non-transitivity can arise for many reasons, for example assessors' inattentiveness, uncertainty in their preferences, and actual confusion, even when one specific criterion for ranking is used. These situations are so common that most pairwise comparison data are in fact non-transitive at the individual level, thus creating a need for methods able to predict individual preferences from pairwise choices that lack logical transitivity, and only involve a very limited number of pair comparisons. Notice that the kind of non-transitivity that we consider in this paper regards only the individual level preferences. A di↵erent type of non-transitivity arises when aggregating preferences across assessors, as under Condorcet (Marquis of [START_REF] Marquis Of Condorcet | Essai sur l'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix[END_REF] or Borda (de Borda, 1781) voting rules.

We propose a new method for the analysis of pairwise comparison data that may contain non-transitive individual pairwise comparisons. The method is based on the classical Mallows rank model [START_REF] Mallows | Non-Null Ranking Models[END_REF] and builds on its recent extension introduced by [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF]. Given pairwise data provided by a collection of individual assessors, the method outputs Monte Carlo samples from the joint posterior distribution for the individual full rankings of all items and an assumed shared consensus ranking between them. In Section 3.3, this hierarchical structure is further relaxed by introducing a mixture model allowing for clustering of the assessors.

The key ingredient, compared to [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF], is to add to the model hierarchy one more layer of latent variables, accounting for the possibility that the assessors can make mistakes. By a mistake we mean that the order from an assigned pairwise comparison is reported in a way which is not consistent with the assessor's own 'true' full ranking, whose existence is assumed in the model. The rationale behind our model can be explained as follows. In an ideal situation an assessor would be fully conscious of her preference ordering of all items, and then simply report the consequent ordering each time a pairwise comparison is requested. More realistically, however, she becomes aware of her potential ranking of the items only progressively in time as more pairs are presented to her for comparison. Then it becomes increasingly more di cult to remember exactly what items had been shown earlier and how they had been ordered, with the consequence that reporting results from pairwise comparisons that do not respect transitivity becomes more and more likely. Under such circumstances, particularly when the number of items is larger, the pair comparison data will almost inevitably contain some answers which do not satisfy the requirement of logical transitivity with the rest. Technical errors, such as mistakes in typing, or concentration errors may also occur.

To describe such imperfections in the assessments, we introduce two alternative variants (described in Sections 3.1 and 3.2) of the probabilistic model for mistakes:

1. The probability of making a mistake is constant, independent of the pairs being assessed, and independent of all other comparisons made by the same assessor. 2. The probability of making a mistake depends on the items being compared, and is higher for pairs which are more similar to each other.

The literature on inferential models for non-transitive pair data arising at the individual level is limited and discussed in Section 5. As far as we can see, the present paper stands out as the only approach to non-transitive pair data, when the individual hidden rankings are of interest, the same pairs are not repeatedly assessed by each assessor and are few, and a Bayesian approach is of interest. One important feature of our Mallows model is the possibility to choose, for the considered specific application, an appropriate distance function. Some problems require a distance able to measure only the disorder in the given domain, while in others a distance more suited for learning preferences in a population would be preferred. In the former case, the Cayley distance [START_REF] Cayley | LXXVII. Note on the theory of permutations[END_REF] would be a natural choice, while Kendall [START_REF] Kendall | A new measure of rank correlation[END_REF] and footrule [START_REF] Spearman | The proof and measurement of association between two things[END_REF] would have advantages in the latter. For instance, consider the two rankings 1 = (1, 2, 3, 4, 5) and 2 = (5, 2, 3, 4, 1), where the top and bottom elements of 1 are reversed in 2 . The normalized Cayley distance between 1 and 2 is 0.25, while the normalized Kendall distance is 0.7. If 1 and 2 represent the rankings of two assessors of five movies, Kendall's distance may be more appropriate, as these rankings represent very di↵erent profiles: one of the two assessors likes most the movie that the other assessor likes least, and vice versa. However, 1 and 2 di↵er by a unique translocation: if they represent genomes, we could consider these rankings as very similar and be more eager to use the Cayley distance as metric in the Mallows model. For a detailed description of the distances mentioned and of their properties we refer to Diaconis (1988, Chapter 6).

Our method provides the posterior distribution of the consensus ranking, as well as the posterior distribution of the latent individual rankings for each assessor. The consensus ranking can be seen as a model-based Bayesian aggregation of individual preferences of a group of assessors. It is analogous to the quantities which are usually of interest in the rank aggregation literature [START_REF] Negahban | Iterative ranking from pair-wise comparisons[END_REF][START_REF] Dwork | Rank aggregation methods for the web[END_REF][START_REF] Kenyon-Mathieu | How to rank with few errors[END_REF][START_REF] Rajkumar | Ranking from Stochastic Pairwise Preferences: Recovering Condorcet Winners and Tournament Solution Sets at the Top[END_REF]. The estimated posterior distributions of the individual rankings can be of great interest, for example, when performing personalized recommendations, or in studying how individual preferences change with assessor related characteristics.

This paper is organized as follows. In Section 2 we describe the application which motivated this study, and then in Section 3 we present our model for the statistical analysis of the consequent data. Numerical inference is based on a Markov Chain Monte Carlo algorithm, outlined in Section 4. Section 5 gives a short overview on other methods for the description and analysis of pairwise comparisons. Section 6 is devoted to simulations, while in Section 7 we apply our method to the sound data, showing that the model identifies meaningful clusters of listeners, with similar perception of electroacoustic sounds. Finally, in Section 8 we summarize the contributions of this paper.

2. Acousmatic music experiment. Acousmatic music is a type of electronic music composed for presentation using loudspeakers, as opposed to live or video recorded performance. The composer manipulates digitally recorded sounds, so that the cause of the sound, being a musical instrument or any other sound making system, remains hidden. Indeed, when sounds are played over loudspeakers there are no visual cues to help listeners understand how the sounds were made. On the other hand, when we hear the sound of musical instruments or sounds from our everyday environment, we are able to recognize their cause, since in visual music we obtain the information that indicates the sounding object, i.e. its causation. Since the advent of recording technology, abstract sounds (that is, sounds transformed with computer tools) have been used in much of the sound-world we experience over the Internet, TV and film.

The question of interest is related to the ability of listeners to identify the presence of human causation through the spatial behavior of abstract sounds. Spatial in this context describes the fact that the causation of sound happens as an action in 3-D space. The starting point for the experiment was a high-speed motion tracking recording of the physical movement used to produce one selected sound: a cellist bowing a down-bow chord. Features of this 3-D movement were successively subtracted, resulting in a series of 12 motion data-sets of varying proximity to the original. The motion data were then made audible by a process called parameter-mapping sonification [START_REF] Grond | Parameter Mapping Sonification[END_REF], where parameters in the data are mapped to parameters controlling computer generated sound. The mapping rules are chosen to draw on our everyday perception of spatial motion, which involves not only absolute 3-D spatial location but in addition changes in volume, intensity and pitch, correlated with changes in proximity and speed. In other words, listeners heard the physical spatial motion through sonification, rather than hearing the sound that the motion created, which, in this instance, was the sound of the cello. Testing how listeners perceive a sound for which we lack a clear and commonly understood descriptive vocabulary is problematic. Therefore pair comparisons is the most appropriate design.

2.1. Pair comparison experiment. The total number of stimuli was 12. Test stimulus 1 (S1) was designed to most clearly sonify all features of the data. Each of the other 11 test stimuli were sonified by modifying one or more features of the data. This involved removing pitch and volume variation, flattening directional changes in the motion, or slowing the overall motion speed (as summarized in Table 1).

Each of the 46 listeners involved in the experiment was exposed to 30 pairs of these sounds, which is ca. 45% of the total number of possible pairs of 12 stimuli. The pairs were chosen randomly, without repetitions, and independently for each assessor. The items in each pair were played in randomized order.

Listeners were then asked to indicate, for each pair, which of the two stimuli most evoked a sensation of human physical movement of any kind, to follow their feelings, rather than imagining to watch a performance. The listeners were not told that the source motion stemmed from a cellist, nor were they asked to identify a specific human spatial movement. Each listener carried out the test sitting centrally to the loudspeaker array. Prior to the experiment, listeners were presented with a short training session of three sounds not used in the test sequence. When the experiment began, the pairs of sounds were played sequentially, listeners noted their answers on a chart, selecting the first or the second from each pair of unlabeled sounds, and were requested to always make a choice even if they found it di cult to decide. If needed, they could ask to hear a test pair for a second time. At the end, they were asked to complete two questionnaires, the aim of which was to assign a Musical Sophistication Index score (MSI) and a rating of Spatial Audio awareness (SAA) to all the listeners. The MSI used was the Ollen musical sophistication index [START_REF] Ollen | A criterion-related validity test of selected indicators of musical sophistication using expert ratings[END_REF], which is an online survey that tests the validity of 29 indicators of musical sophistication. The SAA index consisted of five questions as indicators of how aware listeners were of spatial audio regardless of musical background. Such a test did not exist in the literature, and was custom designed for the experiment.

The choice to rely on a pairwise comparison experiment is crucially based on the listeners' lack of experience with abstract sounds. It is easier for the participants to compare two sounds, rather than to be exposed to sev-eral, which could create confusion. The experiment, indeed, was di cult as expected: 37 listeners (80%) reported non-transitivities in their pair comparisons, only 9 out of 46 listeners were able to stay consistent with themselves.

A complete description of the background, hypotheses, experimental setup, and discussion of results in [START_REF] Barrett | The Impact of 3-D Sound Spatialisation on Listeners' Understanding of Human Agency in Acousmatic Music[END_REF].

3. Bayesian Mallows models for non-transitive pairwise comparisons. We consider the situation where N assessors independently express their preferences between pairs of the n items in O = {O 1 , ..., O n }. In many situations of practical interest the assessors do not decide on the set of pairs to be considered, which are instead assigned to them by an external authority. In this paper we decided not to model the way in which the pairs are chosen, and simply assume that each assessor j receives a di↵erent subset C j = {C j1 , ..., C jM j } of M j  n(n 1)/2 random pairs. Let B j = {B j1 , ..., B jM j } be the set of pairwise preferences given by assessor j, where B jm is the order that assessor j assigned to the pair C jm . For example, if

C jm = {O m 1 , O m 2 }, it could be that B jm = (O m 1 O m 2 ), m 1 , m 2 2 {1, ..., n}, meaning that item O m 1 is preferred to item O m 2 .
Such data can be incomplete since not all items, nor pairs, are always handled by each assessor. We assume no ties in the data, that is, assessors are forced to express their preference for all pairs in the list C j assigned to them, and indi↵erence is not permitted.

We denote a generic ranking by r = (r 1 , ..., r n ) 2 P n , where r i 2 {1, ...n} is the rank of item O i (the most preferred item has rank r i = 1), and P n is the space of n-dimensional permutations. A widely used distance-based family of distributions for ranks is the Mallows model [START_REF] Mallows | Non-Null Ranking Models[END_REF][START_REF] Diaconis | Group representations in probability and statistics[END_REF]. According to the Mallows model, the probability density of a given ranking r = (r 1 , ..., r n ), here denoted by Mal(⇢, ↵), is given by

(1) f R (r | ↵, ⇢) := exp[ ↵ n d(r, ⇢)] Z n (↵) 1 Pn (r).
In (1), ⇢ 2 P n is the location parameter representing the shared consensus ranking, ↵ > 0 is the scale parameter measuring the concentration of the data around ⇢, and d(•, •) is a distance function between two n dimensional permutations that satisfies right-invariance [START_REF] Diaconis | Group representations in probability and statistics[END_REF]

, i.e., d(r, ⇢) = d(r r 0 , ⇢ r 0 ), 8r, r 0 , ⇢ 2 P n , where ⇢ r 0 = ⇢ r 0 = (⇢ r 0 1 , ..., ⇢ r 0 n ).
Rightinvariance is crucial since from this property it follows that the partition function of (1) does not depend on the location parameter, and can then be written as [START_REF] Mukherjee | Estimation in exponential families on permutations[END_REF]). When the distance function in (1) is chosen to be the Kendall, the Cayley, or the Hamming distance, the partition function of the Mallows model is available in closed form [START_REF] Fligner | Distance based Ranking Models[END_REF]. For this reason, most of the work on the Mallows was limited to these distances (see, for example, [START_REF] Fligner | Distance based Ranking Models[END_REF], [START_REF] Lu | E↵ective Sampling and Learning for Mallows Models with Pairwise-Preference Data[END_REF], Irurozki, Calvo andLozano (2016, 2014)). The Mallows with other distance functions was less treated because of its computational complexity. Recently, [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF] gave a procedure to compute Z n (↵) when the footrule and Spearman distances are used, either exactly (up to some moderate values of n), or approximated through an Importance Sampling technique. The authors set the original Mallows model in a Bayesian framework, also allowing for data in the form of transitive pairwise comparisons. We generalize their model (described in Section 4.2 of Vitelli et al. ( 2018)) to handle non-transitive pairwise comparisons.

Z n (↵) = P r2Pn exp ↵ n d(r, 1 n ) , where 1 n = (1, ..., n) (see for example
The main assumption is that each assessor j has a personal latent ranking,

R j = (R j1 , ..., R jn ) 2 P n , distributed according to the Mallows density (1), R 1 , ..., R N |⇢, ↵ i.i.d
⇠ Mal(⇢, ↵). We model the situation where each assessor j, when announcing her preferences, matches the items under comparison with her latent ranking R j . Then, if the assessor is consistent with R j , the pairwise orderings in B j are induced by R j according to:

(O m 1 O m 2 ) () R jm 1 < R jm 2 , (2)
where R jm i denotes the rank of item O m i in R j . In this case the set of pairwise orderings B j contains only mutually compatible (a.k.a. transitive) preferences, since the preferences are induced from a complete ranking in P n that, by definition, is transitive. The transitive closure of a set of pairwise preferences, denoted by tc(B j ), is the smallest set that consistently extends the original preference set: it is defined as the set union of B j and all pairwise preferences that are not explicitly given but are induced from B j by transitivity. In this case it is possible to first compute tc(B j ), and second, to make inference on the posterior distribution of the Mallows parameters by integrating out all the rankings r 2 P n that are compatible with the transitive closure of the preference sets, denoted by r tc(B j ),

(3)

⇡(↵, ⇢| B 1 , ..., B N ) / ⇡(↵)⇡(⇢) N Y j=1 2 4 X r tc(B j ) f R j (r| ↵, ⇢) 3 5 .
This setting was described in [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF], Section 4.2. If the assessor is not fully consistent with her latent ranking, the pairwise orderings in B j may not be mutually compatible. In such a case the transitive closure may not exist and the previous procedure cannot be followed.

Therefore a model able to account for non-transitive patterns in the data is needed in this setting.

We propose a probabilistic strategy based on the assumption that nontransitivities are due to mistakes in deriving the pair order from the latent raking R j . The likelihood assumed for a set of preferences B j (analogous to the summation of eq. ( 3)) is

(4) f (B j |↵, ⇢) = X r2Pn f (B j , R j = r|↵, ⇢) = X r2Pn f R j (r| ↵, ⇢)f (B j |R j = r),
where f (B j |R j = r) is the probability of ordering the pairs in C j as in B j (possibly generating non-transitivities), when the latent ranking for assessor j is R j = r. It can therefore be seen as forming the error model in this context, which will be specified below. The joint posterior of the model parameters is then:

⇡(↵, ⇢| B 1 , ..., B N ) / ⇡(↵)⇡(⇢) N Y j=1 2 4 X r2Pn f R j (r| ↵, ⇢)f (B j |R j = r) 3 5 . (5)
In this paper we have assumed a gamma prior, ⇡(↵) = ( ) ↵ 1 e ↵ 1 R + (↵), for ↵, and the uniform prior on

P n , ⇡(⇢) = 1 Pn (⇢) n! , for ⇢.
This strategy is able to recover possible linear orderings close (in terms of some given distance) to the non-transitive sets of preferences. We developed two basic models for the probability of making a mistake: the Bernoulli model (BM) and the Logistic model (LM). BM assumes that non-transitivities arise from random mistakes while LM assumes that nontransitivities arise from mistakes due to di culty in ordering similar items.

Bernoulli model (BM).

Assume that the pairwise comparisons given by an assessor are conditionally independent given her latent ranking R j ,

f (B j |R j = r) = M j Y m=1 f (B jm |R j = r). (6)
We define here a function of a given comparison

B jm = (O m 1 O m 2 ), and of a given ranking r = (r 1 , ..., r n ) 2 P n , g(B jm , r) = 1(r m 1 > r m 2 )
, where m 1 is the index of the preferred item O m 1 in the m-th comparison B jm of assessor j, and m 2 is the index of the less preferred item. Thus g(B jm , r) = 1 if the preference order of B jm contradicts with that implied by the ranking r (in the sense of eq. ( 2)).

We then assume the following Bernoulli type model for modeling the probability that an assessor j makes a mistake in a given pairwise comparison B jm , that is the probability that she reverses the true latent preference implied by her latent ranking R j :

P(B jm mistake | ✓, R j = r) = P(g(B jm , r) = 1 | ✓, R j = r) = ✓, ✓ 2 [0, 0.5).
Eq. ( 6) is then given by

f (B j | ✓, R j = r) = ✓ ✓ 1 ✓ ◆ P M j m=1 g(B jm ,r) (1 ✓) M j .
We assign to ✓ the truncated Beta distribution on the interval [0, 0.5) as prior, with given hyperparameters  1 and 

2 : ⇡(✓) / ✓  1 1 (1 ✓)  2 1 1 [0,0.5) (✓)
, conjugate to the Bernoulli model. We choose the truncated Beta mainly for identification purposes, but this choice is also motivated by the fact that we want to force the probability of making a mistake to be less than 0.5.

Let B 1:N be a shorthand for B 1 , ..., B N , and R 1:N for R 1 , ..., R N . The posterior density of the model parameters, defined on the support

S = 1 ⇣ {↵ > 0} \ {⇢ 2 P n } \ {R j 2 P n } N j=1 \ {0  ✓ < 0.5} ⌘ , has the fol- lowing form, ⇡(↵, ⇢, ✓|B 1:N ) / ⇡(↵)⇡(⇢)⇡(✓) N Y j=1 2 4 X r2Pn f R j (r|↵, ⇢)f (B j |✓, R j = r) 3 5 . (7)
We sample from the density of eq. ( 7) through an augmented sampling scheme, by first updating ↵, ⇢ and ✓ given B 1:N and R 1:N , and then updating R 1:N given ↵, ⇢, ✓ and B 1:N . The former step is performed by using the conditional density

⇡(↵,⇢, ✓|B 1:N , R 1:N ) = ↵ 1 e ↵ ⇣ + 1 n P N j=1 d(R j ,⇢) ⌘ N ln[Zn(↵)] • ✓ ✓ 1 ✓ ◆  1 1+ P N j=1 P M j m=1 g(B jm ,R j ) (1 ✓)  2 + 1 2+ P N j=1 M j . (8)
The second step is performed by using the density

⇡(R 1:N |↵, ⇢, ✓, B 1:N ) / ⇡(R 1:N |↵, ⇢)⇡(B 1:N |✓, R 1:N ) = = e ↵ n P N j=1 d(R j ,⇢) [Z n (↵)] N ✓ ✓ 1 ✓ ◆ P N j=1 P M j m=1 g(B jm ,R j )
(1 ✓)

P N j=1 M j . (9)
3.2. Logistic model (LM). The idea behind the logistic model for mistakes is that an assessor j is more likely to be confused (and consequently to make a mistake) if two items in a pair are more similar according to her latent rank vector R j . We assume the following logistic type model for the probability of making a mistake in a given pairwise comparison logit

P ⇣ B jm mistake R j , 0 , 1 ⌘ = 0 1 d R j ,m 1 n 2 ,
where d R j ,m is the `1 distance of the ranks of the two items under comparison in B jm , according to R j : if

B jm = (O m 1 O m 2 ), then d R j ,m = |R jm 1 R jm 2 |.
We assume that 1 and 0 are a priori independent and distributed according to a gamma prior, 1 ⇠ ( 11 , 12 ), and 0 ⇠ ( 01 , 02 ). These choices are motivated by the fact that we want to model a negative dependence between the distance of the items and the probability of making a mistake ( 1 > 0), and second, we want to force the probability of making a mistake when the items have ranks di↵ering by 1 to be less than 0.5 ( 0 > 0). The posterior density of the model, defined on the support

S = 1 ({↵ > 0} \ {⇢ 2 P n } \ {R 1:N 2 P n } \ { 1 > 0} \ { 0 > 0}), is then ⇡(↵, ⇢, 0 , 1 |B 1:N ) / ⇡( 0 )⇡( 1 )⇡(⇢)⇡(↵) • N Y j=1 2 4 X r2Pn f R j (r|↵, ⇢)f (B j | 0 , 1 , R j = r) 3 5 . (10)
Analogously to eq. ( 7), we sample from the posterior of eq. ( 10) by first updating ↵, ⇢, 0 and 1 , given B 1:N and R 1:N , i.e. from

⇡(↵, ⇢, 0 , 1 |B 1:N , R 1:N ) / ↵ 1 01 1 0 11 1 1 Q N j=1 Q M j m=1 " 1 + e 0 1 d R j ,m 1 n 2 # • e ↵ ⇣ + 1 n P N j=1 d(R j ,⇢) ⌘ N ln[Zn(↵)] 0 h 02 + P N j=1 P M j m=1 g(B jm ,R j ) i • e 1 h 12 + 1 n 2 P N j=1 P M j m=1 g(B jm ,R j )(d R j ,m 1) i . (11) Secondly, we update R 1:N , given ↵, ⇢, 0 , 1 and B 1:N , from ⇡(R 1:N |↵, ⇢, 0 , 1 , B 1:N ) / ⇡(R 1:N |↵, ⇢)⇡(B 1:N | 0 , 1 , R 1:N ) / / e ↵ n P N j=1 d(R j ,⇢) N ln[Zn(↵)] 0 P N j=1 P M j m=1 g(B jm ,R j ) • e 1 n 2 P N j=1 P M j m=1 g(B jm ,R j )(d R j ,m 1) 2 4 N Y j=1 M j Y m=1 1 + e 0 1 d R j ,m 1 n 2 ! 3 5 1 .
(12) 3.3. Clustering non-transitive assessors. So far we assumed that a unique consensus ranking was shared by all assessors. Since in many situations this assumption is unrealistic, we allow for clustering the assessors into separate subsets, each sharing a consensus ranking of the items. We propose a mixture model generalization of the Bernoulli model of Section 3.1 to deal with heterogeneous assessors expressing pairwise preferences with mistakes.

Let z 1 , ..., z N 2 {1, ..., G} be the class labels indicating how individual assessors are assigned to one of the G clusters. Each cluster is described by a di↵erent pair of Mallows parameters (↵ g , ⇢ g ), g = 1, ..., G, so that the likelihood has the following form:

f (B 1:N |↵ 1:G , ⇢ 1:G , ✓, ⌘ 1:G , z 1:N ) = N Y j=1 8 < : X r2Pn f R j (r|↵ z j , ⇢ z j )f (B j |✓, R j = r) 9 = ; , where f R j (r|↵ z j , ⇢ z j ) = 1 Pn (r) Z n (↵ z j ) exp ⇢ ↵ z j n d(r , ⇢ z j ) .
We assume that the cluster labels are a priori conditionally independent given the mixing parameters of the clusters, ⌘ 1 , ..., ⌘ G , and distributed according to a categorical distribution

⇡(z 1 , ..., z N |⌘ 1 , ..., ⌘ G ) / N Y j=1 ⌘ z j = N Y j=1 G Y g=1 ⌘ 1g(z j ) g
, where ⌘ g 0, 8g = 1, ..., G and

P g ⌘ g = 1.
Finally we assign to ⌘ 1 , ..., ⌘ G the Dirichlet density with parameter . These choices lead to the following posterior density,

⇡(↵ 1:G , ⇢ 1:G ,⌘ 1:G , ✓, z 1:N |B 1:N ) / ⇡(✓) G Y g=1 [⇡(↵ g )⇡(⇢ g )⇡(⌘ g )] • N Y j=1 2 4 ⇡(z j |⌘ 1:G ) X r2Pn f R j (r|↵ z j , ⇢ z j )f (B j |✓, R j = r) 3 5 . ( 13
)
Similarly to the homogeneous case, we then sample from the posterior of eq. ( 13) by first updating ↵ 1:G , ⇢ 1:G , ⌘ 1:G , z 1:N and ✓, given B 1:N and R 1:N , and then updating R 1:N , given ↵ 1:G , ⇢ 1:G , ⌘ 1:G , z 1:N , ✓ and B 1:N . The former step is done by using the conditional density

⇡(↵ 1:G ,⇢ 1:G , ⌘ 1:G , z 1:N , ✓|B 1:N , R 1:N ) / G Y g=1 " ↵ 1 g e ↵g ⌘ ⇠ 1+ P N j=1 1g(z j ) g # • ✓ ✓ 1 ✓ ◆  1 1+ P N j=1 P M j m=1 g(B jm ,R j ) (1 ✓)  2 + 1 2+ P N j=1 M j • N Y j=1 2 4 e ↵z j n d(R j ,⇢z j ) Z n (↵ z j ) 3 5 . ( 14
)
The second step is performed by using the density

⇡(R 1:N |↵ 1:G , ⇢ 1:G , ⌘ 1:G , ✓, z 1:N , B 1:N ) / / N Y j=1 2 4 e ↵z j n d(R j ,⇢z j ) Z n (↵ z j ) ✓ ✓ 1 ✓ ◆ P M j m=1 g(B jm ,R j ) (1 ✓) M j 3 5 . (15)
Since label switching is not handled inside our MCMC, MCMC iterations are re-ordered after convergence has been achieved, by applying the algorithm of [START_REF] Stephens | Dealing with label switching in mixture models[END_REF].

4. MCMC for non-transitive pairwise preferences. We develop a Markov Chain Monte Carlo (MCMC) algorithm which, at convergence, samples from the posterior density of eq. ( 7). As explained in Section 3.1, the MCMC iterates between two main steps:

1. Update ↵, ⇢ and ✓ given B 1:N and R 1:N (using eq. ( 8)): (a) Metropolis update of ⇢ (b) Metropolis update of ↵ (c) Gibbs update of ✓ 2. Update R 1:N given ↵, ⇢, ✓ and B 1:N (using eq. ( 9)). In step 1(a), we propose a new consensus ranking ⇢ p according to a symmetric proposal which is centered around the current consensus ranking ⇢ t .

Definition 1. Swap proposal. At step t, denote the current version of the consensus ordering vector by x t = (⇢ t ) 1 , which is the vector whose n components are the items in O ordered from best to worst according to ⇢ t , i.e.,

x t i = O k () ⇢ t k = i. Let L ⇤ 2 {1, .
., n}. Sample uniformly an integer l from {1, 2, ..., L ⇤ } and draw a random number u uniformly in {1, 2, ..., n l}. The proposal x p has components (16)

x p i = 8 > > < > > : x t i if i 6 = {u, u + l} x t u+l if i = u x t u if i = u + l and the proposed ranking is ⇢ p = (x p ) 1 .
The parameter L ⇤ is the maximum allowed distance between the ranks of the swapped items, and is used for tuning the acceptance probability in the Metropolis-Hastings step. The transition probability of the Swap proposal is symmetric, and given by q(⇢ p ! ⇢ t ) = 1

L ⇤ P L ⇤ l=1 1 n l 1(|⇢ p ⇢ t | = 2l
). The ranking is then accepted with probability ✏ = min{1, a ⇢ }, where

ln(a ⇢ ) = ↵ n N X j=1 h d(R j , ⇢ p ) d(R j , ⇢ t ) i .
In step 1(b) we propose ↵ p from a log-normal density ln N (ln(↵ t ), 2 ↵ ), and accept it with probability ✏ = min{1, a ↵ }, where

ln(a ↵ ) = h ln(↵ p /↵ t ) i  + 1 n N X j=1 d(R j , ⇢) (↵ p ↵ t ) N  ln[Z n (↵ p )/Z n (↵ t )] .
This acceptance probability takes into account the asymmetric transition probability of the chain, that results from the log-normal proposal. The partition function Z n (↵) can be computed exactly or approximated by the importance sampling scheme proposed by [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF], depending on the distance function chosen and on the number n of items considered.

In step 1(c) we sample ✓ from the beta distribution, truncated to the interval [0, 0.5), with updated hyper-parameters,

 0 1 =  1 + N X j=1 M j X m=1 g(B jm , R j ),  0 2 =  2 + N X j=1 M j X m=1 [1 g(B jm , R j )].
Step 2 is a Metropolis-Hastings for the individual rankings. Here we exploit the fact that, when fixing all other parameters and the data B 1 , ..., B N , R 1 , ..., R N are conditionally independent, and that each R j only depends on the corresponding data B j . We thus sample a proposed individual ranking r p j from the Swap proposal, separately for each j = 1, ..., N. The Swap proposal is here advantageous because it perturbs locally not only the current individual ranking r t j , but also the function g(B jm , r t j ). Remark. The Swap proposal always gives a proposed individual ranking r p j 6 = r t j . However, it may happen that g(B jm , r p j ) = g(B jm , r t j ), 8m = 1, ..., M j . This is important for what concerns the acceptance probability of r p j . If g(B jm , r p j ) = g(B jm , r t j ), 8m = 1, ..., M j , the acceptance probability depends only on the ratio of the Mallows likelihoods of r p j and r t j , and is equal to

✏ = min{1, a 1 }, where ln(a 1 ) = ↵ n h d(r p j , ⇢) d(r t j , ⇢) i .
If g(B jm , r p j ) 6 = g(B jm , r t j ) for some m = 1, ..., M j , the acceptance probability depends also on the mistake model, and is equal to ✏ = min{1, a 2 } where ln(a 2 ) = ln(a 1 ) +

M j X m=1 h g(B jm , r p j ) g(B jm , r t j ) i ln [✓/(1 ✓)] .
Example. To illustrate this step of the algorithm, suppose that an assessor expresses the following set of preferences,

B j = {(O 2 O 1 ), (O 5 O 4 ), (O 5 O 3 ), (O 5 O 2 ), (O 5 O 1 ), (O 3 O 2 ), (O 1 O 3 )}.
This set contains the non-transitive pattern

O 2 O 1 O 3 O 2 .
For the illustration, suppose that the current value of the individual ranking vector is r t j = (5, 4, 3, 2, 1), which corresponds to the ordering vector

x t j = (O 5 , O 4 , O 3 , O 2 , O 1 )
, and for which

P 7 m=1 g(B jm , r p j ) = 1. If we sample the proposal x p j = (O 5 , O 3 , O 4 , O 2 , O 1
), this gives g(B jm , r p j ) = g(B jm , r t j ), 8m = 1, ..., 7, and r p j = (5, 4, 2, 3, 1) 6 = r t j . However, if we sample

x p j = (O 4 , O 5 , O 3 , O 2 , O 1
), then r p j = (5, 4, 3, 1, 2) 6 = r t and also Appropriate convergence of the MCMC must in practice be checked by inspecting the trace plots of the parameters, and by monitoring for example the integrated autocorrelation. In Supplement A we explain in detail how the algorithm is adapted to the case of the logistic mistake model, and to the mixture extension.

5.

Other approaches to pairwise preference data. In the classical Bradley-Terry model (BT) for pair comparisons [START_REF] Bradley | Rank analysis of incomplete block designs: I. The method of paired comparisons[END_REF] the probability that item O i is preferred to item O k is expressed as the ratio ( 17)

Pr(O i O k |µ) = µ i µ i + µ k ,
where µ = (µ 1 , . . . , µ n ), is a vector of item-specific consensus ratings shared by all assessors, forming a linear scale of score parameters. From this follows that the odds for

(O i O k ) against (O k O i ) are
given by µ i /µ k . In addition, it is assumed that all pairwise comparisons are conditionally independent given µ. Therefore, the likelihood expression of the BT model corresponding to data consisting of several pair comparisons is the product, across all considered pairs, of terms of the form (17). For this reason, all pairwise data, even when they may have come from a number of individual assessors, are e↵ectively merged when performing inference on µ.

The work of Bradley and Terry was preceded by two important earlier papers, by [START_REF] Thurstone | A law of comparative judgment[END_REF] and [START_REF] Zermelo | Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrscheinlichkeitsrechnung[END_REF]. Thurstone considered a similar preference data context as BT, but the work was based on a Gaussian error model. [START_REF] Zermelo | Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrscheinlichkeitsrechnung[END_REF], in contrast, proposed exactly the same model as Bradley and Terry, but it was presented as a statistical model for the results from a chess tournament, without the presence of individual assessors. After these pioneering works, several extensions of the basic BT model have been presented, mostly in the econometric and psychometric literature. Often these papers apply the logarithmic transformation u i = log µ i of the parameters, with the e↵ect that the probabilities (17) get the familiar logistic form. The logit of the odds for (O i O k ) against (O k O i ) is then equal to the contrast u i u k between the corresponding logarithmic scores. Extensions to regression models that account for the influence of item specific covariates on the comparison results are then readily available; for more comments on this, see below.

Data generated from the BT model are often not transitive, and this is the case particularly when some contrasts u i u k are close to 0. In situations in which the actual data come from a number of individual assessors, as was the case in our musicology experiment, it is a natural idea to try to account in the modeling separately for the two sources that may have created nontransitivity in the combined data: One the one hand, the di↵erences in the assessment profiles of the assessors, and on the other, possible lack of transitivity in the pairwise comparisons coming from each individual assessor. This distinction was made fully explicit in the structure of our BM and BL models of Sections 3.1 and 3.2.

As an alternative to our approach, an anonymous referee suggested a hierarchical two-layer structure based on the BT model. In that suggestion, data coming from an individual assessor would be described by a BT model, but with score parameters µ j = (µ j1 , ..., µ jn ) specific to each assessor j. On the lower level of model hierarchy, the referee suggested that, for each item i, the score parameters µ ji for di↵erent assessors j would be sampled independently from a Gaussian distribution centered at a common value µ i .

We developed such a model, which we call HBT (with H for hierarchical), work in progress [START_REF] Crispino | The Hierarchical Bradley-Terry model[END_REF]. There, we discuss (i) the suitability of the HBT model for data in the form of repeated pairwise comparisons performed by each assessor, and (ii) the poor performance of HBT compared to our Bayesian Mallows approach when data are such that each assessor only performs a limited number of comparisons without repetitions, so that not all pairs of items are compared by every assessor. Such a small incomplete example, with no intransitivities, is considered in [START_REF] Liu | Model-based learning from preference data[END_REF], where further di↵erences between the HBT and the Bayesian Mallows model are discussed. One important reason for the di↵erence in the case of incomplete and sparse data is that often they do not satisfy the strong connection condition [START_REF] Ford | Solution of a ranking problem from binary comparisons[END_REF]. This condition is fulfilled if, for any partition of all items into two non-empty sets, both subsets contain at least one item that was preferred to some item in the other set by at least one assessor, see [START_REF] Yan | Ranking in the generalized Bradley-Terry models when the strong connection condition fails[END_REF]. If this condition is not satisfied, the maximum likelihood estimator does not exist and the posterior inferences based on the HBT model will be highly sensitive to the specification of the prior and will require corresponding sensitivity analyses.

The BT model was represented and fitted as a log-linear model (Dittrich, Hatzinger andKatzenbeisser, 1998, 2002). In these works, the authors introduced assessor specific covariates into their framework, and extended it to the case of dependent pair comparisons. Building on Dittrich, Hatzinger andKatzenbeisser (1998), Francis, Dittrich and[START_REF] Francis | Modeling heterogeneity in ranked responses by nonparametric maximum likelihood: how do Europeans get their scientific knowledge?[END_REF] further introduced random e↵ects for each assessor in order to account for residual heterogeneity that is not included in individual-specific covariates. However, their method is applied to pair preferences derived from full rankings. As such, the pair preferences are complete, that is, n(n 1)/2 pairs are assessed by each assessor, and transitive. Their method cannot be used on our data where each assessor provides a limited number of pairwise preferences, typically smaller than the maximum n(n 1)/2, and is allowed to contradict herself, thus leading to non-transitive patterns in the data.

An interesting literature that builds on the Thurstone's model is the psychometric one [START_REF] Bockenholt | A logistic representation of multivariate paired-comparison models[END_REF][START_REF] Böckenholt | Hierarchical modeling of paired comparison data[END_REF][START_REF] Böckenholt | Individual di↵erences in paired comparison data[END_REF][START_REF] Böckenholt | Thurstonian-based analyses: Past, present, and future utilities[END_REF]. In these works, the authors develop di↵erent generalizations of the Thurstone model, accounting for instance for multidimensional parameters, in case the items are evaluated with respect to multiple aspects, or introducing dependency among the observed pairs, by the inclusion of random e↵ects in the model. However inference is performed when the data include repeated comparisons for each assessor, and all items are compared by each assessor.

Pair comparison data were also recently handled within the Mallows ranking models by [START_REF] Lu | E↵ective Sampling and Learning for Mallows Models with Pairwise-Preference Data[END_REF] and [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF]. However, both papers deal only with transitive pairs, explicitly ruling out the nontransitive patterns in the data. [START_REF] Volkovs | New Learning Methods for Supervised and Unsupervised Preference Aggregation[END_REF] propose a score-based method, called Multinomial Preference model (MPM), that generalizes the Plackett Luce model [START_REF] Luce | Individual choice behavior: A theoretical analysis[END_REF][START_REF] Plackett | The Analysis of Permutations[END_REF]. The main di↵erence between their MPM and our model is in the data generating mechanisms, which [START_REF] Volkovs | New Learning Methods for Supervised and Unsupervised Preference Aggregation[END_REF] assumed to be a multinomial score based process, while our method builds on considering distances between ranking vectors. In addition, their goal is to learn a single consensus ranking of the items, or multiple consensus rankings in case of clustering. Our method instead has the ability to further learn the individual latent rankings for each assessor. [START_REF] Ding | Learning mixed membership Mallows models from pairwise comparisons[END_REF] proposed a model for noisy pairwise ranking data, based on a mixed membership of Mallows models (M4), which generalizes the mixture model of [START_REF] Lu | E↵ective Sampling and Learning for Mallows Models with Pairwise-Preference Data[END_REF]. Their proposal is near to ours, in that both postulate the existence of latent linear orderings. However, [START_REF] Ding | Learning mixed membership Mallows models from pairwise comparisons[END_REF] assume a basic separability property, which would be di cult to justify in contexts similar to our data application. Furthermore, they model the presence of non-transitive patterns in the data as arising because each assessor has multiple latent linear orderings, while we propose a mistake model. Moreover, they consider only the Kendall distance, while our model handles every right-invariant distance.

There is a large body of literature on mixture models for ranking data (e.g [START_REF] Murphy | Mixtures of distance-based models for ranking data[END_REF][START_REF] Gormley | Analysis of Irish third-level college applications data[END_REF][START_REF] Caron | Bayesian nonparametric Plackett-Luce models for the analysis of preferences for college degree programmes[END_REF][START_REF] Meilǎ | Dirichlet Process Mixtures of Generalized Mallows Models[END_REF][START_REF] Jacques | Model-based clustering for multivariate partial ranking data[END_REF]. Although related to our mixture model extension, all these papers are based on data in the form of rankings, and they do not directly apply, or extend, to non-transitive pairwise comparison data. Apart from this di↵erence, the work of [START_REF] Jacques | Model-based clustering for multivariate partial ranking data[END_REF], which presents a mixture extension of the model developed in [START_REF] Biernacki | A generative model for rank data based on insertion sort algorithm[END_REF], has some similarities with ours. These authors assume the existence of a consensus ranking, and of individual rankings, and they model stochastic errors between these permutations of the items, to explain the variability of the individual rankings around the consensus. In this way, the pairwise comparisons are always complete and transitive, in contrast to our setting.

6. Simulation study. The aim of the experiments was to validate the method and to evaluate its performance in some test situations. The data were simulated from the Mallows model with the Bernoulli mistake model, varying parameters ✓, ↵, n, N , and M j , j = 1, ..., N, while always using the footrule distance. The number of items n was always kept below 50, thus enabling us to use the exact partition function [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF]. For a detailed description of the data generation, see Supplement B.

Various point estimates can be deduced from the posterior distribution of ⇢, one being the maximum a posteriori (MAP). We prefer the following sequential construction, called the cumulative probability (CP) consensus ordering in [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF]: first we select the item which has the largest marginal posterior probability of being ranked 1 st ; then, excluding this first choice, we select the item which has the largest marginal posterior probability of being ranked 1 st or 2 nd among the remaining ones, and so on.

In order to assess the performance of our methods, in Figure 1 we plot the posterior distribution of the normalized footrule distance between the estimated consensus ⇢ and the true consensus,

d f (⇢, ⇢ true ) = 1 n P n i=1 |⇢ i ⇢ true i
|, for varying parameters ↵, ✓, M (the average number of pairs given to each assessor) and N , while keeping fixed n = 10.

As expected, the performance of the method improves as the number of assessors N increases (Figure 1a), as the probability of making mistakes ✓ decreases (Figure 1b), as the dispersion of the individual latent rankings R true j around ⇢ true decreases, that is when ↵ increases (Figure 1c), and when the average number of pairwise comparisons becomes larger (Figure 1d). Interestingly, in the last case, the method performs generally well also when the average number of pairs is M = 15, being only 1/3 of the maximal number of pairs possible.

In Figure 2 we plot the posterior distribution of d f (⇢, ⇢ true ) corresponding to simulation experiments with n 2 {15, 25}, when increasing the number of assessors N . Note that the number of pairs assessed by each assessor in the case n = 25 is around 50, which is 1/6 of all the possible pairs.

Next, we studied the performance of the method in terms of the precision of the individual ranking estimation. We quantified the results by the probability of getting at least 3 items right, among the top-5, defined as follows. For each assessor j = 1, ..., N, we found the triplet of items top 3 items, i.e. the triplet that maximized

D j 3 = {O i 1 , O i 2 , O i 3 } that had
P 2P 3 P({R ji 1 , R ji 2 , R ji 3 } = | data)
, where denotes a permutation of the set {1, 2, 3}. This posterior quantity was estimated along the MCMC trajectory. We defined H j 5 to be the set of 5 highest ranked items in R true j , for each assessor j. We then checked whether D j 3 ⇢ H j 5 (that is, if the top-3 estimated items were all among the top-5 of each assessor). The percentages of assessors for which this is true are reported in Table 2. We notice that the results are overall very good: in the cases where n is set to 10 (first 4 sub-tables from the left in Table 2), we consistently learn 3 out of the top 5 items in more than 70% of the assessors (with a peak of 100%). Also in the more di cult cases of n = 15 and n = 25 (first 2 sub-tables from the right in Table 2) the results are very good, especially considering that this percentage does not include the cases where only 2 (or 1) items where correctly estimated in the top positions.

Table 2 Results of the simulated data. Percentage of assessors for which the estimated top-3 items belong to the true top-5. Data correspond to simulations with the same parameter settings as the results shown in Figures 1 and2: from left to right, same parameters as in Figure 1a, Figure 1b, Figure 1c, Figure 1d, Figure 2a and Figure 2b. We then chose randomly one of the simulated data cases and computed the posterior probabilities of correctly predicting the preference order of all pairs not assessed by the assessors, i.e. P h g(B j,new , R j ) = g(B j,new , R true j ) data i .

N % ✓ % ↵ % M % N % N %
Figure 3 shows the boxplots for these predictive probabilities, (left) stratified according to the number of pairs each assessor assessed in the data, and ).

In the case considered, the model had a very good predictive power, especially considering that the simulated data had many mistakes (around 10%). We also notice a slight increase of the predictive probabilities as M j increases (left panel) and as d(⇢ true , R true j ) decreases (right panel). These results are not surprising: it is easier to predict correct orderings of new pairs when (i) the assessor assesses more pairs, and (ii) the assessor's own ranking resembles more the shared consensus.

In Supplement C we report an analysis of data generated by the logistic model The results were very similar to those obtained above. In fact, the posterior distribution of 1 was highly concentrated around 0, which is when LM collapses to BM.

7. Human causation in sounds. We analyzed the data using the mixture model explained in Section 3.3 with footrule distance. With n = 12 sounds we can use the exact expression of the partition function [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF]. In the Dirichlet prior for ⌘, we set = 20, which favors highentropy distributions, thus reflecting our inability to express precise prior knowledge. In the Beta prior for ✓, we set the hyperparameters at  1 =  2 = 1, that is, the uniform distribution on the interval [0, 0.5), and the hyperparameters of the prior for ↵ at = 1 and = 1/10, as discussed in [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF]. We run the MCMC sampler for 10 6 iterations, after a burn-in of 2 • 10 5 . Separate analyses were performed for G 2 {1, . . . , 7}.

In order to choose an appropriate number of clusters, we plot in Figure 4 two quantities: on the left, the within-cluster sum of footrule distances between the individual rankings and the consensus ranking of that cluster, P G g=1 P j:z j =g d f (R j , ⇢ g ); on the right, the within-cluster indicator of mis-fit to the data, P G g=1 P j:z j =g P M j m=1 g(B jm , ⇢ g ). Both these measures are defined in [START_REF] Vitelli | Probabilistic preference learning with the Mallows rank model[END_REF], and tested as good measures to select G.

More traditional information criteria, such as the deviance information criterion [START_REF] Spiegelhalter | Bayesian measures of model complexity and fit[END_REF], were considered, however their performance was quite unstable, possibly attributable to the sparsity of the data. Inference on the number of clusters could have been alternatively performed via a reversible jump MCMC. There appears to be an elbow in the figures at G = 3, to guide us in the choice of the number of clusters. We decided on G = 3, also motivated by the relatively small sample size of the experiment (N = 46).

Table 3 shows the results for G = 3: the maximum a posteriori (MAP) estimates for ⌘ and ↵, together with their 95% highest posterior density (HPD) intervals, are shown at the top of the table. The table also shows the estimated cluster-specific consensus lists of sounds, estimated by the CP procedure. We observe the di↵erences in the three consensus lists. S1, the stimulus with the most dynamic spatial motion, is on top in cluster 3, but at the bottom in cluster 1; S8, the test stimulus that has maximum spatial details but no volume nor pitch change, is on top in cluster 1, but second to the last in clusters 2 and 3. Finally, S5, the stimulus that contains the least movement variation but has pitch and volume suppressed, is ranked third and first in clusters 1 and 2, but towards the bottom of the list in cluster 3. Listeners in cluster 1 found variation in volume or pitch as a negative or distracting feature. They rated S8 at the top, a test stimulus that has maximum spatial details but no volume nor pitch change. Also, S10, S5 and S9, which were ranked next, lack volume and pitch details. The bottom 4 stimuli contain maximum pitch and volume variation. Among them was S3 (mono sound, no space at all), forming a strong contrast to the top ranked S8 (maximum spatial movement). Evidently, space was important for these listeners, while pitch and volume variation was a negative or distracting feature.

In cluster 2 listeners did not like fast movements as a sign of human feature, but they did like correlated pitch and volume (the top 4 sounds feature a low amount of spatial variation, but also correlated pitch and volume, while the bottom 3 sounds are the same as the top 3 but lack correlated pitch and volume variation). Listeners in this cluster prioritized pitch and volume variations above spatial variation, and preferred low spatial variation (slower, or more relaxed movements).

Cluster 3 consists of subjects who, in their evaluation of the test stimuli, appear to include all spatial cues that adhere to our everyday perception of spatial motion. The stimuli with most dynamic spatial motion, enhanced by spatially correlated pitch and volume variations, are in the top-3, while stimuli with the least of these features are in the bottom-3. These listeners prioritize high levels of spatial detail above all other features, and their perception of these details are enhanced by correlated pitch and volume variations. This is indicated in (i) S1 being at the top; (ii) S7, which is the same as S1 but lacks pitch variation, being second; (iii) S11, which is the same as S1 but played 30% slower, being third (i.e. space, volume and pitch variations are just a bit slower); (iv) S8, S9, S10 are in the bottom, and all lack pitch, volume variation, and spatial movement details.

We investigate the stability of the clustering in Figure 5, that shows the heatplot of the posterior probabilities, for all the listeners (shown on the x-axis), for being assigned to each of the clusters identified in Table 3. Most of the probabilities are concentrated on some particular value of c among the three possibilities, indicating a reasonably precise behavior in the cluster assignments. We then computed, fixing these cluster assignments, the marginal posterior probability that each sound is among the top-4 in ⇢ 1:G and in R j , j = 1, ..., 46, respectively. The results are shown in Figure 6. Each heatplot refers to a cluster (G1 (left), G2 (center) and G3 (right)) and represents the marginal posterior probabilities for each sound (y-axis) being ranked among the top-4 in the consensus of that cluster (first column), and in the individual rankings of listeners in that cluster (remaining columns, assessors on the x-axis). As Figure 6 shows, there is considerable variation in the estimated rankings of the sounds between individual listeners even when they are included in the same cluster. For example, looking at Figure 6 left, we see that S8, S10, and S5 have high (> 0.8) posterior probability of being ranked among the top-4 stimuli in the consensus ranking (column 1). However, looking at the estimates for the listeners in cluster 1, we see that the variation is very high: For example, listener 30 (column with label 30) has a very high posterior probability of ranking S3 and S6 among the top-4 stimuli. This aspect is important for what concerns individual estimates.

Here we consider the relationship between the probability of placing some given stimuli in the top (bottom) ranks and the musical sophistication index (MSI), or the spatial audio awareness index (SAA). Figure 7 shows the relationship between listeners' SAA and the probability of sounds S1 and S7 being ranked in the top-4 (both marginally and jointly). Recall that S1 was the original sound, while S7 was identical to S1, but without pitch variation. The plot suggests that spatial listening is a skill that is enhanced through training.

Figure 8 shows the relationship between listeners' MSI and the probability of sounds S8 and S10 being ranked among the bottom-4 (both marginally and jointly). Respondents with a score greater than 500 were classified as musically more sophisticated, and those with a score less than 500 as less sophisticated, as suggested in http://marcs-survey.uws.edu.au/OMSI/omsi.php. Both S8 and S10 suppress pitch and volume variations, which are expected to enhance the implication of human causation. These two stimuli are more likely to be ranked in the last 4 positions by listeners with high MSI. Interestingly, this suggests that musically sophisticated listeners find pitch and volume variations to be qualities for a stimulus to sound human.

8. Conclusions and discussion. The main contribution of this paper is to introduce a new Bayesian method for non-transitive pairwise preference data. The principal advantage of the Bayesian approach comes from its ability to combine di↵erent types of uncertainty in the reported data, coming from di↵erent sources, and from being able to convert such data into the form of meaningful probabilistic inferences. Our method provides the posterior distribution of the consensus ranking, based on pairwise assessment data from a pool of assessors who may have individually violated logical transitivity in their reporting. The method is also able to produce the posterior distributions of the latent individual rankings of the assessors. Such rankings can be used in the construction of personalized recommendations, or in studying how individual preferences change with assessor related covariates. We also developed a mixture model generalization of the main model, able to handle heterogeneity in pairwise and non-transitive preference data. The model was then used to investigate how individual listeners perceive human spatial causation in acousmatic sounds. The data came from a di cult experiment, that involved human perceptions. For this reason, pair comparison of sounds was the only feasible design. The data were noisy, and in particular often logically non-transitive at the individual level. We used our approach to estimate individual rankings, and sub-groups of assessors.

The results revealed how di↵erently people listen to and interpret abstract sounds. We related individual musicological scores to individual rankings, leading to an interesting correspondence between spatial sound feelings and sound expertise.

Sometimes pairwise comparison data contain draws, or ties. A tie occurs when a pairwise comparison between two items does not result in a defined preference of an item towards the other. This situation has been much considered in the literature on pairwise comparisons (e.g. [START_REF] Rao | Ties in paired-comparison experiments: A generalization of the Bradley-Terry model[END_REF][START_REF] Davidson | On extending the Bradley-Terry model to accommodate ties in paired comparison experiments[END_REF]. Our method does not model probabilistically the presence of ties, but it is possible to handle them directly in the MCMC procedure: apply the proposed model, and simply break each tie by tossing a symmetric coin inside the MCMC.

Another extension of the model would be to allow for the possibility of including covariates of subjects and/or items in the analysis. For instance, the probability of making a mistake could depend on some characteristics of the items, so that, the more similar two items are in terms of such characteristics, the more likely it is to make a mistake in reporting the pairwise preference. In our application relevant covariates could be the variation in pitch and volume, or the overall motion speed that characterizes each sound.

The time complexity of our algorithm is linear in terms of the number of assessors N . The increase of the number of items n does not a↵ect computing time of a single MCMC step. However, the larger n is, the longer the chain must be in order to reach convergence.

  Fig 1: Results of the simulated data. Posterior CDFs of d f (⇢, ⇢ true ) as a function of N for ↵ = 3, M = 25, ✓ = 0.1 (a); as a function of ✓ for ↵ = 3, N = 40, M = 25 (b); as a function of ↵ for ✓ = 0.1, N = 40, M = 25 (c); as a function of M for ↵ = 3, N = 40, ✓ = 0.1 (d).

Fig 2 :

 2 Fig 2: Results of the simulated data. Posterior CDFs of d f (⇢, ⇢ true ) as a function of N , for ✓ = 0.1, ↵ = 3.5, M = 25, n = 15 (a), and for ✓ = 0.1, ↵ = 4.5, M = 50, n = 25 (b).

(

  right) stratified according to the footrule distance between the individual ranking R true j and the consensus ⇢ true , d(⇢ true , R true j

Fig 3 :

 3 Fig 3: Results of the simulated data. Posterior probabilities of correctly predicting the preference order of all pairs not assessed by the assessors, (left) stratified according to the number of pairs each assessor assessed in the data, and (right) stratified according to d(⇢ true , R true j

Fig 4 :

 4 Fig 4: Acousmatic data. Boxplots of the within-cluster sum of footrule distances between the individual rankings and the consensus ranking of that cluster (left), and of the within-cluster indicator of mis-fit to the data (right), for di↵erent choices of G.

Fig 5 :

 5 Fig 5: Acousmatic data. Heatplot, for all the listeners (on the x-axis), of the posterior probabilities of being assigned to each of the three clusters (on the y-axis).

Fig 6 :

 6 Fig 6: Acousmatic data. Heatplot of the marginal posterior probabilities for all the stimuli (y-axis) of being ranked among the top-4 for cluster 1 (left), 2 (center) and 3 (right).

Fig 7 :

 7 Fig 7: Acousmatic data. Boxplot of the posterior probabilities for sounds S1 (left), S7 (middle), S1 and S7 jointly (right), of being ranked among the top-4 in the individual ranking R j , stratified by the SAA index. The horizontal dotted line is the threshold in the case of random assignment. Scale of SAA: from 0 to 3, the largest, the more aware of spatial dimension of sounds.

Fig 8 :

 8 Fig 8: Acousmatic data. Boxplot of the posterior probabilities for sounds S8 (left), S10 (middle), S8 and S10 jointly (right), of being ranked among the bottom-4 in the individual ranking R j , stratified by the MSI index. The horizontal dotted line is the threshold in the case of random assignment.

Table 1

 1 Summary of the test sounds.

Table 3

 3 Acousmatic data. Sounds are ordered according to the CP consensus ordering, obtained from the posterior distribution of ⇢g, g = 1, 2, 3.

		Rank	1	2	3	4	5	6	7	8	9	10	11	12
		↵ 1 = 2.66
	G1	(1.14,4.96)	S8	S10	S5	S9	S6	S4	S7	S11	S12	S2	S3	S1
		⌘ 1 =0.31
		(0.21,0.41)
		↵ 2 = 5.16
	G2	(3.15,9.29)	S5	S4	S12	S2	S11	S3	S6	S1	S7	S9	S8	S10
		⌘ 2 =0.33
		(0.22,0.43)
		↵ 3 =5.32
	G3	(3.61,7.66)	S1	S7	S11	S2	S4	S12	S6	S3	S5	S9	S8	S10
		⌘ 3 =0.37
		(0.27,0.48)

Acknowledgments. MC thanks Sonia Petrone and Isadora Antoniano-Villalobos for useful discussions. MC was partially funded by Cariplo during the project. The authors thank the Editor and the anonymous reviewers for their useful comments and suggestions which helped improving the paper. We thank Øystein Sørensen for his contributions to Bayesian Mallows modeling.

SUPPLEMENTARY MATERIAL

Supplement A: Adaptations of the algorithm (doi: 10.1214/00-AOASXXXXSUPP).

Supplement B: Some remarks on the simulated data (doi: 10.1214/00-AOASXXXXSUPP).

Supplement C: Simulations with the Logistic DGP (doi: 10.1214/00-AOASXXXXSUPP).

Supplement D: MCMC diagnostics (doi: 10.1214/00-AOASXXXXSUPP).


Jeanne Schreurs
email: jeanne.schreurs@uhasselt.be

ICT use in school: vision and performance measures

Keywords: ICT vision, performance measure, EFQM excellence model, PI

The implementation of ICT in schools requires a vision on ICT use in school, the formulation of clear strategic goals, and the planning and organisation of the use of ICT in school. The pursued goals are those points we want to reach for the learner by setting up ICT use in school. At the same time the pursued goals are the results of using ICT as expected by the stakeholders, being the learner, the teacher, the ICT coordinator, and on an indirect way the parents, the environment and the funding government. We created a checklist around the following three main criteria ICT vision, enabling school organisation and processes and results. Those has been split again in five criteria, the vision for ICT use in school, secundary processes, resources and partners, primary processes and results. The checklist can be used by the school board as a self evaluation instrument.

Introduction

School management must have a vision on ICT use, taking into account the new developments in ICT, and must formulate a strategy to achieve the ICT vision. It appeared to be that in Flanders the government has a clear vision and strongly supports ICT use in schools. In the annual report from the Educational Inspectorate we can read that the biggest challenge for the schools is not the acquisition of HW, but the integration of ICT in the learning process. It stands for the third phase of the evolution of ICT use in schools. The school strives to excellent performance. The main focus is here on the learner. How can he benefit from the use of ICT in the learning activities. Schools can use performance models and a selfassessment instrument based on one of them. A school needs a specific school performance measurement model /framework. It can be based on the EFQM excellence model. The management team of the school together with the teachers, the ICT co-ordinator and some learners will evaluate the ICT use and decisons about improvements will be taken based on it.

Vision on the role of ICT in school

The Government has a vision and strongly supports ICT use in schools

The Government is strongly supporting developments in ICT in schools through the provision of resources and guidance. It is, however, up to schools to develop a vision for ICT that is right for them and that will meet their projected needs and aspirations. The Flanders government has launched the infrastructure projects PC/PK project, the I-line of Belgacom and the AAL project. Some other initiatives are REN (regional networks of expertise) project organising training of the teachers in using ICT in education, and the funding of an additional staffmember as an 2 [START_REF] Madarsha En Agboola | A study of readiness towards introduction of e-learning as an environment for teaching and learning among lecturars : a case study[END_REF] ICT co-ordinator. The government is co-funding the participation of the schools in European projects. [START_REF] Craemer | ICT in onderwijs, de acties van de Vlaamse overheid[END_REF][START_REF] Mesdom | [END_REF]3,[START_REF] Vandenbroucke | Competenties voor kennismaatschappij: beleidsplan ICT in het onderwijs 2007-2009[END_REF][START_REF]Anytime anywhere learning[END_REF] Klascement and Anywize.net and digikids and klikvast are educational portalsites for teachers, of which the development and the maintenance is done by teachers. [START_REF] Vandenbroucke | Competenties voor kennismaatschappij: beleidsplan ICT in het onderwijs 2007-2009[END_REF] Those are independent, non-commercial and free platforms where teachers and students can interchange learning materials and links. Software packages can be downloaded too. In discussion fora students and teachers can interchange information, experiences and ideas. By the way the schools, teachers, parents and learners are motivated to collaborate and are stimulated to use ICT in a pedagogical way. How can ICT put into educational practice is the subject of a yearly organised conderence. [START_REF] Klascement | [END_REF][START_REF] Anywize | [END_REF][START_REF] Digikids | [END_REF] On digikids portal some competition is organised between schools. It is also an e-shop for the purchase of software at reduced prices for schools. [START_REF]Click safe[END_REF]3] Klikvast informs the teachers and the parents about how to protect the children against the dangers of the internet, chatting, cyber-pesting and how to organise a secure internet class in the school. The European Schoolnet is integrating all local ICT school networks in one European network. [START_REF] Russell | Online and virtual schooling in Europe[END_REF]

Schools are on the way to the third phase: ICT integration in the learning process

In the annual report "Onderwijsspiegel" from the Educational Inspectorate we can read that the biggest challenge for the schools is the integration of ICT in the learning process. It stands for the third phase of the evolution of ICT use in schools.

The schools have finished the phases one and two. In the first phase teachers have learnt the basic principles of using computers. In the second phase applications of ICT have been introduced and the Internet has been made available for learners and teachers. Teachers have started to use powerpoint presentations in the classroom.

In the third phase ICT is becoming somewhat part of the learning activities. F.e. the teacher can formulate a problem and the learners will use the Internet to find the learning materials and additional information about the topic. Learners have to report about the results of their teamwork and will present it to the other learners. Part of the subject matter of teaching can be deliverd as e-learning courses and the learner can achieve it individually. Some schools are already evolving to a new learning model where learner and learning team are central. However, it has become a challenge for all occupied with the organization of education and training to re-engineer the learning process and to implement advanced ICT in order to enhance learning through technology. Collaboration of a team of students is being viewed as co-operating on an assignment and/or joining discussions about relevant or current topics. On a second level the learners will deliver additional content him/herself and share it with other learners in their team. Synchronous communication could be conducted via the chat facility and conference systems, and asynchronous communication may be facilitated through the discussion groups and via email The school management can play an important role in motivating and steering their teachers to the third phase. The availabbility of the ICT infrastructure is a condition but it does not automatically result in advanced use of ICT by the teachers in the learning activities. The school needs a management with vision that defines an effective ICT integration plan. [START_REF] Whitaker En | Developing an effective IT integration and support system[END_REF]

A school's vision for ICT in school

Shaping the vision means finding an answer on the question: "What kind of school do we want to be? And how does ICT fit into our vision?"

ICT supports both 'formal' and 'informal' learning, so that all can share not only in the learning experience but also increasingly personalised learning.

Realising the vision means the integration of the vision into the School Improvement/ Development Plan and to consider how the needed investment will be secured. ICT is a major investment for the school and requires long-term planning and regular monitoring and review. A school's vision for ICT should be driven by the potential to improve teaching, learning and achievement rather than by the technology itself. The question to ask is not 'What resources do we have?' but 'How do we use our resources effectively to enhance the education of our pupils?

3 Strategic goals and performance excellence of a school

Strategic goals

The pursued objectives can be formulated as answers on the desired results of different stakeholders. The direct benefiting stakeholders from the school are the learner, the teacher, the ICT co-ordinator and the school directorate. The indirect ones are the parents, the society and the government. The main focus is here on the learner. How can he benefit from the use of ICT in the learning activities.

We define a number of goals as institution wide ones and a number of learning specific goals.

STAKEHOLDERS direct indirect GOALS learner teacher ICT-coordinator parents society Government

Institution wide goals 1.be prepared to enter in the information society X X X X 2.to create a rich and effective learning environment X X X 3.advanced and innovative education model X X X … Learning specific goals 1.learner friendly learning process X 2.motivate the learner X 3.improve the learning results X X X X Figure 1:strategic goals By focusing on the learning process of the learner, we still have some effects on the other stakeholders in an indirect way. Although in a more general model we could focus also direct on the other stakeholders and formulate other goals. F.e. it is important that the school takes care for the teachers being the staffmembers of the school. So being well educated in the use of ICT and by the way being prepared to integrate ICT in the learning activities is a strategic goal.

Performance excellence and performance management

Performance management should be a main concern of every school. Organizations are seeking new, integrated systems that enable rapid changes through early identification of opportunities and problems, tracking of progress against plans, flexible allocation of resources to achieve goals, and consistent operations. Performance Management pinpoints the need to integrate strategy and key indicators of performance into management processes, and to exploit technology to improve monitoring, management reporting, and decisionmaking. Performance refers to output results and outcomes obtained from processes, products, and services.

The fundamental concepts or characteristics of excellence are results orientation, customer focus, leadership and constancy of purpose, management by processes and facts, people development and involvement, continuous learning, innovation and improvement, partnership development and public responsibility. The characteristics of excellence have links to different evaluation areas and also to each other. Customer focus in vocational education and training f.e. requires identification of the needs of customers, such as students and the world of work, development of products and services based on these, and monitoring and analysis of customer results achieved. Results should be used as a basis to improve operations and set new objectives. Regardless of sector, size, structure or maturity, to be successful, organizations need to establish an appropriate management framework. A Performance Excellence Model will help organizations in measuring where they are on the path to excellence, and will help them understand the gaps and then stimulate the development of solutions.

Performance models

A Performance Excellence Model has to be based on the fundamental concepts or characteristics of excellence and will help organizations in measuring where they are on the path to excellence, and will help them understand the gaps and then stimulate solutions. A performance model is based on the premise that excellent results with respect to performance, customers, people and society are achieved through leadership driving policy and strategy, that is delivered through people, partnerships and resources, and processes.

In the famous "Plan, Do, Check, Act" or PDCA cycle of W.E.Deming, the basic model of the continous improvement strategy, CHECK is one of the critical steps. Most of the performance models have built in a check or control function, and that is often done via self assessment.

A number of models have been developed that can be applied in schools. Three welknown models are the balanced scorecard (BSC), the Baldrige model and the EFQM excellence model. [START_REF] Schreurs | The EFQM self assessment model in performance management[END_REF][START_REF] Kaplan | Alignment[END_REF][START_REF]The Baldrige model for performance excellence[END_REF]

Balanced scorecard (BSC)

Management is steering the organisation based on measurements of a limited set of performance indicators (PI). The indicators have been selected starting from the critical success factors (CSF) of the organisation. The critical aspects of the organisation will be controlled on this way. All CSF's and PI's are structured in a BSC composed of 4 domains.

Baldrige educational criteria for performance excellence framework

This model focuses on all aspects influencing the results of the organisation, being the market, the products, the services, Financial aspects, human resources and organisation characteristics. This model is suitable to do a self evaluation of the organisation by the management team. A questionnaire will be built covering the 6 process criteria and 1 result criteria of the model. Based on a total weighted evaluation score, improvement actions will be planned.

EFQM excellence and Total Quality Management (TQM) model

EFQM helps organisations to determine at what point they find themselves on their way towards excellence. Any organisation that looks to future and work towards excellence should necessarily identify their strengths and areas in which improvements can be made.

EFQM excellence model used as a self assessment instrument.

The EFQM Excellence Model is a practical model to help organizations in measuring where they are on the path to excellence, helping them understand the gaps and then stimulating solutions. It can be used as a framework for self-assesment that enables an organization to identify its strengths and areas for improvement and the extent to which its operations and results are in line with the characteristics of an excellent organization. It compares the individual results of all criteria with the overall resulting mean value. Sometimes it is possible to compare the results with preset norm values. We adopted the EFQM self assesment method to measure the performance. We adopted the questionnaire approach. For all criteria we formulated one or more statements and asked the learner to give his opinion upon the importance or relevance of the criteria and at the same time give a score for the performance of it. The overall performance score is the mean value of the individual performance scores measured for the individual criteria. Performance can be improved by focusing on those criteria with a value less than the overall mean value.

6(12)

Performance measures and measurement

In research the following factors were identified as being good predictors of performance attainment: strategy towards corporate and social responsibility, employee loyalty, staff responsiveness, customer satisfaction as well as market share, cash flow and costs. Predictors for good practice adoption include benchmarking, skill and job training, customer orientation, problem solving, and waste elimination. Performance refers to output results and their outcomes obtained from processes, products, and services Performance can be expressed in non-financial and financial terms The challenge for organisations today is how to match and align performance measures with business strategy, how to deploy the measures so that the results are used and acted upon.

Performance measurement is one of the cornerstones of business excellence. Business excellence models encourage the use of performance measures. All measures need to be chosen to support the attainment of specific performance or behaviour identified by the organisation's leaders as important or necessary to work towards the organisational goals. This being the case, there must be clearly defined goals/objectives and strategies chosen to reach them before measures can be chosen to support their attainment. Business excellence models are used to categorise the measures into the generic business drivers of an organisation.

Performance measurement is fundamental to organisational improvement. The importance of performance measurement has increased with the realisation that to be successful in the longterm requires meeting (and therefore measuring performance against) all stakeholders' needs.

A school performance measurement model /framework based on the EFQM excellence model

The school performance measurement model /framework

Performance measurement is fundamental to organisational improvement of a school. Continuous improvement is set forward. To be successful it is required to meet (and therefore to measure performance against) all stakeholders' needs including learners, teachers, ICT coordinators, parents, the society and government. [17] We adapted the EFQM performance model and customised it to the school situation. We modelled the set of performance indicators to be applied in measuring the performance. First we identified a set of institution wide and some learning specific goals, starting from the desired results by the stakeholders. Based on it we structured our school performance measurement model, composed of the main criteria ICT vision, enabling school organisation and management and the results. The ICT vision main criterium is split in the block "having a vision" and in a second block strategy to achieve the vision. We structured the enabling school organisation and processes main criterium in 3 main blocks, the secundary processes, the resources and partners and the primary processes.

The secondary processes have to support the realisation of the primary processes.

In the secondary processes domain we identified all processes linked with school organisation and management, and all processes linked with the tasks of the teachers and the ICT coordinators in the learning process. The Directorat must have a vision on future developments of ICT and the role of it in education. It is also very important to be open for innovations and new challenges. Teachers and ICT-co-ordinators have to be motivated in the use of ICT in learning and supplementary training for them must be organised.

In ICT infrastructure must be invested. It may not be limited to HW but learning systems have to be implemented too. Participation in projects must be a priority to realise partnership with other schools and to become fundings to realise innovative initiatives.

In the primary processes focus must be on curriculum development taking into account that the learners have to be prepared for functioning afterwards in future education or job. The integration of ICT in the learning process needs attention and last but not least an effective and attractive learning model has to be conceived. The desired results criteria can be structured into the criteria learner, teachers and ICT coordinators, parents, society and governments.

The desired results for the learner are the knowledge increase, using skills learned, using ICT, apprentice of communication skills, being motivated to learn Teachers and ICT co-ordinators want to see the effect of training and the satisfaction of the learner and their parents and the appreciation from the school management The parents want to see that their children will become prepared for functioning afterwards.

The society and the government want to see young people functioning well in their job and becoming stable adult people in the community.

ICT vision

The vision for ICT use in school

Enabling school organisation and processes

Secundary processes

Resources and partners

Primary processes results

Desired results

Having a vision taking into account the new developments in ICT

Identification of performance measures or the performance indicators

We identified a set of performance indicators. The blocks of the framework are the main criteria of the model. The cells within the blocks are the performance indicators or the criteria of our model. For all criteria we formulate statements/ questions.

The evaluation process and the decision to plan for actions to improve the performance based on management reports

In Performance Management (PM) focus is on the need to integrate strategy and key indicators of performance into management processes. Performance indicators have been identified and are playing an important role in management reporting about the enterprise wide performance. Once the performance indicators have been measured, and the mesurements have been analysed decisions can be made on point of needed improvements. Performance can be improved by increasing that mean score resulting from the increase of the individual scores. But it is important to focus on those criteria with a value less than the overal mean value.

Evaluation session

The management team is evaluating the enterprise wide performance. The EFQM model is helping the organisation do this by measuring where they are on the path to excellence, helping them understand the gaps and then stimulating solutions.

For the self-assessment we adopted the questionnaire approach. For all criteria we formulated one or more statements and asked the learner to give his opinion about the importance or relevance of the criteria and at the same time give a score for the quality of it. The overall performance score is the mean value of the individual quality scores measured for the individual criteria.

Reports from the evaluation session

A radar diagram has been created showing the total weighted mean evaluation on detailed statement level and the weighted mean evaluation of criteria level is shown in the same figure.

TABLE of critical statements

This table is delivering a summary of critical statements belonging to the criteria by presenting the evaluation and the relevance/ importance in one diagram. Those statements which are quoted as important and having received a bad evaluation are critical. (red part of the fifure)

The evaluation process and the decision to plan actions to improve the performance

The self-assessment process allows the organization to discern clearly its strengths and areas in which improvements can be made and culminates in planned improvement actions that are then monitored for progress. Is ICT already been implemented and what about the results of it?

We start with the RADAR diagram. We compare the scores of the individual criteria with the overall mean score. We can identify the criteria with lower scores. We go to the Figure 6 to identify the critical statements corresponding to the low scored criteria we identified in the radar. We go to the table and see which statements for those criteria with low score are very critical.

At the end a list of criteria to be improved has to be created in descending order of priority to be improved. The information about being critical of the statements has to be used to define the needed actions.

Conclusion

The biggest challenge for the schools is not the acquisition of HW, but the integration of ICT in the learning process. It stands for the third phase of the evolution of ICT use in schools. A number of strategic goals have been identified some as institution wide ones and some as learning specific ones. They must be seen as answers on the desired results of different stakeholders. The direct benefiting stakeholders from the school are the learner, the teacher, the ICT co-ordinator and the school directorate. The indirect ones are the parents, the society and the government. The main focus is here on the learner. How can he benefit from the use of ICT in the learning activities.

The EFQM model can be used as a framework for self-assesment that enables an organization to identify its strengths and areas for improvement. It is customised here to the school situation. A set of performance indicators has been modelled and can be applied in measuring the performance. A set of performance indicators has been identified too. Once the performance indicators have been measured in the school, and the measurements have been analyzed, decisions can be made on point of the needed improvements. An answer can be given on the question if ICT is already implemented and what is the performance of it?

Figure 2 :

 2 Figure 2: EFQM excellence model

Figure 3 :

 3 Figure 3: The school performance measurement model /framework

Figure 4 :

 4 Figure 4: The school performance measures

Figure 5 :

 5 Figure 5: weighted mean evaluation

Figure 6 :

 6 Figure 6: critical statements