Chris Mccool
email: csmccool79@gmail.com

Sébastien Marcel
email: sebastien.marcel@idiap.ch

Abdenour Hadid

Matti Pietikäinen

Pavel Matějka

Jan Černocký

Norman Poh

Josef Kittler

Anthony Larcher

Christophe Lévy

Driss Matrouf

Jean-Franc ¸ois Bonastre

Phil Tresadern

Timothy Cootes

Bi-Modal Person Recognition on a Mobile Phone: using mobile phone data

Keywords: bi-modal authentication, mobile biometrics, speaker recognition, face recognition

This paper presents a novel fully automatic bi-modal, face and speaker, recognition system which runs in real-time on a mobile phone. The implemented system runs in real-time on a Nokia N900 and demonstrates the feasibility of performing both automatic face and speaker recognition on a mobile phone. We evaluate this recognition system on a novel publicly-available mobile phone database and provide a well defined evaluation protocol. This database was captured almost exclusively using mobile phones and aims to improve research into deploying biometric techniques to mobile devices. We show, on this mobile phone database, that face and speaker recognition can be performed in a mobile environment and using score fusion can improve the performance by more than 25% in terms of error rates.

I. INTRODUCTION

Mobile phones are now a part of many peoples daily lives. They help us to communicate with people by making phone calls and also by sending emails. Given the ubiquitous nature of these devices it is obvious that a real-world test of a pattern recognition algorithm is whether or not it can run on a mobile phone and in a mobile phone environment.

Only recently have researchers considered putting complex pattern recognition algorithms, such as speaker recognition and face recognition, on a mobile phone and this has led to a fractured approach to this problem. In [START_REF] Shen | Secure mobile services by face and speech based personal authentication[END_REF], a bi-modal, face and speaker, authentication system was evaluated on Phase I of the MOBIO database [START_REF] Marcel | On the results of the first mobile biometry (mobio) face and speaker verification evaluation[END_REF], but this system never ran on a mobile device. By contrast, in [START_REF] Kim | Person authentication using face, teeth and voice modalities for mobile device security[END_REF] a face, speaker and teeth authentication system was developed to run on a HP iPAQ rw6100 (a portable digital assistant), however, the authors evaluated this system on an in-house database consisting of only 50 people. Other examples include performing only face recognition on a mobile device [START_REF] Qian | Biometric authentication system on mobile personal devices[END_REF] and evaluating on a small in-house database of 20 people or performing only speaker authentication and evaluating on a small in-house mobile phone database of 50 people [START_REF] Rao | Robust speaker recognition on mobile devices[END_REF]. This fractured approach to applying biometric recognition to mobile devices has occurred primarily because of the lack of a large publicly available database and associated protocols.

This paper presents a novel mobile phone database to help evaluate mobile-based face, speaker, and bi-modal authentication algorithms as well as an initial, and novel, bi-modal recognition system. By doing this we hope to stimulate research in the field of multi-modal recognition in a mobile environment. We present our novel publiclyavailable database (the full MOBIO database) in Section II and the associated protocols are presented in Section III. We then outline our novel bi-modal recognition system, which runs in real-time on a Nokia N900, in Section IV. We then evaluate this bi-modal recognition system on the full MOBIO database in Section V to provide baseline results.

We then conclude and outline directions of future work in Section VI.

II. OVERVIEW OF DATABASE In this section we present the full MOBIO database. This database is unique because it is a bi-modal database that was captured almost exclusively on mobile phones. It consists of over 61 hours of audio-visual data with 12 distinct sessions usually separated by by several weeks. In total there are 192 unique audio-video samples for each of the 150 participants, this is almost twice the size of Phase I of the MOBIO database. This data was captured at 6 different sites over one and a half years with people speaking English 1 .

Capturing the data on mobile phones makes this database unique because the acquisition device is given to the user, rather than being in a fixed position. This means that the microphone and video camera are no longer fixed and are now being used in an interactive and uncontrolled manner. This presents several challenges, including:

• high variability of pose and illumination conditions, even during recordings,

• high variability in the quality of speech, and • variability in the acquisition environments in terms of acoustics as well as illumination and background. Below we describe how the database was acquired, including the recording protocols that were followed.

A. Database Acquisition

The mobile phone database was captured using two mobile devices: one being a Nokia N93i mobile phone and the other being a standard 2008 MacBook laptop. The laptop computer was used to capture only one session (the very first session) while all the other data was captured on the mobile phone, including the first session. The first session was a joint session with two separate recordings, one on the laptop computer and a second recording on the mobile phone captured on the same day at approximately the same time. Capturing the database on a mobile phone meant that the database acquisition was inherently uncontrolled because the mobile phone was given to the user and so the recording device was no longer in a fixed position. To ensure that we obtained useful data we enforced some minimal constraints upon the participants and validated the recorded data using automated tools.

Two constraints were placed upon the users when recording their data. First, we asked the user to ensure that most of the their face was in the image. To help with this, we provided live video feedback as shown in Figure 2 (a), with an ellipse (drawn in red), see Figure 2 (b), being the preferred position of the face. Second, we asked the users to be seated and that they generally talked to the phone. In addition to these minimal constraints, we also validated the data.

To ensure that the acquired data was meaningful and useful two automatic tools were used to validate the audiovideo samples. The first tool was a simple speech/nonspeech detector2 that allowed us to measure the signal-tonoise ratio (SNR). If the SNR for an audio-video sample was higher than 6 dB then the sample was inspected by a human Figure 2: Example images showing how the data was captured when using a mobile phone. In (a) there is an example of a user holding the mobile phone for a recording and in (b) there is an example of the video feedback to the user with the ellipse (in red) in which the face is meant to be within; note that the image in (b) has been digitally altered to enhance the ability of the reader to see the red ellipse.

operator to ensure its intelligibility. The second tool was a simple face detector, similar to the one described in [START_REF] Rodriguez | Face detection and verification using local binary patterns[END_REF]. If this face detector gave a response in less than half of the frames of an audio-video sample then it was inspected by a human operator to ensure that a face was present in the majority of the audio-video frames. If the data failed to pass the inspection of the human operator it was recorded again.

B. Recording Protocols

The database was captured over one and a half years and so it was captured in two phases, Phase I and Phase II. In total more than 61 hours of audio-video data was captured for 150 participants. A summary of the database is given in Table I. To ensure that the data was recorded in a consistent manner at each site for each user a dialogue manager (DM) was implemented on the mobile phone. The DM prompted the participants with: short response questions, free speech questions, and to read a pre-defined text. The DM also presented instructions to the participants such as the pre-defined, and fictitious answers, for the short response questions.

As already noted, the data collection was conducted in two phases: Phase I and Phase II. Each session of Phase I consisted of 5 short response questions, 5 short free speech questions, 1 pre-defined text, and 10 free speech questions.

Phase II was made shorter and consisted of 5 short response questions, 1 pre-defined text, and 5 free speech questions. In all cases the users were asked to speak in a natural manner and so people often correct themselves or add extra words, as is normal for realistic free speech. In addition, the responses to the free speech questions are always considered to be fictitious and so do not necessarily relate to the question as the sole purpose is to have the subject uttering free speech.

1) Short Response Questions: The short response questions consisted of 5 pre-defined questions, which were: (i) "What is your name?", (ii) "What is your address?", (iii) "What is your birth date?", (iv) "What is your license number?", and (v) "What is your credit card number?". Fictitious responses were provided for each participant and were constant throughout the recordings. The response for the license number was the same for each participant. Table I: We present a summary of the number (NB) of: subjects per site, sessions per subject and shots (questions) per session.

2) Free Speech Questions: The free speech questions were designed to ensure that the participant talked for approximately 10 seconds; for the short free speech questions we only required that they speak for approximately 5 seconds. The questions were randomly selected from a list of approximately 40 questions and the answer did not have to relate to the question.

3) Pre-Defined Text: The users were asked to read out a pre-defined text. This text was designed to take longer than 10 seconds to utter and the participants were allowed to correct themselves while reading these sentences. The text was the same for all of the recordings.

III. EXPERIMENTAL PROTOCOLS

In this section we present the authentication protocol for the full MOBIO database. The database is split into three non-overlapping partitions for: training, development and evaluation. The data is split so that two of the six sites, used for data collection, are used in totality for one partition. This ensures that no identity in one partition will be present in the other partitions.

The training partition can be used in any way deemed appropriate and all of the audio-video samples (192 separate audio-video samples for each participant) are available for use. The normal use of the training set would be to derive background models or sub-spaces. For instance the training partition could be used to derive a linear discriminant analysis (LDA) sub-space. This partition consists of the data collected from LIA and UNIS and has a total of 50 (13 female and 37 male) subjects.

The development and evaluation partitions are used to evaluate the performance of the systems. The development partition consists of data acquired from UMAN and UOULU and has a total of 42 (18 female and 24 male) subjects. It can be used to derive hyper-parameters, such as the optimal number of feature dimensions or weights to perform fusion of the scores from the face and speaker authentication systems, such as for the system described in Section IV-C. The development partition is also used to derive thresholds that are then applied to the evaluation partition to obtain the final system performance, see Section III-A for details. The evaluation partition should be used sparingly as it provides the final, unbiased, performance of the system. It consists of the data collected from BUT and IDIAP and has a total of 58 (20 female and 38 male) subjects.

The development and evaluation partitions have a similar protocol which consist of: (i) making a model for each client (client enrolment), and (ii) testing client models to produce true access scores and false access scores. We make a model for each client by using the first 5 questions from the their first session, these are the short response questions and we refer to this as the enrolment data. Testing of the client models is performed in a gender-dependent manner using the free speech questions from the remaining 11 sessions. We use the free speech questions because it is different to the enrolment data and thus represents a harder and more realistic problem. This leads to 105 true access scores, or samples, for each client; a true access score corresponds to a sample (video or audio) of the i th client that is compared against the model of the i th client. The false access scores are obtained by using the 105 samples from all of the other clients of the same gender; a false access score corresponds to a sample (video or audio) of the i th client that is compared against a model(s) that is not the i th client. Thus, if there are 30 models for a particular gender then false scores would be produced by using the 105 test samples from the other 29 users. For clarity and ease of use, the database, protocol files and annotations have been made publicly-available at http://www.idiap.ch/dataset/mobio3 .

A. Performance Evaluation

In this section we outline how to measure the performance of an authentication system. This should not be confused with a recognition system, the recognition system consists of multiple steps one of which is authentication. For instance face recognition consists of face detection followed by face authentication. Authentication is the process of confirming if the sample t came from the claimed identity i, by comparing it against their model θ i . The comparison against the model produces a score s which is compared against a threshold τ and is accepted as confirming their identity if s > τ .

To measure the performance of an authentication system we propose three methods. The first is to present a single number to represent the authentication system's performance using the half total error rate (HTER). The second is to present a summary of the authentication system's performance using a detection error tradeoff (DET) plot. The third is to present the unbiased performance of the authentication system using an expected performance curve (EPC).

The HTER is used to represent the performance of an authentication system with a single number. The HTER is obtained by deriving a threshold, τ * eer , on an independent data set at the equal error rate (EER) point. This threshold is then applied to the evaluation partition (D evl ). The HTER is the weighted error rate (WER) [START_REF] Bengio | The Expected Performance Curve[END_REF] of the false alarm rate (FAR) and the false rejection rate (FRR) when β = 0.5,

W ER(τ * eer , D evl , β) =β • F AR(τ * eer , D evl ) + (1 -β) • F RR(τ * eer , D evl ), (1)
The DET [START_REF] Martin | The DET curve in assessment of detection task performance[END_REF] and EPC [START_REF] Bengio | The Expected Performance Curve[END_REF] plots provide a quick summary of the overall system performance. The DET plot consists of the miss probability (FRR) versus the probability of false acceptance (FAR). However, the DET plot can be misleading because it presents the authentication system performance without an associated threshold. Therefore, we also propose the use of the EPC which presents a summary of the WER for different operating points [START_REF] Bengio | The Expected Performance Curve[END_REF]. When producing an EPC an optimal threshold τ * o has to be derived for multiple operating points, o, corresponding to a tradeoff between the FAR and FRR. Thus, the optimal threshold τ * o is computed using different values of β o ∈ [0; 1] corresponding to different operating points,

τ * o = argmin τ W ER(τ, D dev , β o ), (2)
where D dev denotes the development set. Finally, the performance for different values of β o is then computed on D evl using the threshold tuned for each operating point.

IV. BI-MODAL RECOGNITION SYSTEM

The bi-modal recognition system consists of three parts: face recognition, speaker recognition, and fusion. Each of these parts is described in detail below.

A. Face Recognition System

Face recognition consists of two components: face localisation and face authentication. Both components rely on using local binary patterns (LBPs) [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF], or a variant called the modified census transform (MCT) [START_REF] Froba | Face detection with the modified census transform[END_REF], to efficiently encode the face image.

The face localisation component consists of finding the largest face in each video frame. To do this we apply a face detector that is trained as a cascade of classifiers using MCTs. We took the implementation of [START_REF] Rodriguez | Face detection and verification using local binary patterns[END_REF] and reimplemented it to use integer arithmetic; an integer arithmetic approach was taken to reduce the computation overhead of running on a mobile phone. This detector outputs a single detection, or localisation result, by considering the largest detected face as being the face of interest, this simplification is valid because it is expected that the user is interacting with the device and consequently their face should be the largest detected face. After the face localisation stage all detected frames are normalised to be 64 × 80 pixel gray scale images, with a distance between the eyes of 33 pixels. The detected frames are then passed onto the face authentication component.

The face authentication component divides the detected faces into non-overlapping regions and each region is then represented using a histogram of LBP features. This is a parts-based scheme which is similar to [START_REF] Zhang | Local gabor binary pattern histogram sequence (lgbphs): A novel non-statistical model for face representation and recognition[END_REF] but avoids the need to compute the 40 different Gabor wavelets. For each region, r, we calculate an average histogram, θ r , by taking the average over all of the detected frames from the enrolment or test videos. Thus, given the enrolment histograms for client i, θ i , and the test histograms for the video t, θ t , the similarity for the r th region is,

s r (θ i,r , θ t,r ) = J j=1 min (θ i,r (j), θ t,r (j)) J j=1 θ i,r (j) , ( 3
)
where J is the total number of histogram bins corresponding to the distinct number of LBP codes (J = 256 in this work). The similarity is then summed for all of the regions to produce a single similarity measure, or score, s = R r=1 s r . In our work we had R = 80 non-overlapping regions.

B. Speaker Recognition System

Speaker recognition consists of two components: voice activity detection (VAD) and speaker authentication.

The role of VAD is to select only speech and to discard silence and noise. The VAD component used in this work uses a Hungarian downscaled phoneme recogniser which is the cascade of 3 neural networks. The input is the long temporal context of mel-filter banks and the output is a posterior probability of detected phonemes [START_REF] Schwarz | Towards lower error rates in phoneme recognition[END_REF]. The posterior probability of all silence classes are summed for each frame to form a silence class and the same is applied for phonemes to form a speech class. Viterbi decoding is applied to smooth the output. After VAD all valid frames are passed to the speaker authentication component.

The speaker authentication component uses an i-vector extractor [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF] to obtain features which are then modelled using probabilistic linear discriminant analysis (PLDA) [START_REF] Prince | Probabilistic linear discriminant analysis for inferences about identity[END_REF]. The i-vector approach obtains a low dimensional fixed length feature vector to represent a variable length utterance. It does this by taking a set of observations from an utterance which are then used to obtain a point MAP estimate [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF], this point MAP estimate then yields the low dimensional i-vector.

The fixed length i-vectors extracted per utterance can be used as input to a standard pattern recognition algorithm. We use a PLDA model [START_REF] Prince | Probabilistic linear discriminant analysis for inferences about identity[END_REF] that provides a probabilistic framework applicable to fixed-length input vectors. The ivectors w X are assumed to be distributed according to the form

w X = m + Vy + Ux + (4)
where V is a sub-space of identity variability and U is a sub-space of channel (or session) variability. Using the PLDA model, one can directly evaluate the log-likelihood ratio, to obtain a score, for the hypothesis test corresponding to "whether the two i-vectors were or were not generated by the same speaker." Note that the difference between the enrolment segment (on which a model was created) and the test segment (which is scored against the model) vanishesscoring with i-vectors is symmetrical.

For the system to work on the mobile device we chose the following parameters. We obtained observations from the audio signal using 19 Mel-frequency cepstrum coefficients [START_REF] Reynolds | Robust text-independent speaker identification using Gaussian mixture speaker models[END_REF] and C0 features. This base feature vector was augmented with their delta and double delta coefficients resulting in 60 dimensional feature vectors which were then processed by short time Gaussianisation [START_REF] Pelecanos | Feature warping for robust speaker verification[END_REF] over a 50 frame window. A universal background model (UBM) [START_REF] Reynolds | Speaker verification using adapted Gaussian mixture models[END_REF] with 128 Gaussians was used (but usually it is 2048). The ivector was trained such that it resulted in a M = 400 dimensional vector. We used one simplification (constant GMM component alignment) for running an i-vector extractor on the mobile phone because of memory consumption issues [START_REF] Glembek | Simplification and optimization of i-vector extraction[END_REF]. The parameters for PLDA were 90 dimensions for the speaker sub-space V and full-rank (400) for the channel subspace U.

C. Score Fusion

Score fusion consists of normalising the individual face and speaker authentication scores and then fusing them. The similarity scores, for face authentication, and the loglikelihood scores, for speaker authentication, were normalised to produce probabilities such that the scores lie in the range [0, 1]. We achieve this by using logistic regression, hence taking y i as the input and producing probability, P i , as the output:

P i = (1 + exp(-ay i + b)) -1
(5) where a is a scaling factor and b is a bias.

We chose to use logistic regression for two reasons. First, it is a well establishe method in statistics, belonging to a family of methods called generalized linear models. Its optimisation procedure, known as "gradient ascent" [START_REF] Hastie | The Elements of Statistical Learning[END_REF], is well understood; it converges to a global minimum, meaning that there is a unique solution. Second, its output can be interpreted as a probability and so presenting this information to the user is more meaningful than just the raw score.

The final combined score is obtained by taking the product of the two scores, P video for face authentication and P audio for speaker authentication. Although the sum rule could have also been used. Both rules result in similar generalisation performance.

V. EXPERIMENTAL RESULTS

In this section we present the results for our fully automatic bi-modal system on the full MOBIO database. This system runs in real-time on a Nokia N900 and demonstrates that automatic bi-modal authentication can be deployed on a mobile phone; the system processes approximately 15 frames per second. To evaluate the performance of our face, speaker, and bi-modal systems we processed the MOBIO database on the mobile phone (a Nokia N900). A demonstration of our system can be found at http://www.mobioproject. org/demonstrations. We present the result for all three authentication components of our system, face, speaker, and bi-modal authentication, in Figure 3. The DET plot in Figure 3 (a) shows that the performance of the face and speaker authentication components are quite modest, while the EPC in Figure 3 (b) shows that their performance is consistent across the development and evaluation partitions. The modest performance of the face and speaker authentication systems can be attributed to the fact that simplified systems have been used in order to run in the mobile environment. Despite this, the fusion (bi-modal) system appears to perform quite well.

It can be seen in Figure 3 that the bi-modal system outperforms either modality on its own. The single best performing modality is audio (speaker authentication) which achieves an HTER of 17.7% for female trials and 18.2% for male trials, see Table II. Despite the fact that the audio modality performs as much as 37% better than the video modality it can be seen that performing bi-modal authentication, score fusion, leads to substantial improvements in performance. For instance, the performance is improved by 25% for female trials, to 13.3%, and by 35% for male trials, to 11.9%.

From these results we have shown that it is possible to implement a fully automatic bi-modal recognition system Table II: The results obtained on the mobile phone database for the face (video), speaker (audio) and bi-modal (fusion) authentication system, for male and female (fem.). All of these systems run in real-time on a Nokia N900 mobile phone. which can run in real-time on a mobile phone. To the best of our knowledge this is the first such system that has been benchmarked using a publicly-available database captured on mobile phones.

VI. CONCLUSIONS AND FUTURE WORK One of the main outcomes of this work has been to acquire and make publicly-available a mobile phone database. This database has associated protocols, that are well defined and so will allow for easy comparison between different systems.

To encourage the development of other mobile phone based recognition systems we have described and evaluated our own novel bi-modal recognition systems. This system runs in real-time on a Nokia N900 and performs face, speaker, and bi-modal recognition. We have shown that this can be effectively deployed on a mobile phone and can be used for authentication in real-world conditions for a mobile phone. Finally, we have also shown that performing the fusion of face and speaker recognition can provide a significant performance improvement with a relative gain in performance of more than 25%.

We hope that future work will examine other techniques to perform accurate face, speaker and bi-modal authentication on a mobile phone using the public and free database described in this paper.

Figure 1 :

 1 Figure 1: Example images from two individuals. It can be seen that there is significant pose and illumination variation between all of the images. Also, it can be seen that the facial hair, (e)-(h) as well as hair style and make-up, (a)-(d), fluctuates significantly.

Figure 3 :

 3 Figure 3: We present in (a) a DET plot and (b) an EPC plot of the results obtained on the mobile phone database. Results for the face (FV), speaker (SV) and bi-modal (Fusion) authentication systems are presented in red, blue and black respectively. The gender-dependent trials for males and females are presented as the solid and dashed lines respectively.

The data was recorded at the following sites: University of Oulu (OULU), Idiap Research Institute (IDIAP), University of Avignon (LIA), University of Manchester (UMAN), University of Surrey (UNIS) and the Brno University of Technology (BUT).

Available at www.irisa.fr/metiss/guig/spro/ (a) (b)

The development partition consists of 1, 890 true and 32, 130 false access scores for female trials and 2, 520 true and 57, 960 false access male trials. The evaluation partition consists of 2, 100 true and 39, 900 false access scores for female trials and 3, 990 true and 147, 630 false access scores for male trials.

ACKNOWLEDGEMENTS

This work has been performed by the MOBIO and Tabula Rasa projects of the 7th Framework Research Programme of the European Union (EU) grant agreement numbers 214324 and 257289. For more information about the MOBIO consortium please visit http://www.mobioproject.org.


Anthony Larcher
email: anthony.larcher@univ-avignon.fr

Jean-François Bonastre

John S D Mason

Jean-François Bonastre

Jean-Francois Bonastre
email: jean-francois.bonastre@univ-avignon.fr

Short Utterance-based Video Aided Speaker Recognition

come

I. INTRODUCTION

The efficiency of speaker recognition systems in a realistic application could be influenced by several constraints. For example, an application which should be immediately usable will strongly limit the enrolment material and hence could lead to poor recognition accuracy. Given that limited data can greatly reduce the recognition performance, ergonomic constraints could also impose short test sequences influencing similarly performance. Some other constraints can be the memory and computational resource limitation or the use in variable environments. Embedded systems might well present these conditions. State-of-the-art speaker recognition engines tend to be assessed on text-independent inputs and follow often the GMM/UBM (Gaussian Mixture Model/ Universal Background Model) paradigm [START_REF] Bimbot | A tutorial on text-independent speaker verification[END_REF]. This solution gives a high level of performance as shown during the NIST evaluations [START_REF] Przybocki | NIST speaker recognition evaluations utilizing the mixer corpora -2004[END_REF]. Unfortunately, the UBM/GMM depends strongly on the quantity of training data available to enrol a speaker in contrast to the context considered here which involves relatively short duration speech material. A solution to this problem is to increase the amount of information taken into account by the system by including text dependencies, like in a user-customised utterance scenario [START_REF] Benzeghiba | User-customized password speaker verification using multiple reference and background models[END_REF]. In this case, the Temporal Structure Information (TSI) gathered from the utterance can help to compensate for the short duration of the audio sequences. In order to model the TSI of speech while achieving statistical modelling, a word recognition system could be combined with a speaker recognition system [START_REF] Hebert | Phonetic class-based speaker verification[END_REF], [START_REF] Navratil | A speech biometrics system with multigrained speaker modeling[END_REF]. To satisfy application ergonomic constraints and to allow the speaker to chose is own customised-utterance, the system should accept all kinds of utterances, especially short duration ones, and also be language-independent. Adding language options when using a phoneme-based word recognition system would seem viable with an appropriate choice of phonemes covering the languages. However, this solution could be expensive in terms of storage and computational cost. Furthermore, an embedded system could be confronted with strongly variable environments. Due to this constraint the acoustic modelling used in the recognition system has to be adapted to the environment and the computational cost of the adaptation has to follow the targeted context resource constraints. HMM adaptation does not seem well suited as it normally requires a relatively large amount of training data. The solution proposed in this paper tries to associate the well known advantages of a GMM based statistical acoustic model with an original architecture able to deal with the application context constraints and to incorporate external temporal information. It uses the GMM/UBM paradigm for the general acoustic space modelling and its text-independent speaker recognition capabilities. It also involves an HMM/Viterbi approach in order to incorporate the text-dependent and TSI aspects using a Semi-Continuous HMM (SCHMM) [START_REF] Young | The general use of tying in phoneme-based hmm speech recognisers[END_REF]. Such a combined system was originally proposed in [START_REF] Bonastre | Gaussian dynamic warping (gdw) method applied to text-dependent speaker detection and verification[END_REF] for speaker recognition and extended to word recognition in [START_REF] Lévy | Mobile Phone Embedded Digit-Recognition[END_REF]. Two mains approaches are possible to take into account the bimodal aspect of speech (audio-video) for speaker recognition. Generally, this problem is view as a fusion process between the two modalities. An early fusion at the data level is difficult due to the different nature of the parameters and their asynchronism. Several works were proposed, mainly in speech recognition [START_REF] Chibelushi | Feature-level data fusion for bimodal person recognition[END_REF] [10] and show a performance improvement only when noisy audio data are used. The fusion at the score level is more often proposed due to its simplicity [START_REF] Cetingul | Multimodal speaker/speech recognition using lip motion, lip texture and audio[END_REF], but such a fusion process does not take advantage from he temporal joint-information and it is still costly in terms of computational resources (separate systems are needed). Finally, an interesting alternative to a fusion process consists in a joint decoding processing of both modalities. However the asynchronism aspect of audio-video modalities leads to complex algorithms like in [START_REF] Bengio | Audio-and Video-Based Biometric Person Authentication[END_REF] and [START_REF] Chetty | Liveness detection using cross-modal correlations in face-voice person authentication[END_REF].

In this paper we propose a such joint-decoding which, thanks to the specific aspects of our system, shows an acceplevel of complexity. In order to reinforce the relaxed synchronisation between states and frames due to the SCHMM structure of the TSI modelling, we propose to embed video information during the audio decoding by adding further time-constraints gathered from a video synchronisation process. The specific three stage architecture and the enrolment algorithms are described in Section II as well as the way of reinforcing the TSI with an video-learnt synchronisation. The experimental protocol and results are described in Section III including a description of the audio-video database, MyIdea. Section IV summarizes the benefits of this approach and presents possible future work directions.

II. DESCRIPTION OF THE APPROACH

The proposed approach, called EBD for Embedded LIA SpkDET takes advantages of three mechanisms which contribute to the overall scoring process.

• GMM/UBM, the architecture of the two first layers of the EBD is similar to a classical GMM/UBM speaker recognition system. Speaker text-independent model of every client is trained by adapting the Universal Background Model. This adaptation is described below. • SCHMM, the previous text-independent speaker model is then used to obtain an SCHMM with the goal of harnessing the TSI of the utterance chosen by this speaker. Each state of the SCHMM is trained from a part of that utterance using an iterative Viterbi decoding process.

During the test, Viterbi decoding is again performed with this SCHMM. Details of the training and test are given below.

• Video-Learnt information, the goal here is to use further information to assist in the overall verification task by adding additional constraining components. These constraints are computed during the training phase and used to constrain both the training and test Viterbi decoding.

As the video information is only computed during the train, it could be computed off line. This information is here labelled "video-learnt" to reflect that it is only computed during the training phase.

A. EBD Hierarchical Architecture

Figure 1 shows the architecture of the EBD system. Nodes of this structure are GMM models. The upper layer is the least specialised one. It is a classical UBM which aims at modelling the acoustic space. The middle layer contains speaker specific text-independent models. These GMMs are obtained by a classical GMM/UBM adaptation process. Each model is derived from the UBM by using the EM algorithm and following the Maximum A Posteriori (MAP) criterion [START_REF] Gauvain | Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains[END_REF]. Only the mean parameters are adapted and other parameters are shared with the UBM. The bottom layer uses the ability of the left-right SCHMM to capture the TSI of the user-customised utterances. Each of the SCHMM states is a GMM derived from the corresponding middle level model. As explained below, the transformation function works only on the weights of the GMMs, the other parameters are taken from the middle layer model.

B. Training Step

The EBD model is trained in three steps, each corresponding to one level of the architecture. The UBM is firstly trained to model the largest part of the acoustic space. It is built off line using a suitably large amount of representative data. It is trained with a classical EM/ML algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the em algorithm[END_REF]. The training of the speaker text-independent models consists in adapting the UBM/GMM with the available data pronounced by the client. An energy labelling is performed on the signal and only the frames deemed to be speech are kept. The model is obtained by adapting the UBM using the EM algorithm with the MAP criterion. In order to initialise an utterance SCHMM model, the utterance sequence is cut into S segments {seg i } of the same length. Each state i of the SCHMM is adapted from the speaker text-independent model using the speech-labelled frames of seg i . An EM/MAP algorithm is applied on the weight parameters. Then the SCHMM is optimised using a classical Viterbi algorithm (a new segmentation is achieved by Viterbi and is used to adapt the state models). The number of states of the SCHMM is experimentally determined. Finally, the transition probabilities of the SCHMM are computed using the relative length of each segment. One advantage of this process is that all parameters except weights are tied between the states of the SCHMM and the text-independent speaker model. The log-likelihood for an input frame is only computed for each Gaussian component of the text-independent model. Then the log-likelihood of this frame with each state of the SCHMM only involves a weighted sum which is negligible compared to the full log-likelihood computation.

C. Testing Step

During a test, the score between the input signal and an utterance SCHMM model is derived from the corresponding Viterbi path. All the input frames are used during this Viterbi decoding phase. The log-likelihood of a frame sequence could be expressed as a sum of two log-likelihood accumulations, one using speech-labelled frames and the other using nonspeech-labelled frames, as shown in Equation 1.

log p(X|λ) = log p(X speech |λ) + log p(X non-Speech |λ) (1)
The final speaker matching score corresponds to the loglikelihood computed with the log p(X speech |λ) only. As for the training, the log-likelihood for an input frame is only computed for each Gaussian component of the textindependent model. The computation of the log-likelihood of this frame with each state of the SCHMM which involves a linear combination is negligible. This scoring process equivalent in terms of computation to a classical GMM/UBM system produces two scores. The first is obtained with only the textindependent speaker model and the second is computed with the SCHMM model, which itself has two operational modes, namely without further constraints or constrained by the videolearnt information from the initial training phase. These two scores are normalised using the log-likelihood of the UBM. They are then combined to give a final score for the decision stage. An empirically-tuned weighted linear combination is used.

D. Video-Learnt Synchronisation

Synchronisation points are extracted from a very simple video process. The video stream is first pre-processed to obtain a black-and-white sequence which is the Y component of the sequence resulting of an RGB to YCbCr transformation. A mono-dimensional temporal signal is issued from this blackand-white video stream in order to estimate the quantity of change between successive frames. Subtractions are processed between the pixels of one image and thus of the following one. The absolute values of pixel subtractions are summed to obtain a value of the discrete temporal signal S. This computational process is described by Equation 2.

S(n) = W w=0 H h=0 abs(I n (w,h) -I n+1 (w,h) ) (2)
where W and H are respectively the width and the height of the video images, I n and I n+1 are two consecutive images of the video stream and I n (w,h) is the value of the pixel (w, h) of the image I n . S(n) is the discrete temporal signal from which the synchronisation points are extracted. Local maxima of the signal S are found by applying a sliding window algorithm. These local maxima are stored to become the video-learnt synchronisation points. These points, which could be generated off-line during the training phase only, are subsequently used during both the training and test phase to strongly constrain the Viterbi decoding. This result is obtained by allowing or forbidding transitions of the SCHMM corresponding to the synchronisation points and labelled V , for video, in Figure 2. Other transitions, labelled A are computed from the Audio and not modified when adding the video-learnt segmentation. The

III. EXPERIMENTS

A. MyIdea Database

Experiments are performed on the BIOMET part of the MyIdea database. This database contains audio-video records from 30 male speakers. In this subpart of MyIdea, 25 sentences are recorded in 3 sessions for each speaker. Twelve of these sentences are the same for all the speakers, ten short (about 3 seconds) and 2 long sentences (about 6 seconds). Other occurrences are speaker or session dependent. The three sessions are recorded under controlled acoustic and illumination conditions. However, MyIdea presents several drawbacks for our work. The recordings are not made in a real environment. The sentence duration variability is limited (2 or 3 seconds for the utterance occurrences), the sentences are too long for a real password dedicated system, and the number of speakers is small for a speaker verification experiment.

B. Protocol

The 30 male speakers are separated into two groups -A and B-each with 15 speakers. Each group is successively used as the Client-set with the others used to train the UBM. The UBM is trained using the whole recorded material of the 15 speakers of the UBM-set. When using the A-group data set to train the UBM model, the speakers from the B-group are used for enrolment and tests. Due to the small number of speakers available, a jacknifing process is used by training a client model for each available speaker session. Each of the 15 speakers of this Client-set is successively considered as a client for which the 14 other speakers of the Client-set are impostors. Two conditions are defined:

• 1-occ in this condition, each client text-independent GMM model is derived from the UBM by using two long sentences and one occurrence of the selected short sentence (around 8 seconds of speech). The utterancedependent model is trained with the same short sentence occurrence (around 2 seconds of speech). With the jacknifing process, 900 utterance models are trained (10 short sentences, 3 sessions and 30 clients). • 2-occ this condition is the same as above except that an additional occurrence of the selected short sentence (which is the chosen utterance) is used to train both the text-independent and the utterance models. The number of utterance models is still the same as above. The number of target trials is condition-dependent. The short sentences not used for utterance training are compared to the client model. 1,800 client tests are performed in the 1-occ condition (2 test occurrences for each of the 900 utterances) while 900 client tests are performed in the 2-occ condition (1 test occurrence for each of the 900 utterances). Three configurations of impostor tests are proposed. The speaker and utterance models are compared to the 14 impostors who are the remaining speakers of the same group.

• UNKNOWN configuration the linguistic content of the test occurrences is different from the training material of client models. Each speaker model is compared to three randomly selected short sentences (one per session) out of the 9 remaining sentences of each of the 14 impostor speakers. 37,800 impostor tests are performed in this configuration.

• KNOWN configuration the linguistic content of the test sequences is the same as the occurrences used to train the client models. Each utterance model is compared to three randomly selected sentences from each of the 14 other speakers of the Client-set. 37,800 impostor tests are performed in this configuration. • ALL configuration the tests are from both the KNOWN and the UNKNOWN configurations. The number of impostor tests in this configuration is 75,600.

C. System Configuration

Mel-scaled frequency cepstral coefficients (MFCC) are used here, computed every 10ms. An energy labelling is applied to separate the speech frames from the non-speech frames. Acoustic feature frames are 32-dimension vectors, 15 cepstral coefficients, the log-energy and the corresponding ∆ coefficients. In the experimental configuration, the number of components in GMMs is fixed to 128 and all mean parameters of the textindependent speaker models are adapted. Only the 32 most significant weights parameters are adapted for each SCHMM state. The linear-combination of the two scores described in Section II-C is computed with: 0.3 for the score computed with the text-independent speaker model; 0.7 for the final EBD text-dependent score. These coefficients are empirically determined.

D. Results

Experiments are conducted to assess the contributions coming from the three components, GMM/UBM, SCHMM and video-learnt information. The GMM is regarded as the baseline and the benefits of the other two are predicted to come from TSI. The experimental results presented in Table I, expressed in terms of equal error rates (EER), show performance of the EBD system depending on the number of states, the nature of the impostor tests and the quantity of training data. No video-learnt synchronisation is used in these experiments. Performance of the baseline GMM in the same conditions is provided for comparison. It is important to note that the GMM system reflects the middle layer of the EBD system. The first two rows in Table I show the results when the impostors do not know the speaker utterances. Error rates fall from 2.72 down to 2.00 and 0.87 to 0.55 for the 1-occ and 2-occ respectively when SCHMM is used. Increasing the number of states up to 20 allows successive improvements in reducing EER. Unfortunately this result is not confirmed when the impostors know the client utterances (KNOWN). A loss of performance is observed when the number of states increases. It seems that, in speaker matching scores, the TSI is dominated by the utterance content information rather than the speaker specific information, i.e. the system recognises the utterance instead of the speaker. This results are confirmed in Table I when two utterances occurrences are used for the training (2-occ). Indeed, the GMM system performs better than the EBD and seems to gain more from the increase in training data than the EBD except in the UNKNOWN condition where the increase of training data also increases the advantage of the EBD by reinforcing the TSI exploited by the SCHMM models. In order to evaluate the effect of a video-learnt synchronisation, a new experiment is performed. Results of this experiment are presented in Table II. The first column is the original GMM baseline and the next two columns give the EBD results as above in Table I for the 20 state case, and then with the use of the video-learnt information. As expected, performance of the EBD improves when using video-learnt segmentation. This is particularly highlighted in 1-occ conditions where the system possesses less speaker specific data. The EER in the ALL configuration fall from 3.72 down to 3.16. Moreover performance of the EBD system is greatly increased in the case of impostor knowing the utterances (KNOWN configuration), since performance of the EBD becomes equivalent to the baseline GMM/UBM in this configuration (the EER in 1occ KNOWN configuration are 4.49 for the baseline GMM and 4.44 for the EBD). It seems that the video segmentation provides information on the utterance but also on the speaker. Moreover, the EBD using an video-learnt synchronisation gains more from the increase of training data than the same system without this temporal information.

IV. CONCLUSIONS AND FUTURE WORKS

The approach proposed in this paper is designed for embedded applications. It takes advantages of a GMM/UBM text-independent approach and the HMM/Viterbi speechrecognition power. In addition we propose to reinforce the Temporal Structure Information modelling by a synchronisation issued from a video stream. The use of the temporal information in a speaker recognition system allows to improve performance particularly when a relatively small quantity of speech data is available for training and test. Performance of our approach is equivalent to the GMM/UBM baseline system when not considering the linguistic content (example of EER in KNOWN condition, GMM: 4.49, EBD 4.56 ) whereas the proposed approach outperforms the GMM/UBM when impostors do not know the client utterance (EER in UNKNOWN condition, GMM: 0.87, EBD: 0.56). Furthermore, the video-learnt synchronisation leads to a gain in all situations, when impostors know or not the clientutterance, and outperforms or is equivalent to the GMM/UBM baseline in all situations. Future work will focus on the multi-modality by using the video segmentation even during the testing phase. By incorporating this strong constraint in the training and also in the testing phases we aim at increasing performance and to thwart replay attacks. The EBD approach will be tuned to better balance the speaker and utterance specific information in order to outperform the baseline GMM in all conditions. Moreover, as the first results show the ability of the EBD approach to take advantage of the temporal structure of customised utterances, more tests have to be performed to evaluate the performance of the EBD system with more utterance-variability, for example considering the utterance duration.

Fig. 1 .

 1 Fig. 1. General view of the EBD model architecture.

Fig. 2 .

 2 Fig. 2. Use of an Video-Learnt Segmentation in the bottom layer of the EBD system to constrain the Viterbi decoding and reinforce the TSI in the utterance SCHMMs. The V labelled transitions are constrained by the videolearnt segmentation as the A labelled ones still unchanged

TABLE I EER

 I OF GMM AND EBD SYSTEMS (WITH DIFFERENT NUMBERS OF STATES) USING ONE OR TWO TRAINING OCCURRENCES IN DIFFERENTTEST CONFIGURATIONS

TABLE II EER

 II OF GMM COMPARED TO THE EBD SYSTEM IN 20 STATE CONDITION, WHEN USING OR NOT A VIDEO-LEARNT SEGMENTATION

ACKNOWLEDGMENT

BioBiMo (Biométrie Bimodale sur Mobile) is a project supported by the French A.N.R. (Agence Nationale pour la Recherche). http://biobimo.eurecom.fr/ Mistral is a project supported by the French A.N.R. (Agence Nationale pour la Recherche). http://mistral.univ-avignon.fr/ MoBio is a European Funded Project (FP7-2007-ICT-1). http://www.mobioproject.org/


domain: math shs

text:
Jeffrey A Mclean
email: jmclean@stlawu.edu

Helen M Doerr
email: hmdoerr@syr.edu

Students' conception of a sample's relation to the population when constructing models of resampling

Keywords: Statistics education, modeling, informal inferential reasoning

In this study, we examined the models of resampling that a group of students constructed in order to use one sample to make informal inferences about a population of data. Students participated in a model eliciting activity which aimed to elicit the resampling process of bootstrapping. We will discuss the model of resampling and inference constructed by one group of students, the factors which led to the group constructing a model of resampling with replacement, and how these students' conceptions of the sample related to their model. We suggest that the students' conception of the relationship between the sample and population was a key factor in constructing a model of resampling and inference similar to the method of bootstrapping.

Introduction

Bootstrapping is one method of simulating data through resampling that has become an important tool for statisticians, who suggest that it is intuitive to novice statistics students [START_REF] Lock | Statistics: Unlocking the power of data[END_REF]. The term bootstrap is used in the idiom to pull oneself up by one's bootstraps, which means to accomplish a goal with the resources on hand. Rather than take many samples to estimate a sampling distribution, the bootstrap method uses the one sample on hand to accomplish the same goal. Bootstrapping is a process of simulating data beginning by drawing one sample from a population. Resamples are constructed by choosing elements from the original sample, one at a time with replacement, until as many elements are drawn as in the original sample. This new sample is called the bootstrap sample. The process is repeated many times to create a collection of such bootstrap samples. A statistic from each of these bootstrap samples is aggregated to form an empirical bootstrap sampling distribution, which can then be used to make inferential claims about the population from which the original sample was drawn.

Bootstrapping was introduced by [START_REF] Efron | Bootstrap methods: Another look at the jackknife[END_REF] as an alternative to earlier resampling methods. He asserted that the bootstrap was more widely applicable and dependable than earlier resampling methods, while also using a simpler procedure. New curricula for introductory statistics courses at the secondary and tertiary levels have been created which focus on simulation and resampling methods such as bootstrapping [START_REF] Garfield | Developing statistical modelers and thinkers in an introductory, tertiary-level statistics course[END_REF][START_REF] Pfannkuch | Bootstrapping students' understanding of statistical inference[END_REF][START_REF] Tintle | Development and assessment of a preliminary randomization-based introductory statistics curriculum[END_REF]). Pfannkuc h et al. (2013, p. 2) asserted that "the bootstrapping and randomisation methods using dynamic visualisations especially designed to enhance conceptual understanding have the potential to transform the learning of statistical inference." [START_REF] Lock | Statistics: Unlocking the power of data[END_REF] claimed that bootstrapping capitalizes on students' visual learning skills and helps to build students' conceptual understanding of key statistics ideas. While these curricula and subsequent research have examined students' understandings of the use of method of bootstrapping, studies have not examined how students construct and develop these understandings. We suggest that this lack of research on student understandings of eliciting the concept of bootstrapping is a gap in the research literature. In this study we examine aspects of the model of resampling and bootstrapping that one group of students constructed in order to make informal inferences about a population of data and these students' conception of the sample's relation to the population. The research question driving this study was what factors led students to construct models of resampling with replacement.

Review of literature

The current research literature has viewed the bootstrapping method through the lens of formal inferential reasoning, such as using the method as a means to estimate standard errors and construct confidence intervals [START_REF] Garfield | Developing statistical modelers and thinkers in an introductory, tertiary-level statistics course[END_REF][START_REF] Pfannkuch | Bootstrapping students' understanding of statistical inference[END_REF]. In this study, rather than focus on students' understandings of how this method could be used for formal inference, we focused on how students constructed and developed methods similar to bootstrapping and made informal inferential claims from the resulting empirical bootstrap distributions. Key to eliciting students' models of resampling and inference in this study was understanding the use of students' informal inferential reasoning to make claims about a population. Informal inferential reasoning is the drawing of conclusions from data that extend beyond the data, from viewing, comparing, and reasoning with distributions of data [START_REF] Makar | A framework for thinking about informal statistical infere nce[END_REF][START_REF] Pfannkuch | Year 11 students' informal inferential reasoning: A case study about the interpretation of box plots[END_REF].

In order to make informal inferential claims in this study, groups of students first needed to determine a method of simulating data. [START_REF] Saldanha | Conceptions of sample and their relationship to statistica l inference[END_REF] explored students' conceptions of a sample in relation to the population while the students participated in an activity which constructed an empirical sampling distribution by collecting repeated samples from a population. Students' conceptions of a sample in relation to the population and sampling distributions were categorized as either additive or multiplicative. Those with an additive conception of the sample only viewed the part-whole relationship between the sample and the populations, with multiple samples representing multiple parts of this whole. The resemblance and relationship between the sample and populatio n distributions was not a factor for those with this conception. Those with a multiplicative conception of the sample viewed the sample as a "quasi-proportional, mini version" (p. 266) of the populatio n. The sample can be used to approximate the distribution of the population, with an understanding that various samples' distributions may bear more or less resemblance to the distribution of the populatio n. We posit that this multiplicative view may be critical to understanding the use of bootstrapping for informal inferential reasoning, since the proportion of each element in a sample is assumed to represent the proportion of those elements in the population.

The focus of analysis for this study was the model of resampling and bootstrapping which participants constructed and developed while engaged in a model eliciting activity [START_REF] Lesh | Model development sequences[END_REF]. Models are "conceptual systems … that are expressed using external notation systems, and that are used to construct, describe, or explain the behaviors of other system(s)" (Lesh & Doerr, 2003, p. 10). Model-eliciting activities encourage students to generate descriptio ns, explanations, and constructions in order to reveal how they were interpreting situations. Model eliciting activities are designed in order for students to:

 Make sense of the situation drawing on both their school mathematics real-life sense-making abilities;

 Recognize the need to construct a model to complete the activity, rather than produce only an answer.  Create documentation that shows solution paths, patterns, and irregularities that the students considered while constructing their model;  Assess when their responses need to be improved, refined, or extended,  Create models that can be extended to use in a broader range of situations [START_REF] Lesh | Principles for developing thought revealing activities for students and teachers[END_REF].

By using a modeling approach to examine student reasoning, we viewed reasoning as dynamic and developing over the course of instruction. Observing students as they participated in a model eliciting activity allowed us to view the construction and development of their thinking of resampling and inference.

Design and methodology

This study is a qualitative case study and part of a larger study [START_REF] Mclean | The development of informal inferential reasoning via resampling: Eliciting bootstrapping methods[END_REF]) that consisted of an eight-class-session teaching experiment that was enacted in four introductory statistics classes at the high school and community college levels (n=68) in the United States. This study focuses on the student reasoning that developed during one class-session as a group of four students from a community college participated in a model eliciting activity that aimed to elicit the method of bootstrapping. During the model eliciting activity, we collected written classwork from all participants and videotaped the group of students in order to document the group's model construction and development. We analyzed the videos and written classwork for evidence of informal inferential reasoning, quantities used and the relationships between these quantities, and the representations and explanations used in their arguments in order to reconstruct the development of the reasoning that were created by the participants.

The group of students participated in a model eliciting activity, using hands-on manipulatives, where they were given one sample from a population and constructed a model of resampling and infere nce in order to make claims about a population. The activity was designed to elicit a model that could be used to resample from a sample, with replacement, in order to construct a bootstrap sample of the same size as the original sample and use the distribution of these bootstrap samples to determine which outcomes are most likely to occur. The model eliciting activity asked groups of students to help a manager in a grocery store predict the percentage of peanuts in a certain brand of mixed nuts.

The students were given a sample of mixed nuts in the form of seven craft sticks marked with a 'P' for peanut and 18 not marked to represent other kinds of nuts. The model eliciting activity prompt and manipulatives are shown below (Figure 1).

[Grocery store] carries many types of nuts, dried fruits and candies in their bulk food section. The manager of bulk food is always interested in bringing new types of food for her customers to try. She recently ordered a sample of a new brand's mixed nuts. From past experiences, she has determined that customers prefer mixed nuts with fewer peanuts. She plans to order a large shipment of mixed nuts and is considering this new brand. Before she orders, the manager wants to know more information about the percentage of peanuts in this new brand. From this one sample of mixed nuts, the manager has asked that you determine a likely range for the percentage of peanuts in the entire brand of mixed nuts. She would also like to know the methods that you

Thematic Working Group 05

Proceedings of CERME10

develop to come to your conclusion. Your methods may of use for future bulk food purchases. The bag of sticks is the sample of mixed nuts. Sticks marked with a 'P' are peanuts. Those not marked are other types of nuts.

Results

In the larger study from which this data was gathered [START_REF] Mclean | The development of informal inferential reasoning via resampling: Eliciting bootstrapping methods[END_REF], groups of three or four students constructed two categories of models for making inferential claims of a population from one sample. The first category of model (n=16 groups of students) treated the sample of 25 nuts in a manner similar to a population and collected resamples without replacement, from the 25 nuts. The second category of model (n=4 groups of students) discussed or collected resamples from the sample of mixed nuts in a manner which preserved the makeup of the sample while resampling. We will report findings from one of the four groups who constructed this second category of model. This group constructed a model of resampling with replacement similar to the method of bootstrapping. We will discuss two findings of student understanding that were key to constructing these models: the representativeness of the sample to the population; and a method of resampling that preserved this representativeness. Findings like these were unique to the groups of students who constructed the second category of model of resampling and inference, which preserved the make-up of the sample while resampling.

represented the population and constructed a model of resampling with replacement that simulated resamples which upheld the quasi-proportional relationship of the sample to the population. These finding have implication for future curricula design by examining the key understandings that students may need before instruction on methods of resampling, such as bootstrapping. The finding also suggested that the use of model eliciting activities are useful design for instruction in introductory statistics classrooms in order to analyze students' developing thinking of resampling and inference.

Two limitations of this study were that: although this group of students constructed a model of resampling similar to bootstrapping, they did not collect resamples that were equal in size to the original sample of mixed nuts; and the time needed to sample by replacement with manipulatives did not allow the students to collect a distribution of samples from which to draw informal inferences. Collecting resamples of the same size is key in order to observe the variability in a statistic for samples like the original. While designing the model eliciting activity, we did not foresee issues with how the context of the activity encouraged students to make claims about the populations of mixed nuts, rather than the proportion of peanuts in future samples from the population of mixed nuts. Within this context, rather than use a bulk supply of mixed nuts in which samples of many sizes could be taken, the model eliciting activity may have been better designed to focus on the small packages of mixed nuts with consistent sample sizes. The time demanding nature of resampling could have been alleviating by using smaller original samples or with transitioning to the use of technology. A different context from mixed nuts where smaller samples felt natural, rather than merely a simplification may have allowed more resamples to be simulated. We made the choice to initially explore resampling with manipulatives and in later activities use technology to simplify the resampling process.

Figure 1 :

 1 Figure 1: Model eliciting activity manipulatives Prior to the model eliciting activity, the group of students participated in a model development sequence which elicited and developed models of repeatedly sampling from a populatio n, constructing an empirical sampling distribution, and making informal inferential claims regarding the population. In this model eliciting activity, students could no longer repeatedly sample from a population in order to construct an empirical sampling distribution and needed to construct a new model of resampling and inference that could use only one sample from the population to make informal inferential claims.

Proceedings of CERME10

Representativeness of the sample

Before discussing methods of resampling, the group of students first discussed how they believed that one sample could be used to make inferential claims and how their sample of mixed nuts related to other possible samples of mixed nuts taken from the sample population Susan:

I would say from this sample, that a little over a quarter of the peanuts, of the nut s are peanuts, from our random sample.

Randy: But it's only one sample.

Ted: But this is the only sample we have.

Brenda:

So if you pick another random sample what's going to happen?

Ted: It's most likely going to change.

Susan: It's going to change, but I feel like it will probably be still about the same.

Susan made an assertion about the percentage of peanuts in the population under the condition that she was basing her assertion on the random sample, which Randy stressed is only one random sample. Susan emphasized that although other samples of nuts will be different from their one sample, they will probably be "still about the same". Susan demonstrated an understanding that the percentage of peanuts in the sample likely represents the percentage of peanuts in the population. This is a key aspect of inferential statistics. When making inferential claims you take for granted that the sample likely represents the population because as Ted stated, "this is the only sample we have" and as Susan asserted, "it will probably still be about the same" as other samples.

Preserving the representativeness of the sample when resampling

The group initially decided to take a resample of 14 mixed nuts by drawing one at a time from the bag of mixed nuts, without replacement. The group only took one sample with this method, which yielded five peanuts out of 14 nuts. The instructor then came back over to the group to discuss how the group had collected the sample of 14 nuts.

Instructor From this exchange Ted considered how the sampling would change if they resampled with replacement instead of resampling without replacement. By using the term "probability simulato rs " he was approaching the idea that if you do not replace the stick after choosing each one, the probabilities of choosing a peanut or another nut will change. This term was not used previously in the coursework for this class, but likely came from his earlier experience in a mathematics or statistics classroom. Ted was combining the idea of the representativeness of the sample to the population, that the group discussed earlier, with the idea of the probabilities staying constant for each choice of nut, the key concepts of bootstrapping. The group continued to discuss how this process of replacement was different than their initial approach without replacement.

Randy: What are we going to do now?

Susan: Now I'm going to hand you back the Popsicle stick and you're going to mix it back in.

Randy: So you're going to draw … Susan: From the 25, not from, you know how before, like when I drawed [sic] and set it down, it went less and less and less and less and less?

Ted: So we're going to do it again, we're still going to draw 14, we're just going to put them in.

Susan: So each time we're drawing from 25, instead of a reduced … Ted: 'Cause as we would draw, in this one we would draw and there were 10, that meant that there were only 15 left in the bag, which doesn't account for the sample, right? 'Cause you're reducing it.

They asserted that when not using replacement, the nuts that they were drawing from the bag no longer represented the sample after some nuts were drawn and not returned. The group followed this procedure of resampling with replacement to collect a sample of 14 nuts. This was the only sample that they had time to gather before the class was reconvened to discuss each group's approach to determine the likely range of peanuts in this new brand. The group did not use this one sample to draw a conclusion since time ran out for developing their model.

Discussion and conclusions

We assert that a key difference between this study and previous research addressing data simulation and bootstrapping [START_REF] Garfield | Developing statistical modelers and thinkers in an introductory, tertiary-level statistics course[END_REF][START_REF] Pfannkuch | Bootstrapping students' understanding of statistical inference[END_REF], is the elicitation of bootstrapping methods by groups of students rather than the instruction of students on how to use the method. By eliciting the method, we were able to view students' statistical reasoning which led students to construct a process similar to bootstrapping: the representativeness of the sample to the population; and a method of resampling that preserved this representativeness. We assert that these students exhibited a multiplicative conception of the sample [START_REF] Saldanha | Conceptions of sample and their relationship to statistica l inference[END_REF]. These students claimed that the distribution of the sample likely