Emine Çatman Aksoy

Ayşe Nur Ucuzoğlu
email: aucuzoglu19@posta.pau.edu.tr

Mine Işiksal Bostan

Sibel Kazak
email: skazak@pau.edu.tr

Seçil Yemen Karpuzcu
email: secil.karpuzcu@dpu.edu.tr

Reyhan Tekin Sitrava

How Does Online Professional Development Nurture Teachers' Noticing Skills of Students' Thinking?: The Case of Data Analysis

Keywords: Collaboration, data analysis and interpretation, online professional development, teacher noticing

This paper presenting one aspect of a design-based research project examines the development of a teacher's noticing skills in the context of data analysis and interpretation during her involvement in an online professional development program (OPD). For this purpose, the noticing skills of the teacher for the strategies used in the given students' responses were identified according to the modified version of the Jacobs et al.'s (2010) framework. By examining the teachers' discussions in the OPD, how the teacher's noticing expertise evolved throughout this process was investigated.

Findings show that the teacher had difficulties in each interrelated skill of noticing at first. However, enhancement in her noticing of students' thinking in the context of data analysis was observed throughout her participation in the OPD.

Teacher noticing, which is a key part of teaching expertise [START_REF] Van Es | Expanding on prior conceptualizations of teacher noticing[END_REF] and has been a prominent construct in the mathematics education literature for the last 20 years, is generally defined as all the processes that teachers engage to deal with the ongoing information in the classroom [START_REF] Sherin | Mathematics teacher noticing: Seeing through teachers' eyes[END_REF]. Teacher noticing is essential for efficient instructional practice [START_REF] Blömeke | Beyond dichotomies: Competence viewed as a continuum[END_REF] and has a direct influence on students' learning [START_REF] Van Es | Expanding on prior conceptualizations of teacher noticing[END_REF]. Although different conceptualizations exist about teacher noticing, researchers generally regard it as including two main processes: paying attention to and making sense of the notable situations in the classroom [START_REF] Sherin | Mathematics teacher noticing: Seeing through teachers' eyes[END_REF]. Jacobs and her colleagues (2010) add one more process to the above description, which is decision-making. Furthermore, they focus particularly on noticing students' mathematical thinking, which is an essential practice to promote students' learning (National Council of Teachers of Mathematics [NCTM], 2014).

Despite the value of teacher noticing, the relevant studies suggest that noticing is not an innate ability and the teaching experience alone does not provide sufficient improvement [START_REF] Jacobs | Professional Noticing of Children's Mathematical Thinking[END_REF]. Therefore, researchers attempted to develop teachers' noticing skills through variety of ways. For instance, [START_REF] Fernandez | Prospective mathematics teachers' development of noticing in an online teacher education program[END_REF] showed evidence of enhancement in prospective teachers' noticing skills through sharing narratives about their own practices in an online forum and collaborating with their partners and tutor. Moreover, [START_REF] Fernandez | Prospective mathematics teachers' development of noticing in an online teacher education program[END_REF] and several other researchers (e.g. [START_REF] Klein | Improving feedback through online professional development[END_REF] agree that online systems "slow down the process of learning" (Clay, Silverman, & Fischer, 2012, p. 762) by giving teachers extra time to reflect on the situations, which allows them from different contexts to collaborate around instructional situations. The necessity for such environments has been felt even more during the COVID-19 pandemic. Taking into account the advantages of online environments for teachers' professional development and its particular need during the pandemic, in this paper, we report one aspect of a larger research project which aims to support middle school mathematics teachers' noticing skills of students' mathematical thinking through their involvement in an Online Professional Development (OPD) program.

Although we focus on various mathematical contents in the project, the specific domain of students' mathematical thinking considered in this paper is about analyzing and interpreting data which is the core process of the statistical reasoning (Jones et al., 2004). More specifically, this process includes "recognizing patterns and trends in the data and making inferences and predictions from data." (Jones et al. 2004, p.103). In this research paper, our aim is to examine to what extent a middle school mathematics teacher attends to and interpret students' inferences from a given data and how the teacher base their decisions on students' understandings. Moreover, we are interested in how the teacher's participation in the OPD could support the development of her noticing skills. In other words, the following research question guided the research study: How do a middle school mathematics teacher's noticing of students' thinking in the context of analyzing and interpreting data develop throughout her involvement in an online professional development program?

Alternative to the studies that use online systems for developing teachers' noticing skills (Fernandez, Lilinares, & Valls, 2012;[START_REF] Fernandez | Prospective mathematics teachers' development of noticing in an online teacher education program[END_REF], synchronous modes of communication were also used in the present study. Furthermore, in this research, we work with in-service teachers who have less than 15 years of experience. By this way, we have a chance to investigate novice teachers' development of noticing skills regarding students' understanding through collaboration with other colleagues from different schools across Turkey. Lastly, we particularly focus on the content of data analysis and interpretation, which gets relatively less attention in the available literature. Hence, this study may provide valuable information to the teacher noticing literature about the nature and development of middle school mathematics teachers' noticing skills in the context of data analysis.

METHOD

This study is part of a large research project, which adopt design-based research as its methodology [START_REF] Bakker | Design research in education[END_REF]. However, in the current study, we limited ourselves to a teacher, Aslı, as our case and examined how her noticing skills developed during her participation in the OPD.

Participants

The teacher Aslı was selected among the project participants who are 35 middle school mathematics teachers whose professional experience did not exceed 15 years and working in public schools in different cities of Turkey. We selected Aslı, who has 8 years of teaching experience, for this study because she was one of the very active teachers in the OPD. She tried to enhance collaborative discussion environment by challenging the other teachers through her questions and making comments to their opinions.

Data collection

The data of the study was collected through the basketball problem, shown in Figure 1 below. This item was adapted by [START_REF] Gökce | Examining Middle School Mathematics Teachers' Content and Pedagogical Content Knowledge of Statistical Reasoning[END_REF] from the work of [START_REF] Mcgatha | An analysis of students' initial statistical understanding: Developing a conjectured learning trajectory[END_REF]. In the problem, the scores of two basketball players were given during the last 10 games and the students were asked to decide which player the coach should choose. To decide, students should analyze the given data by using appropriate measures of central tendency and variability, and interpret the results in the given context. Research suggests two strategies that students generally use while answering such comparison questions. Therefore, student responses including those strategies are provided under the problem to examine how teachers notice them. As shown in Figure 1, Student 1 (S1) chose Barış by focusing on the mean score of each player. Her reasoning was partially correct since she ignored the variability of the given data. On the other hand, Student 2 (S2) also chose Barış by only focusing on the scores above a certain value, 15. Since he ignored the data less than 15, reasoning of S2 was not correct.

Figure 1. The basketball problem

To determine the teachers' noticing skills before the discussions held in the OPD, teachers were asked to answer three questions suggested by [START_REF] Jacobs | Professional Noticing of Children's Mathematical Thinking[END_REF] based on the given cases. Subsequently, the basketball problem was discussed in the OPD with all teachers and research team during 3 weeks period. To summarize the discussion in the OPD, a synchronous session was conducted by the facilitator, a member of the design team and researcher who has many research studies on statistical thinking. In that session, teachers were expected to ask any questions they have in their mind regarding the problem and state their opinions regarding the online asynchronous discussion held among the teachers. Moreover, the facilitator mentioned some big ideas, such as the importance of the context in interpreting the given data, that did not get enough attention in the discussion. Lastly, semi-structured interviews were conducted to get in-depth information regarding the change in teacher's noticing skills after asynchronous and synchronous discussions. Teacher Aslı's written answers about two students' strategies, participating teachers' comments in the asynchronous discussion held in the OPD and summary session, and Aslı's interview transcripts were constituted the data for this study.

Data Analysis

For the analysis of the data, the theoretical framework by [START_REF] Jacobs | Professional Noticing of Children's Mathematical Thinking[END_REF] was modified by taking into consideration the data collected and the Teacher Recognition Skills framework developed by [START_REF] Kilic | Preservice Mathematics Teachers' Noticing in Action and in Reflection[END_REF]. Accordingly, while teachers' attending skills was examined at four levels, the skills of interpreting students' understanding and deciding how to respond were examined at five levels. In particular, teachers' attending skills, the ability of teachers to identify the mathematical details in students' strategies, were categorized as No Attempt, Lack, Limited and Robust Evidence as different from Jacobs et al.'s (2010) framework. For interpreting skill, identification of students' understanding based on their strategies, one more category, Substantial Evidence was added to the categories of attention. In relation to the last skill, the teachers' ability to base their decisions on students' understandings, were not evaluated as lack, limited and robust. Instead, categories named as No Attempt, Ignorance, Questioning, Challenging and lastly Responding to child and Incorporating were used based on the data. Based on this framework, we firstly determined the noticing skills of teacher Aslı. Then, by examining the teachers' discussions in the OPD and the interview of Aslı, we tried to find evidence in the development of her noticing skills and how those developments occurred.

Findings

In this paper we aimed to examine the development of a teacher's noticing skills during her involvement in the OPD program. Below, teacher's responses and the changes in her ideas during and after the discussions in the OPD will be provided for each noticing skill, respectively.

The teacher's evolving expertise in attending and interpreting skill

The below response presents teacher's attention to the given students' strategies before the discussion in the OPD: Aslı:

For S1, mean is the best way while comparing two groups. For S2, if the range is too much among the given data, it is correct to use median and the data which are above the median. We can say that statistical reasoning of S2 was correct if we just evaluated the player Barış. However, if the range is not too much while comparing two groups, as in the two groups in this question, comparison using mean will be a better approach just like S1 did. (Pre-test)

In this response, teacher attended that S1 used mean while comparing two groups, but she could not notice that S1 did not consider the variability while interpreting the given data. In other words, since she could not attend all the details in S1's strategy, she evaluated her reasoning inaccurately. The teacher managed to attend S2's reasoning as not correct; however, she focused on a concept, median, which was not used by the student. For these reasons, the level of teacher attention to the given student strategies was coded as lack.

Aslı began the discussion in the OPD with the same arguments in the pre-test . Following her comments in system, the below discussion held among some teachers.

T1: …I believe that S2 chose Barış by considering the mode of the given data. Even though the answer of S2 is correct, I do not think that his reasoning is true. Aslı:

If S2 used the mode of the given data, should not he consider 18 [the mode of the data for Barış]? If S2 were to use mode, did not he say that 18 repeated much in the given situation; thus, I choose Barış? I thought that since the sixth data is 15 when all of them are ordered, S2 focused on the median of the given data. T1:

I stated that S2 used the mode by grouping all the data, which are above 15. I do not think that we can take 15 as median since there are 10 games in total. T3:

Differently from above discussion, I think that neither S1 nor S2 reasoned correctly about this question. S1's reasoning is missing since she does not know that it is not enough to just look at one central tendency when the averages are so close to each other. In here, she should look at the ranges of the given data additionally. I could not understand which statistical concept S2 used with his statement of '15 and more points'. I do not think that he used the mode or median as mentioned in above comments.

Since teachers could not agree on evaluating S2's strategy, facilitator posed some questions to let teachers think and interpret the meaning of mode and median for the given data sets. After a long discussion in the system, teachers agreed that S2 did not use the concepts of mode or median, but simpler reasoning. In other words, they agreed that he only concentrated on the highest points in the given data. The following comment of Aslı in the discussion indicated that teacher's expertise in attending S2's strategy improved after this fruitful discussion.

Aslı: I believe that to organize data by grouping (She means to group the data with respect to highest and lowest scores) is correct to understand the nature/tendency of the data. However, it is missing. This is also valid for S2. While thinking the scores which are above 15, the data which are below 15 were lost. (Discussion)

When the data was analyzed in terms of interpreting skills, again her response indicated the lack of evidence in interpreting the students' mathematical understanding. The following response was received before the discussion in the OPD when Aslı was asked to interpret the students' mathematical understanding by considering the given students' strategies:

Aslı: For S1, the mean; that is, all values in the data group, is important because it is affected by the change of each data. For S2, on the other hand, if the difference among the data is high, to look at the median and the data above the median is a more accurate way to interpret the data. Because of the skewed distribution of the data , looking at the median is the most useful method for S2. (Pre-test)

Although teacher thought that S1considered the importance of all data, S1's solution is not enough to make this claim. For S2's strategy, the teacher again focused on the concept of median as she attended in the first question. She argued that S2 knew to use the median when the data set is skewed; which indeed cannot be concluded from the student's strategy. Furthermore, the teacher did not mention variability or representativeness while interpreting the students' mathematical understanding.

When teachers were asked to interpret students' mathematical understanding in the given student responses in the OPD, Aslı's comment was as follows:

Aslı: S1 used only the mean when comparing the data groups, but did not include the concept of range, she should interpret these two concepts together. In S2's interpretation, grouping (She means to group the data with respect to highest and lowest scores) can be used to organize the data, but it is insufficient on its own for data analysis. It is seen that S2 has no knowledge of the concepts of mean, median, and range when looking at the solution. I think that the starting points of both students' reasoning are correct, but they are insufficient. To interpret the data with a single concept will not be sufficient in data analysis. (Discussion)

Most of the other teachers agreed with Aslı's comment in the following parts of the discussion. It was observed that Aslı mentioned the concepts of mean and range, which are necessary to interpret the mathematical understanding of the S1's strategy. For S2, teacher gave up her idea of median. Since teachers did not focus on the meaning that S1 attributed to the mean, facilitator let teachers discuss this through her questions. After this discussion, she interpreted S1's strategy by referring to the concept of representativeness during the interview while she did not mention it in her first comment in the OPD. Lastly, Aslı's following expression regarding her own development gave us one more evidence that discussions nurtured her interpreting skills of students' understanding. "The discussions in the OPD enabled us to brainstorm. It made me realize some new ideas. In particular, I became aware of the concept of range for this question".

The teacher's expertise in deciding skill

Before the discussion in the OPD, Aslı responded her next moves as in the following:

Aslı: To show that both students' reasoning can differ with respect to the different situations, I give two data sets whose ranges are high and low and I ask which student is more successful. For example, Case 1: Ayşe's savings during five months: 10, 50, 90, 90, 100 (Mean=68) Case 2: Ali's savings during five months: 70, 80, 85, 85, 100 (Mean=84) If we look at the mean, we interpret correctly and say that Ali is more successful. I can relate this question with the given problem situation by stating that it is not correct to say that Ayşe is more successful since she has 3 data which are above 90. (Pre-test)

During the interview, Aslı stated that she would use the above example for S2. Since Aslı believed that the use of mean is better for this problem, she asked such a question to S2 so that student can realize that he should use mean while answering this question. In other words, her aim of asking this question is to lead S2 to the teacher's correct answer. In addition, Aslı added some questions to understand both students' strategies better such as "Why did you use 15 in your answer?" or "Why did you use the mean?". For these reasons, the level of her expertise in deciding how to respond based on the given students' strategies were determined as questioning.

On the other hand, Aslı began the asynchronous discussion held in the OPD with the argument below: Aslı:

S1 should feel the need to look at the range while interpreting the data. So, for this purpose, I would ask which players she would choose if the mean of them were equal. Similarly, I would ask S2 which players he would choose if the number of data, which are above 15, were equal. My aim in this part is S1 and S2 should notice that their reasoning is missing and only one method is not enough while analyzing data. (Discussion)

Different from her first comment in the pre-test, Aslı asked some probing questions for S1 to let her notice that she should consider the variability of the given data. In the same way, the questions for S2 were to guide him to consider the data which are below 15. In other words, the teacher started challenging students, a higher level in the framework, to make them realize that their reasoning is underdeveloped. Although Aslı did not provide any explanation regarding how to guide S2 to the use of average in her first comment in the OPD, she offered some ways for this aim after the facilitator's question. During the discussion in the OPD, most of the teachers agreed with the Aslı's first comment.

On the other hand, during the interview, when she was asked about her next problems to the students after the basketball problem, she stated that "I again present a problem that students can provide different comments…Indeed, I would change the context. However, the characteristics of the data would be the same so that to calculate just the mean would not be sufficient." With this description, we can say that the teacher thought providing new contexts for students. However, her aim seems to make students practice the ideas discussed in the given problem, not to emphasize the importance of the context, which was discussed by the teachers and facilitator during the synchronous session.

Discussion

The case of Aslı was presented to illustrate that the OPD can help teachers in their development of noticing skills. At the beginning, the teacher had difficulties in each interrelated skill of noticing. However, the findings showed enhancement in her noticing of students' thinking in the context of data analysis throughout her participation in the OPD. During the discussion, teachers had a chance of observing different ideas, posing questions, and making suggestions to each other. Also, the facilitator let teachers focus on the important concepts through her prompts. All these interactions might direct the teacher's focus on students' mathematical thinking deeply as consistent with the study of [START_REF] Fernandez | Prospective mathematics teachers' development of noticing in an online teacher education program[END_REF]. In other words, it can be inferred that the collaboration among the teachers and the facilitator throughout the discussion might allow the teacher to attend the details of the given strategies and interpret students' understanding by covering all the essential concepts such as variability and representativeness. Moreover, the teachers were provided with two students' strategies and the solution of S2 was particularly challenging and unfamiliar for the teachers. This could have led the discussion environment to be more productive, which in turn might have increased the collaboration among teachers.

Another factor contributing to the development in the teacher's noticing skills could be the asynchronous nature of the OPD. [START_REF] Clay | Unpacking online asynchronous collaboration in mathematics teacher education[END_REF] argues this mode of communication slows down the process of learning, which could enable teachers spent more time to consider and revise their ideas (Fernandez et al., 2012;[START_REF] Klein | Improving feedback through online professional development[END_REF]. Indeed, this was shown by Aslı's one of the statements during the interview: "Discussions in the OPD provided us to think more on the students' answers." Thus, asynchronous modes of communication provided via the OPD might play an effective role in developing the teacher's attention to the given task; leading fruitful communication among teachers which in turn enhance her noticing skills.

Although there was clear improvement in attending and interpreting skills of the teacher, deciding how to respond based on students' understandings was more difficult for her. Preceding the discussion, the teacher's questions were like directing students to the correct answer which was also observed by [START_REF] Klein | Improving feedback through online professional development[END_REF]. Even though the teacher suggested asking probing questions to challenge students throughout the discussion, she had difficulty in extending students' understanding to a further point after the discussions. Although some new ideas were presented by the facilitator during the synchronous session such as the use of graphical representations of the data presented in the basketball problem and the importance of context for interpreting data, these ideas were not reflected in the teacher's responses during the interview. Therefore, in future research, we suggest carrying such specific ideas into the OPD to enable teachers create more productive discussion environment for the deciding skill. Lastly, it would be good to examine how the teachers' evolving expertise is reflected into her instructional practices in further research studies.

Acknowledgement

This research has been supported by the Scientific and Technological Research Council (TUBITAK), Turkey.


Sibel Kazak
email: skazak@pau.edu.tr

Preservice mathematics teachers' experiences of acquiring and organizing image-based data

Keywords: Statistical inquiry, data acquiring, data organizing, image-based data, categorical data

There is a growing interest in engagement with non-traditional data generated from a variety of sources, such as sensory devices, smartphones/watches, and social media tools. This paper reports on experiences of a group of pre-service mathematics teachers in acquiring and organizing imagebased data to analyze categorical data as part of a course assignment. To identify the actions preservice teachers took to transform observations of butterflies' photos to cases to answer statistical questions related to categorical data, the data from retrospective interviews and artifacts generated during the data exploration and organization were analyzed. The findings suggest that data handling and structuring became an important component of the statistical investigation before formulating statistical questions. While the use of hierarchical tables in structuring data by hand appeared to be intuitive, it could be challenging to use such a structure already built in a data organizing tool.

Introduction

Understanding and reasoning with data become increasingly essential part of our everyday lives as we need to handle information related to global issues on health, environment and so on, such as the COVID19 pandemic and wild fires due to extreme weather conditions. The nature of data also evolves with the advanced digital technologies. Large quantities and different forms of data are generated from a variety of sources, such as sensory devices, smartphones/watches, social media tools etc., every day. To analyze and make predictions from these data require new skills. Therefore, an important goal of statistics education is to develop necessary data skills starting from early school years.

Statistics is typically taught as part of school mathematics curriculum and there has been a shift from simply computing numerical and graphical representations of data to an inquiry-based approach (Watson, Jones, & Pratt, 2013). In this approach to the teaching of statistics, the focus is on the statistical problem-solving process [START_REF] Franklin | Guidelines for Assessment and Instruction in Statistics Education (GAISE) Report: A Pre-K-12 Curriculum Framework[END_REF][START_REF] Bargagliotti | Pre-K-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) report II[END_REF]. According to the Pre-K-12 Guidelines for Assessment and Instruction in Statistics Education II (GAISE II): A Framework for Statistics and Data Science Education, the purpose of this process is "to collect and analyze data to answer statistical investigative questions" (Bargagliotti et al., 2020, p. 13). As seen in Figure 1, it has four components that are interlinked: (1) Formulate statistical investigative questions, (2) Collect/consider the data, (3) Analyze the data, and (4) Interpret the results. Traditional statistical education focuses on data from random samples and making inferences from a sample to an unknown population. In today's digital age, however, data come in various forms (big, messy, unstructured data, repurposed data, image/text/sound-based data etc.) and do not necessarily comply with the structure required for statistical inference in the traditional sense [START_REF] Gould | An analysis of secondary teachers' reasoning with participatory sensing data[END_REF]. Along with the emerging field of data science, statistics education tends to focus more on the engagement with such non-traditional data sets.

Image-based data, i.e., the use of photographs as data, becomes increasingly common in daily encounters. The GAISE II report [START_REF] Bargagliotti | Pre-K-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) report II[END_REF] acknowledges the use of this kind of nontraditional data as part of the statistical problem-solving process and helping students to make sense of these non-traditional data. As an example, Bargagliotti et al. use data from Dollar Street website as part of the Gapminder Foundation project (www.gapminder.org/dollar-street). The Dollar Street website displays the world as a street ordered by monthly income per person in the family from different countries and continents, and currently uses 43685 photos of 422 families in 66 different countries. Using these image-based data, students are expected to investigate "How are people's concepts of family and living spaces similar or different across the world?" (Bargagliotti et al., 2020, p. 63). The guidelines in the GAISE II report offers some instructional ideas with regard to the use of these image-based data through each four components of the statistical problem-solving process mentioned above. [START_REF] Bargagliotti | Pre-K-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) report II[END_REF] suggest that this kind of non-traditional, multivariate data sets requires a great amount of data exploration time to make observations and wonderings to analyze data. Engagement with other types of non-traditional (large and complex) data from secondary data sources, such as participatory sensing data [START_REF] Gould | An analysis of secondary teachers' reasoning with participatory sensing data[END_REF] and public data sets [START_REF] Wilkerson | Exploring variability during data preparation: a way to connect data, chance, and context when working with complex public datasets[END_REF]), also appears to require more emphasis on considering data and data preparation phases in statistical investigation process. As seen in these studies, such existing multivariate data sets are constructed and made publicly available by others with a particular purpose and investigators often repurpose them to explore new questions. To do so, data handling and structuring becomes an important part of statistical investigations.

The Collect/Consider the Data component of the statistical problem-solving process involves recording/acquiring, measuring and organizing the data to answer statistical questions with the acknowledgment of variability in data [START_REF] Bargagliotti | Pre-K-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) report II[END_REF]. [START_REF] Konold | Modeling as a core component of structuring data[END_REF] call attention to this important process of transforming observations into data and focus on table format as a typical means of data recording and structuring either by hand or using software. Two common table formats in statistics are case-data table and summary table as called by [START_REF] Konold | Modeling as a core component of structuring data[END_REF]. While case-data tables are mainly used for collecting and storing raw data, summary tables usually include only some of the information gathered and are organized in a way to compare groups or detect trends in the data. The data analysis software, such as TinkerPlots [START_REF] Konold | TinkerPlots 2.0: Dynamic data exploration[END_REF] and CODAP (https://codap.concord.org/), uses attribute-based structure for recording data where each column includes a variable and each row holds one observation, called a 'case', as described by [START_REF] Konold | Modeling as a core component of structuring data[END_REF]. However, entering the information collected to organize the data in such a format can be challenging for the learners. Konold et al. point out the need for thinking of data based on attributes which requires considering properties of the observations along with their values case by case. Their study revealed that students and adults tended to create two types of case tables, flat and hierarchical, when they were asked to construct data sheets to record and organize the data by hand from the pictures showing traffic along two road segments (including information about the vehicle type, speed, direction etc.) in a given time and date. Compared to the flat tables (attributes as columns and cases as rows) as seen in most data analysis software, the hierarchical tables include the cases at more than one level created by nested structure. This type of hierarchical structure of observations is available in CODAP and allows exploration of multilevel data sets.

Given the scarce of research on how students/adults come to record and organize data, especially with non-traditional image-based data, as part of a statistical problem-solving process, the aim of this paper is to present a case of three pre-service mathematics teachers who chose to use photos of butterflies as data to analyze categorical data. More specifically, the following question will be investigated in this exploratory study: What actions did pre-service teachers take to transform observations of butterflies' photos to cases to answer statistical questions related to categorical data?

Method

As part of 3 rd year course, Teaching of Statistics and Probability, in the mathematics education program, the pre-service teachers were given a group assignment to design a statistical investigation activity that is aligned with the learning outcomes in the middle school mathematics curriculum. Two of the groups were expected to focus on analyzing categorical data and comparing two or three groups using existing data sets available in publicly available internet sources. One of these groups, including Arya, Yelda and Hale (pseudonyms), chose image-based data using "Kelebek-Türk" (Butterflies-Turk) group's website (https://www.kelebek-turk.com/) while the other group designed their data investigation activity involving data collected through survey questions on readings books.

Due to the uniqueness of transforming image-based data into a statistical analysis of categorical data with group comparisons, an interview was conducted with these three pre-service teachers to gain insights into their approaches when acquiring and organizing information from photographs of butterflies. Other artifacts, such as the documents created during the data exploration and organization, were also collected for analysis to answer the research question. Recordings of group interview and the student artifacts were analyzed using the perspectives on collect data/consider data suggested by [START_REF] Bargagliotti | Pre-K-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) report II[END_REF] and [START_REF] Konold | Modeling as a core component of structuring data[END_REF].

Findings

Preservice teachers began with considering possible data appropriate for analyzing categorical data with comparing groups, which was given in the course assignment. Due to the difficulty in finding at least three categorical variables in data sets using publicly available data sources, such as Turkish Statistical Institution website, they stated that some of the data explored during the course, such as the ladybug data (Bargagliotti et al., 2020, p. 32), inspired them to consider similar data about other animals. Yelda came up with the idea of searching butterfly data on the internet and found the group of Kelebek-Türk that provides photographs and observational data provided by their members from different age and occupation groups. After reading the "About us" information on the website, the pre-service teachers got their initial information about how the data were collected. Then they decided to use some of these photographs to construct their own observations involving categorical data.

According to the Kelebek-Türk website, there are 416 types of butterflies under nine different families in Turkey and their members have photographed 393 of them. As seen in Figure 2, there are nine butterfly families with different number of types, such as the Family Hesperiidae has 43 types, observed in Turkey and Figure 3 shows a sample of photos of different butterfly types belonging to the Family Hesperiidae. Following the exploration of the information given in these pages, the pre-service teachers decided to focus on two families with a similar number of butterfly types (the Family Hesperiidae with 43 types and the Family Pieridae with 39 types) for comparison purposes given in the assignment. To collect information about these two families, they then began to click on each photo as seen in Figure 3. For example, Figure 4 shows various information about Muschampia poggei from the Family Hesperiidae, including numerical (frequencies of photographs by cities), geographic (distributions of butterflies across Turkey), photographic (captured images of butterflies with/without identified gender), tabular (months they were seen) and graphical (distribution of altitudes seen by months) representations. From this information, they selected to focus on the image of butterfly with unidentified gender (since not all types had gender information) to generate attributes, such as wing color and wing appearance which were considered as distinguishing characters of butterflies, and the seasons the butterflies seen, which involved grouping of months data given on the website. After this extensive exploration on the website, the pre-service teachers began to make an ordered list for each type of butterflies in both families, including the name of the butterfly, its photo, and the season it is seen, on a Word document. As part of the course assignment, they needed to use CODAP to analyze the data to answer questions considered in the Formulate statistical investigative questions phase of the statistical problem-solving process in Figure 1. Three questions emerged during this exploration: 1) What colors are the wings of the Family Hesperiidae and the Family Pieridae? 2) How do the appearances of the wings of the Family Hesperiidae and the Family Pieridae look like? 3) What seasons do the Family Hesperiidae and the Family Pieridae tend to be seen? Then they constructed a table, as seen in Figure 5, displaying the family name, type name, wing color, wing appearance, and season seen in columns respectively and case values in rows. Organizing the data as shown in Figure 5 made entering them into CODAP table relatively easier due to its case-based structure. One difficulty the pre-service teachers encountered in this process was having more than one value for season variable for some cases, such as Pyrgus serratulae seen in both spring and summer. Therefore, they chose to create two different data sets, one including 70 cases of butterflies in two families with family names, wing color and wing appearance variables (Figure 6, table on the left) and the other including 120 cases with season variable for two families (Figure 6, table on the right). Due to the repeated cases for each season a butterfly seen, the number of cases was increased in the second data set. The pre-service teachers decided to separate multiple seasons since they wanted to answer their third question: What seasons do the Family Hesperiidae and the Family Pieridae tend to be seen? In either of the data sets, they also did not include type name which was displayed in the table in Figure 5 as they considered that having the type names would not allow to see two categories (the families) which were compared in the data analysis.

Concluding remarks

This paper presented insights from a group of pre-service teachers' experience of acquiring and organizing image-based data to analyze categorical data by comparing two groups as part of a course assignment. As pointed out by [START_REF] Bargagliotti | Pre-K-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) report II[END_REF], the pre-service teachers spent a great deal of time to explore the non-traditional data involving photographs and other types of data collected by others for specific purposes as mentioned in the Kelebek-Türk website. In accord with the findings of [START_REF] Gould | An analysis of secondary teachers' reasoning with participatory sensing data[END_REF] and [START_REF] Wilkerson | Exploring variability during data preparation: a way to connect data, chance, and context when working with complex public datasets[END_REF], data handling and structuring became an important component of the statistical investigation before formulating statistical questions. They needed to consider attributes/variables and criteria for identifying categoric values for them, such as assigning values, plain or pattern, for wing appearance based on the dots or lines seen in the photos of butterflies. The pre-service teachers did not seem to have any issue with preparing the data for the analysis in CODAP as they were able to consider the data as attribute-based. This could be due to their prior experiences with using CODAP for data analysis during the Statistics course taken in the previous semester. However, transforming the initial table created by hand (Figure 5) into a single data set table in CODAP was challenging for them. Since their initial table included subsets of data within the season variable, the flat table did not work. Instead, they needed to create hierarchical data sets, which is possible in CODAP, but this feature was not used in the course before. It was also evident in the work of [START_REF] Konold | Modeling as a core component of structuring data[END_REF] that the use of hierarchical tables in structuring data by hand was more intuitive, but it could be challenging to use such a structure that is already built in CODAP environment.

Figure 1 :

 1 Figure 1: Statistical problem-solving process (reproduced from Bargagliotti et al., 2020)

Figure 2 :Figure 3 :

 23 Figure 2: Screenshot of the website displaying the information about all nine different butterfly families

Figure 4 :

 4 Figure 4: Screenshot of the website displaying various information about Muschampia poggei from the Family Hesperiidae

Figure 5 :

 5 Figure 5: Part of the table created for organizing the data

Figure 6 :

 6 Figure 6: The data set tables created in CODAP


Ruggero G Pensa
email: ruggero.pensa@isti.cnr.it

Jean-François Boulicaut

Constrained Co-clustering of Gene Expression Data

. We illustrate the added value of our approach in two applications on gene expression data.

Introduction

In many application domains, the data analyst has to consider possibly large data sets that record numerical values of given properties for given objects (say objects × f eatures matrices). From that perspective, the popular context of basket data analysis is the special case where boolean values record whether a product belongs to a transaction or not. In this paper, we will consider gene expression data sets that record gene expression values for given genes in given biological samples (see, e.g., microarray data analysis in [START_REF] Arbeitman | Gene expression during the life cycle of drosophila melanogaster[END_REF]). A toy example data set is given in Fig. 1. In such a data set X r , properties may denote biological samples and each object may be associated to one particular gene. For example, in X r , one would say that the gene expression value for Gene 2 in Experiment 3 is 5.

Exploratory data analysis processes often make use of clustering techniques to get insights about global patterns within the data, i.e., to propose partitions of objean-francois.boulicaut@insa-lyon.fr jects and/or of properties such that a grouping quality measure is optimized. Many clustering methods can compute partitions but suffer from the lack of explicit cluster characterization. This has motivated the research on conceptual clustering, e.g., the co-clustering approaches described in [START_REF] Robardet | Efficient local search in conceptual clustering[END_REF][START_REF] Dhillon | Information-theoretic co-clustering[END_REF][START_REF] Banerjee | A generalized maximum entropy approach to bregman co-clustering and matrix approximation[END_REF]. The objective of coclustering is to compute co-clusters that are associations of (possibly overlapping) sets of objects with sets of properties. A co-clustering algorithm computes simultaneously linked partitions on both row and column dimensions. An example of a bi-partition in X r would be {{1, 2, 3, 4, 5}, {6, 7}} for objects, {{1, 4, 5}, {2, 3}} for properties. This co-clustering indicates that the characterization of objects from {1, 2, 3, 4, 5} is that they tend to share similar values from {1, 4, 5}. Also, properties in {2, 3} can be used to characterize objects in {6, 7}. Co-clustering has been well studied in the context of gene expression data analysis because it provides valuable information about putative regulation mechanisms and biological functions [START_REF] Cheng | Biclustering of expression data[END_REF][START_REF] Madeira | Biclustering algorithms for biological data analysis: A survey[END_REF]. Intuitively, a co-cluster extracted from a gene expression matrix denotes a set of genes with similar expression profiles along its associated set of biological samples.

We are interested in new co-clustering methods for enforcing the relevancy of computed bi-partitions in general and their application to gene expression data analysis in particular. Given a (co-)clustering algorithm, the analyst has generally a weak control on the clusters he/she obtains. Typically, he/she can decide for ad-hoc parameter settings which are quite operational and conceptually far from the declarative specification of desired properties. A co-clustering algorithm tries to optimize an objective function (e.g., Goodman-Kruskal's τ coefficient in [START_REF] Robardet | Efficient local search in conceptual clustering[END_REF] or the loss of mutual information in [START_REF] Dhillon | Information-theoretic co-clustering[END_REF]) but it may also ensure that some user-defined constraints are satisfied (e.g., the fact that some objects and/or properties have to be together or not). Enforcing constraints can lead to lower values for the objective functions, and it is clear that combining objective function optimization and the satisfaction of other user-defined constraints is challenging.

The last 5 years, several researchers have studied mono-dimensional constrained clustering for simple types of user-defined constraints, mainly the so-called must-link and cannot-link constraints [START_REF] Wagstaff | Constrained k-means clustering with background knowledge[END_REF][START_REF] Basu | Semisupervised clustering by seeding[END_REF][START_REF] Klein | From instance-level constraints to space-level constraints: Making the most of prior knowledge in data clustering[END_REF][START_REF] Basu | A probabilistic framework for semi-supervised clustering[END_REF][START_REF] Davidson | Clustering with constraints: Feasibility issues and the k-means algorithm[END_REF][START_REF] Davidson | Agglomerative hierarchical clustering with constraints: Theoretical and empirical results[END_REF][START_REF]Constrained Clustering: Advances in Algorithms, Theory and Applications[END_REF]. To the best of our knowledge, constrained coclustering has been rarely studied. In [START_REF] Pensa | Towards constrained co-clustering in ordered 0/1 data sets[END_REF][START_REF] Pensa | Constraint-driven co-clustering of 0/1 data[END_REF], we have studied co-cluster discovery when at least one of the dimensions is ordered and when interval constraints are defined w.r.t. orders. A typical application concerns kinetic gene expression data analysis. In this case, objects denote gene expression level measurements performed for successive time points. For an organism like Plasmodium Falciparum [START_REF] Bozdech | The transcriptome of the intraerythrocytic developmental cycle of plasmodium falciparum[END_REF], we see that during its life cycle, groups of genes are activated and then inhibited, being somehow characteristic of development stages. Using interval constraint can support the discovery of such groups from experimental data.

In this paper, we introduce our constraint-based coclustering approach which is quite different from the one in [START_REF] Pensa | Towards constrained co-clustering in ordered 0/1 data sets[END_REF] extended in [START_REF] Pensa | Constraint-driven co-clustering of 0/1 data[END_REF]. First, these papers concern only 0/1 data mining. More importantly, we propose here to work directly on the data, i.e., without any postprocessing of collections of local patterns that have to be computed beforehand (See Section 2). Our method builds a bi-partition that satisfies the user-defined constraints while optimizing the objective function introduced in [START_REF] Cho | Minimum sum-squared residue co-clustering of gene expression data[END_REF], namely the sum-squared residues.

The rest of the paper is organized as follows. Section 2 discusses the related work. Section 3 is dedicated to the problem setting for constrained coclustering. Section 4 provides details about our coclustering algorithms. Section 5 is an empirical study about the added-value of our framework on two gene expression data set analysis. Section 6 briefly concludes.

Related work

Constrained co-clustering is a new approach to gene expression data analysis. To the best of our knowledge, only our previous work [START_REF] Pensa | Towards constrained co-clustering in ordered 0/1 data sets[END_REF] has addressed the problem of co-clustering under user-defined constraints. We consider that optimizing the objective function and enforcing the number of co-clusters are more or less implicit constraints. In other terms, we would say that an algorithm like CoCluster [START_REF] Dhillon | Information-theoretic co-clustering[END_REF] just performs co-clustering and not constrained co-clustering.

In [START_REF] Pensa | Towards constrained co-clustering in ordered 0/1 data sets[END_REF][START_REF] Pensa | Constraint-driven co-clustering of 0/1 data[END_REF], we have proposed a local-to-global approach to build bi-partitions under interval constraints. This is performed by postprocessing collections of local patterns (e.g., closed sets) extracted from 0/1 data sets. The basic idea is to translate the interval constraint into a relaxation which can be enforced in the collection of local pattern. Then, it uses a k-means-based approach to obtain a partition of local patterns. Finally, such a partition can be post-processed to determine the coclustering structure over the data. We also suggested possibilities to process co-clustering counterparts of the popular must-link and cannot-link constraints for standard clustering. The main difference of our contribution w.r.t. this previous work, is that here we compute coclustering directly by alternatively computing clusters on columns and rows w.r.t. a common objective function. A second difference is that our current proposal ensures, when needed, the satisfaction of the interval constraint on the computed bi-partition. This was not the case in the method in [START_REF] Pensa | Towards constrained co-clustering in ordered 0/1 data sets[END_REF][START_REF] Pensa | Constraint-driven co-clustering of 0/1 data[END_REF]. Third, our new approach works on numerical matrices: it is not limited to 0/1 data anymore. Last, but not least, we significantly improve must-link and cannot-link constraint processing over both sets of objects and attributes. Other related works can be identified in the areas of constrained clustering, and available co-clustering techniques.

Constrained clustering Constrained clustering is a relatively new field of research. It has been mainly studied as one approach to semi-supervised learning 1 . Semi-supervised clustering can support classification tasks when labeled data are limited and/or expensive to collect. A solution is to use the knowledge given by available labeled instances within a clustering algorithm. In [START_REF] Wagstaff | Constrained k-means clustering with background knowledge[END_REF], the authors have proposed a simple adaptation of k-means which enforces must-link and cannotlink constraints during the clustering process. [START_REF] Basu | Semisupervised clustering by seeding[END_REF] proposes a constraint-based clustering approach which uses labeled data during the initialization and clustering steps. An example of metric-based approach is presented in [START_REF] Klein | From instance-level constraints to space-level constraints: Making the most of prior knowledge in data clustering[END_REF], while [START_REF] Bilenko | Integrating constraints and metric learning in semi-supervised clustering[END_REF] integrates both constraint-based and metric-based approaches in a k-means-like algorithm. In [START_REF] Basu | A probabilistic framework for semi-supervised clustering[END_REF], the authors propose a probabilistic model for semi-supervised clustering, which also combines the two approaches. Other related work focuses on constraint feasibility on a k-means-like approach [START_REF] Davidson | Clustering with constraints: Feasibility issues and the k-means algorithm[END_REF], and on an agglomerative hierarchical clustering approach [START_REF] Davidson | Agglomerative hierarchical clustering with constraints: Theoretical and empirical results[END_REF]. In such a related work, the goal is to improve accuracy in classification when only few instances are labeled, whereas our goal is not to support prediction: we are working within an unsupervised framework. For us, constraints are used to specify user expectation (say subjective interestingness) and thus to improve the relevancy of extracted results. In the case of interval constraints, there are no pair-wise constraints (such as must-link and cannot-link), but the user has only to specify if he/she wants intervals or not, without knowing if a particular object x is in the same cluster than another object y. Moreover, notice that, for us, constraints can be applied both on objects and attributes.

Other applications of constrained clustering are the so-called sensor network and k-anonymity problems. In both applications, a possible solution is to find compact clusters containing a balanced number of objects. In [START_REF] Banerjee | On scaling up balanced clustering algorithms[END_REF][START_REF] Banerjee | Scalable clustering algorithms with balancing constraints[END_REF], the authors propose algorithms to discover balanced clusters. Recently, [START_REF] Ge | Constraintdriven clustering[END_REF] has proposed an algorithm which finds an a priori unspecified number of compact clusters under combinations of minimum significance constraints and minimum variance constraints. Similarly to these two approaches, our algorithms enable to define constraints which are more related to the shape of the clusters rather than to pairs of objects.

Co-clustering Many co-clustering methods have been developed, possibly dedicated to gene expression data analysis. Kluger et al. [START_REF] Kluger | Spectral biclustering of microarray data: coclustering genes and conditions[END_REF] propose a spectral co-clustering method. First, they perform an adequate normalization of the data set to accentuate co-clusters if they exist. Then, they consider that the correlation between two columns is better estimated by the expression level mean of each column w.r.t. a partition of the rows. The bi-partition is computed by the algebraic eigenvalue decomposition of the normalized matrix. Their algorithm critically depends on the normalization procedure. Dhillon et al. [START_REF] Dhillon | Information-theoretic co-clustering[END_REF] and Robardet et al. [START_REF] Robardet | Efficient local search in conceptual clustering[END_REF] have considered the two searched partitions as discrete random variables whose association must be maximized. Different measures can be used. Whereas Cocluster [START_REF] Dhillon | Information-theoretic co-clustering[END_REF] uses the loss in mutual information, Bi-Clust [START_REF] Robardet | Efficient local search in conceptual clustering[END_REF] uses Goodman-Kruskal's τ coefficient to evaluate the link strength between the two variables. In both algorithms, a local optimization method is used to optimized the measure by alternatively changing a partition when the other one is fixed. The main difference between these two approaches is that the τ measure is independent of the number of co-clusters and thus Bi-Clust can automatically determine the number of co-clusters.

Lazzeroni et al. [START_REF] Lazzeroni | Plaid models for gene expression data[END_REF] propose to consider each matrix value as a sum of variables. Each variable represents a particular phenomenon in the data and corresponds to a co-cluster. In each co-cluster, column or row values are linearly correlated. Then, the method consists in determining the model minimizing the Euclidean distance between the matrix and the modeled values. This method is similar to the eigenvalue decomposition used in [START_REF] Kluger | Spectral biclustering of microarray data: coclustering genes and conditions[END_REF] without the orthogonal constraint on the computed variables.

In the context of gene expression data analysis, several authors have considered the computation of potentially overlapping local patterns that they call biclusters (see [START_REF] Madeira | Biclustering algorithms for biological data analysis: A survey[END_REF] for a survey). Ihmels et al. [START_REF] Ihmels | Revealing modular organization in the yeast transcriptional network[END_REF] propose a simple algorithm which builds in two steps a single association called a bi-cluster starting from a column set. First, they consider that the rows having a high score (greater than a threshold on the normalized matrix) on these columns belong to the bicluster. Then, they use the same principle to increase the original column set. In [START_REF] Cheng | Biclustering of expression data[END_REF], Cheng et al. propose a co-clustering algorithm for numerical data. They define a bi-cluster as a subset of rows and subset of columns with a low mean squared residue. When the measure is equal to 0, the bi-cluster contains rows having the same value on the bi-cluster columns. When the measure is greater than 0, one can remove rows or columns to decrease the value. Thus the method consists in finding maximal size bi-clusters such that the measure is inferior to a threshold. Various heuristics can be used for this purpose. The same definition of residue is used in [START_REF] Cho | Minimum sum-squared residue co-clustering of gene expression data[END_REF] to define the objective function which is also used in our current proposal. Authors propose two different residue measure, and show that the one proposed by Cheng et al. fits better to gene expression data analysis. Then, they introduce their co-clustering algorithm which optimizes the sum-squared residues function. This approach has been the starting point for our contribution.

Recently, Banerjee et al. [START_REF] Banerjee | A generalized maximum entropy approach to bregman co-clustering and matrix approximation[END_REF], proposed a coclustering formulation that is based on matrix approximation. The approximation error is measured using a large class of loss functions called Bregman divergences. The authors introduce a meta-algorithm whose special cases include the algorithms from [START_REF] Dhillon | Information-theoretic co-clustering[END_REF] and [START_REF] Cho | Minimum sum-squared residue co-clustering of gene expression data[END_REF].

A constrained co-clustering setting

Let X ∈ R m×n denote a data matrix. In the rest of the paper, the data set to be mined is the matrix X, and we always talk about rows and columns instead of objects and properties. Let x ij be the element corresponding to row i and column j. For instance, x ij might contain the expression level of gene i in the experimental condition j. Let x i. and y .j denote the vectors associated to, respectively, row i and column j.

A co-clustering C k×l over X produces simultaneously a set of k × l co-clusters (a partition C r into k groups of rows associated to a partition C c into l groups of columns). To obtain a first quality criterion, we first try to optimize a certain objective function.

Definition 3.1. (optimization constraint) Let us assume an objective function

f (X, C k×l ), an optimiza- tion constraint c opt (f, X, C k×l ) is satisfied iff C k×l = argmin φ∈L C k×l f (X, φ) where L C k×l is the the collection of all possible co-clusterings.
Some examples of objective functions are the Goodman-Kruskal's τ coefficient and the loss of mutual information [START_REF] Robardet | Efficient local search in conceptual clustering[END_REF][START_REF] Dhillon | Information-theoretic co-clustering[END_REF]. In this paper, we use the sumsquared residue function introduced in [START_REF] Cho | Minimum sum-squared residue co-clustering of gene expression data[END_REF]. For computational feasibility reasons, co-clustering algorithms relax these optimization constraints, e.g., using local optimization heuristics. One can be interested in other kinds of constraints which are now defined. Definition 3.2. (must-link/cannot-link) If rows i a and i b (resp. columns j a and j b ) are involved in a must-link constraint, denoted c = (i a , i b ) (resp. c = (j a , j b )), they must be in the same cluster of C r = r 1 , . . . , r k (resp

C c = c 1 , . . . , c k ). If rows i a , i b (resp. columns j a and j b ) are involved in a cannot-link con- straint, denoted c = (i a , i b ) (resp. c = (j a , j b )), they can- not be in the same cluster of C r = r 1 , . . . , r k (resp C c = c 1 , . . . , c k ).
These forms of constraints have been studied, even in the semi-supervised clustering framework [START_REF] Bilenko | Integrating constraints and metric learning in semi-supervised clustering[END_REF]. We generalize them to be able to apply them both to row sets and column sets. In an expression matrix, we can now exploit the knowledge about genes and/or experimental conditions. For example, if we know that gene i a and gene i b have the same function (say F ) in the biological process, we can enforce a mustlink constraint between these two genes to privilege the search for co-clusters associating genes having this function F and then identify a transcription module which could support such a biological function. We could also add some cannot-link constraints to avoid associations between experimental conditions which we would like to separate.

Let us now assume that a real value s c (j) (resp. s r (i)) is associated to each column j (resp. row i). Then we have s r : {1, 2, . . . , m} → R and s c : {1, 2, . . . , n} → R. For instance, s c (j) (resp. s r (i)) could be a temporal or spatial measure related to j (resp. i). In microarray data, s c (j) might be the sampling time related to the DNA chip (say experiment) j. Another example would be to consider s r (i) as a measure of the absolute spacial position of a gene i in the DNA sequence of the studied organism. The two functions s r and s c enable to define an order over the set of columns and/or rows. Indeed, we say that j a j b iff s c (j a ) ≤ s c (j b ). In the rest of the paper, we say that, if a function s c exists, then all the elements j are ordered, i.e., ∀j a , j b s.t. j a < j b , s c (j a ) ≤ s c (j b ) (the same property holds for rows). It could also be interesting to search for co-clusters which are coherent with the order defined by functions s r and s c . For instance, if we are interested in the different stages of the cell development, and we want to discover those genes that are mainly involved in each stage, we will look for clusters which are contiguous w.r.t. time. For this purpose, we can enforce an interval constraint.

Definition 3.3. (interval constraint) If an order ( ) is defined over the column set (resp. row set), an interval constraint over this set, denoted c int (C c ), specifies that each cluster in C c has to be an interval:

∀c ∈ C c , if j a , j b ∈ c then ∀j c such that j a j c j b , j c ∈ c.
In general, the satisfaction of the must-link, cannotlink and interval constraints decreases the theoretical optimum of the objective function. We want a coclustering algorithm which is able to take into account such constraints while trying to optimize the retained objective function. Notice that the satisfaction of a conjunction of constraints c = , c = and c int is not always feasible. For instance, for three objects i 1 , i 2 , i 3 such that s(i 1 ) < s(i 2 ) < s(i 3 ), the conjunction c = (i 1 , i 3 ) ∧ c = (i 1 , i 2 ) ∧ c int (C r ) can never be satisfied, even though the sub-constraints of this conjunction do not cause any problem. In this paper, we assume that the processed conjunction of constraints is feasible. The constraint feasibility problem for both partitioning and hierarchical methods has been studied in [START_REF] Davidson | Clustering with constraints: Feasibility issues and the k-means algorithm[END_REF][START_REF] Davidson | Agglomerative hierarchical clustering with constraints: Theoretical and empirical results[END_REF].

4 Using sum-squared residues Our approach to constrained co-clustering is based on an iterative algorithm that minimizes the sum of squared residues. This objective function has been introduced in [START_REF] Cho | Minimum sum-squared residue co-clustering of gene expression data[END_REF] for unconstrained co-clustering applied to gene expression data. It is an adaptation of a measure designed for local pattern discovery [START_REF] Cheng | Biclustering of expression data[END_REF].

Given a data matrix X ∈ R m×n , we search for a partition of X into k row clusters, and l column clusters. Let I be the set of indices of the rows belonging to a row cluster, and J the set of indices of the columns belonging to a column cluster. The sub-matrix of X determined by I and J is called a co-cluster. We use the definition of residue in [START_REF] Cheng | Biclustering of expression data[END_REF]. Definition 4.1. (residue) Given an element x ij of X, the residue of x ij in the co-cluster defined by the sets of indices I and J, and whose respective cardinalities are 1 4 5 2 3 1 4 5 2 3 1 2 3 4 5

X 1 r =          
3 2 4 0 0 1 1 2 4 5 4 4 5 1 0 2 3 4 0 1 1 1 2 4 5 1 0 0 4 6 0 0 0 5 6

          1 2 3 4 5 6 7
X 2 r =           3
          1 3 4 2 5 6 7
X 3 r =           3 0 0 2 4 4 1 0 4 5 2 0 1 3 4 1 4 5 1 2 1 4 5 1 2 1 4 6 0 0 0 5 6 0 0           1 3 4 2 5 6 7
Figure 2: Three co-clustering results on X r (permutations on columns and rows to emphasize the co-clusters)

|I| and |J|, is given by

h ij = x ij -x Ij -x iJ + x IJ (4.1)
where x IJ = i∈I,j∈J xij |I|•|J| x Ij = i∈I xij |I| x iJ = j∈J xij |J| . Let H = [h ij ] ∈ R m×n
denote the matrix of residues computed using the previous definition. The objective function to be minimized is the sum of squared residues [START_REF] Cho | Minimum sum-squared residue co-clustering of gene expression data[END_REF] computed as follows:

||H|| 2 = I,J ||h IJ || 2 = I,J i∈I,j∈J h 2 ij (4.2)
We can rewrite the residue matrix in a more compact form. Let us introduce the matrices R ∈ R m×k and C ∈ R n×l which are defined as follows: each element (i, r)

(1 ≤ r ≤ k) of R is equal to m -1/2 r if i is in co-cluster r (m r is the number of rows in r), 0 otherwise. Each element (j, c) (1 ≤ c ≤ l) of the matrix C is equal to n -1/2 c
if j is in c (n c being the number of columns in c), 0 otherwise. The residue matrix becomes:

H = (I -RR T )X(I -CC T ) (4.3)
The proof of validity of this equation is given in [START_REF] Cho | Minimum sum-squared residue co-clustering of gene expression data[END_REF]. The authors first demonstrate that (RR T X) ij = x Ij , (XCC T ) ij = x iJ and (RR T XCC T ) ij = x IJ , before showing that (4.3) is true. They conclude that, if we consider the projection (I -RR T )X of the matrix X, then ||H|| 2 gives the objective function of k-means for this modified matrix.

Let us now consider our algorithmic contribution. Our approach uses the so-called "ping-pong" technique to process alternatively (applying a k-means method) columns and rows. Thus, matrix C is updated only after determining the nearest column cluster for each column (and similarly for rows). For that, we can decompose the objective function in terms of columns. Given X P = (I -RR T )X, X C = (I -RR T )XC, and XP = (I -RR T )XCC T = X C C T , we can rewrite the objective function as follows:

||X P -XP || 2 = l c=1 j∈Jc ||X P .j -XP .j || 2 = l c=1 j∈Jc ||X P .j -(X C C T ) .j || 2 = l c=1 j∈Jc ||X P .j -n 1/2 c X C .c || 2 .
In the same way, setting X P = X(I -CC T ), X R = R T X(I -CC T ), and XP = RR T X(I -CC T ) = RX R , we obtain the following decomposition in terms of rows:

||X P -XP || 2 = k r=1 j∈Ir ||X P i. -m 1/2 r X R r. || 2 .
Then, matrices X C and X R correspond to the cluster centroids for columns and rows respectively. Now we can introduce our constrained co-clustering algorithm. First, we give a version to solve the satisfaction problem for a conjunction of must-link and cannotlink constraints. Then, we introduce a version which processes the interval constraint. Finally, we sketch a possible strategy to integrate those two principles.

Satisfying must-link and cannot-link constraints

The transitivity of the must-link constraint is a well known property. We can then transform a set of must-link constraints over rows into a collection M r = M 1 , . . . , M N , where each M i is a set of rows involved by the same transitive closure of must-link constraints. Let us denote M c the same set built for columns and let C r and C c be the sets of cannot-link constraints for rows and columns respectively. ConsCoClust(X,k,l,M r ,M c ,C r ,C c ) Input Data matrix X, k, l, cannot-link sets C r and

C c , collections M r et M c Output Matrices R and C Initialize R et C ∆ = 1; τ = 10 -2 ||X|| 2 t = 0 obj t = ||(I -RR T )X(I -CC T )|| 2 while ∆ > τ t = t + 1 X C = (I -RR T )XC X P = (I -RR T )X foreach 1 ≤ j ≤ n L = ∅ if ∃M v ∈ M c t.q. j ∈ M v MLColumnAssign(X,l,L,M v ,C c ) else L = {1 ≤ c ≤ l | ∄j c | γ t [j c ] = c ∧c = (j, j c ) ∈ C c } γ t [j] = argmin c∈L ||X P .j -n -1/2 c X C .c || 2 Update C using γ X R = R T X(I -CC T ) X P = X(I -CC T ) foreach 1 ≤ i ≤ m if ∃M u ∈ M r t.q. i ∈ M u MLRowAsssign(X,k,L,M u ,C r ) K = ∅ else K = {1 ≤ r ≤ k | ∄i r | ρ t [i r ] = r ∧c = (i, i r ) ∈ C r } ρ t [i] = argmin r∈K ||X P i. -m -1/2 r X R r. || 2 Update R using ρ obj t = ||(I -RR T )X(I -CC T )|| 2 ∆ = |obj t -obj t-1 |
Algorithm 4.1 enables to co-cluster data when conjunctions of must-link et cannot-link constraints are given. It starts with some initialization (e.g., random initialization) of matrices C and R. During each iteration, the algorithm associates each column (resp. row) to the nearest column (resp. row) cluster which does not introduce any cannot-link violation. If a column (resp. row) is involved in a must-link constraint (see Algorithms 4.2 and 4.3), the algorithm associates the whole set of columns (resp. rows) involved in the transitive closure of this constraint to the column (resp. row) cluster such that the average distance is minimum, and controlling that there is no cannot-link constraint which is violated by this operation. Then the algorithm updates the matrix C (resp. R) following the assignment schema resulting from the previously described oper-ations. This process is iterated until the diminution of the objective value turns to be smaller than a userdefined threshold τ . Algorithm 4.2. Constrained column assignment.

MLColumnAssign(X,l,L,M v ,C c ) foreach j v ∈ M v L = L ∪ {1 ≤ c ≤ l | ∄j c | γ t [j c ] = c ∧c = (j v , j c ) ∈ C c } γ t [M v ] = argmin c∈L jv ∈Mv ||X P .j -n -1/2 c X C .c || 2 |Mv| Algorithm 4.3. Constrained row assignment. MLRowAssign(X,k,L,M u ,C r ) foreach i u ∈ M u K = K ∪ {1 ≤ r ≤ k | ∄i r | ρ t [i r ] = r ∧c = (i u , i r ) ∈ C r } ρ t [M u ] = argmin r∈K iu∈Mu ||X P i. -m -1/2 r X R r. || 2 |Mu|
Notice that the initialization step should not necessarily take into account constraints, since their satisfaction is ensured by the first iteration of the algorithm. A possible improvement would consist in using a best assignment criterion for objects involved in cannot-link constraints. Moreover, we know that satisfying a set of cannot-link constraints for a given number of clusters is NP-complete [START_REF] Davidson | Clustering with constraints: Feasibility issues and the k-means algorithm[END_REF].

4.2 Satisfying the interval constraint Algorithm 4.4 enables to solve the satisfaction problem for the interval constraint (the part concerning row assignment (⋆⋆) is omitted here). The initialization (⋆) of partitions interested by this constraint should produce a number l (resp. k) of intervals over columns (resp. rows). Then, the assignment process only considers the frontiers between intervals. More precisely, it first processes the left frontier, then the right frontier iteratively. A column (resp. row) can be assigned to the adjacent interval if the distance is smaller than the distance computed over its original interval. In this case, we continue processing the remaining columns (resp. rows). When the left frontier and the right frontier of an interval correspond to the same column (resp. row), the algorithm starts to process the next frontier. If there is no necessity to reassign the column (resp. row), the algorithm stops the current frontier processing and it skips to the following one. Notice that, contrary to [START_REF] Pensa | Towards constrained co-clustering in ordered 0/1 data sets[END_REF][START_REF] Pensa | Constraint-driven co-clustering of 0/1 data[END_REF], the satisfaction of the interval constraint on the computed bi-partition is here ensured.

We do not consider the combination of these two algorithms to process a conjunction of must-link, cannotlink and interval constraints. However, let us sketch research guidelines for this purpose. We first have to ensure that each set M r ∈ M r (or M c ∈ M c ) is an interval. For instance, for a set of objects {i 1 , i 2 , i 3 , i 4 , i 5 }, and a set M r = {i 2 , i 4 }, we should include object i 3 into M r because of the definition of intervals. Then, we need the initialization to produce a partition which takes into account the whole set of constraints (notice again that satisfying a conjunction of cannot-link constraints is a NP-complete problem). Finally, it is possible to reuse the strategy described by Algorithm 4.1 only on the frontiers, following the schema presented in Algorithm 4.4.

Notice that this approach is intrinsically different than applying existing mono-dimensional constraintbased clustering algorithms alternatively on row and column vectors. In fact, each column (row) reassignment step of the algorithm takes into account the previous row (column) reassignment step through the common objective function. As a consequence, constraints on one dimension potentially influence the partition on the other dimension.

A running example

According to the objective function given by Equation 4.3, an optimal co-clustering result for X r (see the toy example in Fig. 1), is given by matrix X 1 r in Fig. 2. If we enforce Object 1 and Object 2 to be in two different clusters by setting a cannot-link constraint c = (1, 2), then we obtain the results shown by matrix X 2 r . Notice that, even if Object 5 is not concerned by this constraint, in the final bi-partition it is clustered together with Object 2. In fact, Object and Object 5 share the same feature values.

If we set an interval constraint on the set of columns (namely c int (C c )), we obtain the co-cluster structure shown by X 3 r in Fig. 2. Notice that, though only the column set is constrained, the resulting object partition is also different from the one obtained when no constraint is given. This is a direct consequence of using an objective function which takes into account both object and attribute partitions.

Computational complexity

Let us first consider Algorithm 4.1. Computing (I -RR T )X(I -CC T ) only requires the number of operations needed to compute R T XC, i.e., kn(m + l). It gives O(N ) time complexity (when N = mn) under the reasonable hypothesis that k ≃ l << m ≃ n. Assigning columns and rows to the new clusters can be performed in O(N (k + l)) time at each iteration. The overall complexity of the algorithm is then O(N (k+l)t), where t is the total number of iterations to complete the co-clustering. Complexity of Algorithm 4.4 is trivially the same. In general, the fact that we only process the interval frontiers increase the performances during the assignment step. IntCoClust(X,k,l) Input Data matrix X, k and l, int r , int c Output Matrices R and C Initialize R, C, lef t, right;

(⋆) ∆ = 1; τ = 10 -2 ||X|| 2 t = 0 obj t = ||(I -RR T )X(I -CC T )|| 2 while ∆ > τ t = t + 1 X C = (I -RR T )XC X P = (I -RR T )X foreach 1 ≤ c ≤ l f inish = f alse while f inish = f alse ∧ right[c] > lef t[c] if c > 1 j = lef t[c] if ||X P .j -n -1/2 c-1 X C .c-1 || 2 < ||X P .j -n -1/2 c X C .c || 2 γ t [j] = c -1 lef t[c] = lef t[c] + 1 right[c -1] = right[c -1] + 1 else f inish = true f inish = f alse while f inish = f alse ∧ right[c] > lef t[c] if c < l j = right[c] if ||X P .j -n -1/2 c+1 X C .c+1 || 2 < ||X P .j -n -1/2 c X C .c || 2 γ t [j] = c + 1 lef t[c + 1] = lef t[c + 1] -1 right[c] = right[c] -1 else f inish = true Update C using γ X R = R T X(I -CC T ) X P = X(I -CC T ) (⋆⋆) Update R using ρ obj t = ||(I -RR T )X(I -CC T )|| 2 ∆ = |obj t -obj t-1 |

Experimental validation

We have studied the impact of our constraint-based co-clustering approach on two well-documented microarray data sets, plasmodium [START_REF] Bozdech | The transcriptome of the intraerythrocytic developmental cycle of plasmodium falciparum[END_REF] and drosophila [START_REF] Arbeitman | Gene expression during the life cycle of drosophila melanogaster[END_REF]. The first one concerns the transcriptome of the intraerythrocytic developmental cycle of Plasmodium Falciparum, i.e., a causative agent of human malaria [START_REF] Bozdech | The transcriptome of the intraerythrocytic developmental cycle of plasmodium falciparum[END_REF]. The data provide the expression profile of 3 719 genes in 46 biological samples. Each sample corresponds to a time point of the developmental cycle: it begins with merozoite invasion of the red blood cells, and it is divided into three main phases, the ring, trophozoite and schizont stages. These data, as well as functional groups descriptions, are available on ftp://ftp.camda.duke.edu/CAMDA04_DATASETS. Missing values (about 0.5%) were replaced by zeros.

The second data set concerns the gene expression of the Drosophila melanogaster during its life cycle [START_REF] Arbeitman | Gene expression during the life cycle of drosophila melanogaster[END_REF]. The expression levels of 3 944 genes are evaluated for 57 sequential time periods divided into embryonic, larval and pupal stages. It has been retrieved from http://genome-www5.stanford.edu2 . Missing values (less than 1%) were replaced by zeros.

Our algorithms are implemented in C, and all the experiments have been performed on a PC (Windows, Intel Core 2 Duo 2.00GHz, 2GB RAM). In all our experiments the value of the stopping parameter τ was set to 10 -4 ||X|| 2 . As the initial partitions were randomly generated, our algorithm has been executed 20 times for each group of constraints. Running time is about 60 seconds for plasmodium data, and 100 seconds for drosophila.

Results for must-link et cannot-link constraints

The first collection of experiments concerns the use of must-link and cannot-link constraints in two different situations. In the first one, we assume we know a certain amount of information about some functional groups of genes. We use it to set up a number of pairwise constraints, and then we measure the improvement in co-clustering results w.r.t. the unconstrained version of the same algorithm. In the second situation, we try to discover one fixed partition by using the co-cluster label to define some sets of pairwise constraints both on the gene set and the biological condition set.

Constraints on gene set

We studied must-link and cannot-link constraints processing only for the gene set. For this purpose, we used the plasmodium data set, for which we can take advantage of a certain amount of information about genes involved in the different developmental stages. In particular, we considered the cytoplasmic translation machinery (denoted CTM) group (159 genes), which is known to be active during the first phase of the life cycle of the bacterium, the merozoite invasion (denoted MI) group (87 genes) particularly active during the second phase, and the early ring transcripts (denoted ERT) group (34 genes), which charac-terizes the last stage of the developmental cycle. All these functional groups have been described in [START_REF] Bozdech | The transcriptome of the intraerythrocytic developmental cycle of plasmodium falciparum[END_REF]. We randomly selected 20 sets of constraints based on these well-described functional groups. Each set contains a variable number of constraints, and the amount of genes involved in each set is between 20% and 50% of the genes involved in the threes functional groups. For this experiment, we set k = 3 and l = 3 to be try to identify the three developmental stages of Plasmodium Falciparum. The results presented in Table 1 and 2 show for each functional group the amount of genes involved in each co-cluster. In general, co-clusters are better characterized when constraints are defined. The number of genes belonging to each characterizing group increase in the co-cluster involving experimental conditions which are known to have this function activated. The improvement is much more significant for the second group of genes (merozoite invasion), while the last group (early ring transcripts) seems to be less influenced by the exploitation of these forms of constraints.

CTM

Constraint on gene and condition sets We measured the impact of combining constraints over row sets and column sets on the plasmodium data. For this purpose, we selected a bi-partition among the unconstrained co-clustering results. In particular, we chose the co-clustering results with the minimum objective function value obtained at the end of the iterative process. This value is about 1.99×10 4 . Then, we generated 20 random sets of constraints involving gene and biological conditions. The number of genes involved in those constraints is about 5% to 10% of the total size of the gene set. For biological conditions, this number is about 15% to 25%. To evaluate the agreement between the selected bi-partition and the ones discovered by our constrained algorithm, we used the adjusted Rand index [START_REF] Hubert | Comparing partitions[END_REF]. If C = {C 1 . . . C z } is the partition built by where

|π(C)| = z k=1 |C k |(|C k | -1) 2
|π(P )| = z k=1 |P k |(|P k | -1) 2 .
Then, the maximum value for a is:

max(a) = 1 2 (|π(C)| + |π(P )|)
The agreement between C and P can be estimated as follows:

AR(C, P) = a -exp(a) max(a) -exp(a)
Notice that when AR(C, P) = 1, we have identical partitions.

Results obtained by using constraints have been compared with those obtained by the unconstrained algorithm. Table 3 summarizes this experiment. We can see that using constraints gives rise to an obvious improvement of the agreement between the two bipartitions (AR r and AR c are the agreement indexes measured on rows and columns respectively). As side effects, the average number of iterations needed by the algorithm to complete the co-clustering process is slightly smaller. Moreover, specifying constraints enables to improve the final value of the objective function. Indeed, the ||H|| 2 value decreases by about 2%.

Examples of co-clusters discovered by our constrained techniques are given in Fig. 3. Even if gene clusters contain more than 1000 genes, it remains possible to identify similar expression profiles in the plots which show groups of 100 randomly picked genes for each co-clusters. If we consider the group of 100 genes which are the most similar with the average co-cluster, the related plots show more coherent expression profiles.

Notice that, since the sets of constraints are randomly selected, these experiments include results achieved through constraints which can either positively influence the results or do not introduce any quality improvement. However, as average clustering results are good, we can conclude that our approach enables to obtain more relevant bi-partitions according to prior biological knowledge.

Results for the interval constraint

We evaluated the added value of the interval constraint by applying our algorithm to the drosophila data set. Here, our goal is to rediscover the three phases of the drosophila life cycle using, as unique information, the number of clusters (k = l = 3). We compared the adjusted Rand index for the constrained and unconstrained versions of our algorithm and for a collection of 20 randomly initialized runs. The results (see Table 4), show that using an interval constraint enables to find more accurately the three stages of the drosophila life cycle (the measured improvement for the adjusted Rand index is about 85%). Moreover, the number of iterations needed to complete the coclustering process is considerably smaller than the one obtained when an interval constraint is enforced. Notice that the final value of the objective function for the unconstrained version of the algorithm is better than for the constrained version. It means that the structure which our algorithm is able to discover is unlikely to be the global optimum for this data set. Despite of this, the unconstrained algorithm has never managed to find intervals.

In Fig. 4, we plotted the average behavior of the objective function value w.r.t. iterations. For this type of data set, an interval-based initialization step gives rise to a smaller initial objective function. Interestingly, the objective function value for the unconstrained coclustering algorithm starts to be better after the first iteration. Afterwards, the difference between the two curves is just an offset, while the two convergence speeds are quite similar. This example illustrates the tradeoff between the objective function optimization and the constraint satisfaction processes. We can comment such a process considering the metaphor of the "tug of war" game. When the only competitor is the objective In order to measure the impact of the initialization method on the iterative behavior of the algorithm, we measured all the performances parameters already used for the previous comparison (see Table 4). Initializing the column partitions with a set of interval improve the average adjusted Rand index value over 20 runs, but this value is still far from the one achieved by the constrained version. The number of iteration is the only performance index which improves w.r.t. both unconstrained and constrained algorithms. Notice that, even if a constraint-oriented initialization has been used, none of the 20 executions has been able to find intervals. Finally, Fig. 5 illustrates that, after an improvement in the initial objective function, there are no significant differences between the totaly unconstrained algorithm and the interval-initialized one. It means that the active constraint process introduced within Algorithm 4.4 is critical: a constraint-oriented initialization is not sufficient to satisfy the interval constraint.

Conclusion

Co-clustering is an interesting conceptual clustering approach. Improving co-cluster relevancy remains a difficult task in real-life exploratory data analysis processes. First, it is hard to capture subjective interestingness aspects, e.g., the analyst's expectation given her/his domain knowledge. Next, when these expectations can be declaratively specified, using them during the computational process of bi-partitions is challenging. In [START_REF] Pensa | Towards constrained co-clustering in ordered 0/1 data sets[END_REF][START_REF] Pensa | Constraint-driven co-clustering of 0/1 data[END_REF], a simple approach was suggested that was dedicated to 0/1 data analysis and incomplete w.r.t. constraint processing. In this paper, we have proposed a new constrained co-clustering algorithm. We explained how to exploit user-defined constraints like must-link, cannotlink , and interval constraints when co-clustering numerical matrices. Applications on kinetic gene expression data analysis have been considered. Many other applications rely on ordered data analysis and may benefit from such constrained co-clustering approaches.

A short-term perspective is to combine properly the strategies for exploiting must-link and cannot-link constraints one one hand, and interval constraints on another hand. So far, our approach does not look for overlapping co-clusters while this may be interesting for many applications. We may study the possibility to discover overlapping co-clusters, like many local biclustering approaches already do (see, e.g., [START_REF] Cheng | Biclustering of expression data[END_REF][START_REF] Ihmels | Revealing modular organization in the yeast transcriptional network[END_REF]). Also, we are convinced that our approach can be easily extended towards other kinds of objective functions (e.g., mutual information [START_REF] Dhillon | Information-theoretic co-clustering[END_REF] or the more general setting of Bregman functions [START_REF] Banerjee | A generalized maximum entropy approach to bregman co-clustering and matrix approximation[END_REF]), and other user-defined constraints (e.g., balancing constraints [START_REF] Banerjee | Scalable clustering algorithms with balancing constraints[END_REF]). Extending our algorithms and using them in other application domains (e.g., document clustering , privacy preserving clustering) is also planned.

Figure 1 :

 1 Figure 1: A toy example X r

Algorithm 4 . 1 .

 41 Co-clustering under must-link and cannot-link constraints.

Algorithm 4 . 4 .

 44 Co-clustering with an interval constraint.

Figure 3 :

 3 Figure 3: Two different views of 3 co-clusters captured with conjunctions of must-link and cannot-link constraints. The three top plots show the expression level of 100 randomly picked genes. Bottom plots show the expression level of the 100 genes closest to the average expression profile.

Figure 4 :

 4 Figure 4: Object value vs. iterations

Figure 5 :

 5 Figure 5: Object value vs. iterations

Table 2 :

 2 Genes in the 3 clusters under constraints

	MI	ERT

Table 3 :

 3 Adjusted Rand index, final objective function value and number of iterations.

		AR r AR c	||H|| 2	Nb.Iter
	Const.	0.88 0.73 2.16×10 4	9.18
	Unconst. 0.70 0.43	2.21×10 4	9.35

Table 4 :

 4 Adjusted Rand index, final objective function value and number of iterations.

		AR	||H|| 2	Nb.Iter.
	Const.	0.76 8.23×10 4	11.60
	Unconst.	0.41 7.73×10 4	14.60
	Const.Init. 0.54 7.86×10 4	11.05

Another approach is called the metric-based approach for which a metric is trained considering labeled data and then standard clustering is applied.

The preprocessed and cleaned version of this data set is available on http://www-kdd.isti.cnr.it/˜pensa/datasets/.

Acknowledgements. The authors wish to thank the anonymous reviewers for their valuable suggestions. This research is partially funded by EU contracts IQ IST-FP6-516169 and GeoPKDD IST-FP6-014915.