Peter Baumgartner
email: peter.baumgartner@donau-uni.ac.at

Improving Reusability of OER Educational Patterns for Content Sharing

The effect of Open Educational Resources (OER) on Higher Education is still disappointing. (Re)use of materials, which can be accessed and employed freely, has not developed in such a way that it has changed the attitudes and behavior of teachers. After analyzing several aspects of the problem the article will focus on educational reasons to improve this situation. It is argued that to strive for context free learning objects is heading in the wrong direction. The author proposes to link OER not only with an educational taxonomy of learning outcomes but also with typical patterns of educational scenarios.

1

Barriers to overcome for using OER

The work for reusable learning objects (RLO's) started almost 15 years ago [START_REF] Hodgins | The future of learning objects[END_REF][START_REF] Longmire | A primer on learning objects[END_REF][START_REF] Mcgreal | Learning objects: A practical definition[END_REF]. In combination with the idea of open educational resources (OER) -material, which can be accessed and used freely -it was assumed that the typical provision of learning material would change radically: from printed material protected by copyright to OER delivered electronically by the internet.

There is growing critique about the missing impact of RLO & OER in Higher Education. In February 2013 Gerd Kortemayer summed up the situation in educause.edu: "OERs have not noticeably disrupted the traditional business model of higher education or affected daily teaching approaches at most institutions" [START_REF] Kortemeyer | Ten Years Later: Why Open Educational Resources Have Not Noticeably Affected Higher Education, and Why We Should Care[END_REF]. He is only one member of the increasing camp of skeptics and there are many different assumptions why there is so little success and acceptance of OER. The following paragraphs summarize some of the hurdles to overcome.

Difficulty to find the appropriate learning material

It is still not easy to find quickly the appropriate material for the intended learning/teaching purpose. There exist different dimensions of this problem:

─ Economy of scale: Even with objects in the magnitude order of billions we face the problem of dragnet investigation. We are not looking for educational material as such but for an object with many detailed characteristics. This desired list of qualities are linked with the "and" operator and are therefore limiting the search result with every additional property. It is very doubtful if teachers trying to find these kinds of specified objects might succeed. Imagine for instance a teacher searching • for a course in a specified subject (e.g. mathematics)

• for a specified very detailed teaching/learning area (e.g. factorizing quadratic trinomials) • for a specified pedagogical strategy (e.g. to explain, to practice, to demonstrate, to visualize) • for a specified language (e.g. German)

• for a specified target group (e.g. adults with rudimental mathematic knowledge/experience) • for a specified number of learners • for a certain learning time (in hours)

• for a specified learning environment or learning platform (e.g. lecture hall, moodle, etc.) • for a specified license model (e.g. creative common: by name, commercial and share alike) • … ─ Educational metadata: In spite of sophisticated federated search engines and well known huge content portals we are still missing a formal educational taxonomy where important sectors of the educational community can agree. The LOMstandard is for the above specified educational purposes ridiculous weak. What does it mean for instance that some educational elements like level of interactivity, semantic density, and difficulty vary in five categories (very easy, easy, medium, difficult, very difficult)? And what is the yardstick for these properties and who judges them? -But even though there are agreed application profiles: Who will undertake the tedious task of filling in all the many necessary details? Experiences show that most of the material collected in portals or found with sophisticated search engines lack educational metadata at all. ─ Educational culture: There is the well know problem to overcome the barrier that objects created for a limited personal usage have to undergo still a long and cumbersome enterprise to make it fool proof for every possible standard situations. Who will get the payoff for this work? In order to promote the development and improvement of OER educational systems would have to cherish exchange or gift cultures in contrast to traditionally predominant business models. ─ Educational quality assurance: Evaluating the quality of OER for learning/teaching purposes has to overcome different hurdles:

• Who has the necessary qualification and authority? This is not only a question of competence but in a participatory community model also a question of regulatory procedure and power relations. • What kind of agreed and fast procedure is to follow? The blind peer review as the traditional model of quality assurance in science is not only far too slow but also seems inadequate in an open community model of fine grained different needs and diverse interest/target groups committed to a variety of educational models and approaches.

As one can see I have focused my list of difficulties to the organizational and pedagogical sphere and not elaborated on technical problems related to RLO's and OER. This concentration on organizational and educational issues is not only governed by my own competences in pedagogy but is also a result of my conviction that we have to enforce the pedagogical point of view in order to move forward OER practices considerably.

For a conceptual turn -Context (not content) is king

During the last 15 years I have argued from an educational point of view that content is just another element of the complex learning situation (also known as "context"). I have stressed the relationship of educational theory such as behaviorism, cognitivism and constructivism to the dynamics of content provision [START_REF] Baumgartner | The Zen Art of Teaching -Communication and Interactions in eEducation[END_REF]. I described the different learning attitudes as Learning I, II and III and demonstrated the different perspective to the role of content in these three models. Only in "Learning I" is the transfer of "correct" or "true" knowledge the predominant strategy. In constructivist environments ("Learning III") even "bad" content can be used to the best advantage of learning processes (e.g. when students have to find mistakes and wrong assumptions in order to improve or elaborate the material).

I believe that there are two key features essential for a paradigmatic turn:

From sender/receiver model to the self-determinated learner. The idea that high reusable content has to be context free as much as possible is still following the -at least in education -long ago outdated communication model where the teacher (sender) is just transmitting neutral information to the learner (receiver). The congenial categorical teaching model of this approach is the so-called "educational triangle" (cf. Fig. 1), which has only a unidirectional sequence from teacher to learner transferring the content [START_REF] Fricke | Methoden der Evaluation von E-Learning-Szenarien im Hochschulbereich[END_REF][START_REF] Schulmeister | Didaktisches Design aus hochschuldidaktischer Sicht: ein Plädoyer für offene Lernsituationen[END_REF].

Nowadays categorical teaching models are not only more complex but are centered on the learner and not on the teacher. In addition to simply transfer content there is also the contextual/situational learning challenge learner have to meet. (Cf. as an example of an advanced categorical educational model Fig. 2. The numbers in the diagramm shows all the different bilateral connections as a subset of a dynamic network of the huge variety of possible relationships between the different educational categories.)

Additionally we know that various types of motivations shape learning experiences essentially. This qualification refers not only to the somewhat crude and well known distinction of external and internal motivation but also -as Deci/Ryan have shown empirically and convincingly -to different degrees between these opposites [START_REF] Deci | Handbook of self-determination research[END_REF][START_REF] Ryan | Intrinsic and extrinsic motivations: Classic definitions and new directions[END_REF][START_REF] Ryan | Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being[END_REF]. From thinking in separated modules to a holistic network approach. The second new important change in the conceptual orientation to overcome problems in using OER is abandoning the so-called Lego approach of learning objects. According to this now criticized view we have to build small content units with standardized interfaces [12-14, 1, 3]. Similar like Lego's building blocks we can assemble complex structures by putting these different components via their interfaces together. A consequence of using the Lego metaphor is the discussion on granularity: How small grained should the standardized building block be? [START_REF] Ip | What is a learning object, technically? WebNet[END_REF][START_REF] Mcgreal | Online education using learning objects[END_REF][START_REF] Polsani | Use and abuse of reusable learning objects[END_REF] But this approach does not even work with Lego as the ingenious enterprise knows. Lego is providing quite a different range of building blocks to support a variety of usage and construction ideas (cf. Fig. 3). And as one can see the heuristic rule "build content block as small as possible in order to be maximized for reuse" does not work even in the basic metaphor: The Lego "atom" cannot provide even for the simple constructions in the right hand upper blue arc sitting on a red bridge. But what is wrong with a module approach in pedagogy? Aren't we using this conception of building blocks, so-called "modules", for the development of our course curricula as well? No, there is one big difference: Planning a curriculum is a holistic enterprise, starting top-down, not bottom-up: The first question is: "What are the learning outcomes (necessarily acquired competences) for a specific curriculum?" Only after answering this primal question we are concerned with modularization. Knowing all the time that a good curriculum should not only have modules which build on each other but also have as many relations between modules as possible. The Lego approach in learning objects goes the reverse direction: "What are feasible small learning units which could be used many times in different situation/curricula so that there is a high return of investments (ROI) for the development costs?".

From the educational point of view there is also a ROI schema, which is very different: I call it the Reusability of Instruction paradox (ROI paradox): Instead of a standardized learning environment which covers all possible situations the art of instructional design has to reflect and integrate all the different contextual conditions of the learning environment in order to be a learning process of high efficiency. Some of the questions to be answered and to be accounted for are:

• What is the specific purpose of the intended learning process?

• What is the previous knowledge/experience of the learners? • How much time is available (for learner as well as for teacher)?

• What group size is expected?

• What kind of (virtual) educational environment can be accounted for?

• What kind of intervention for the intended learning outcome is most appropriate? • …

In the rest of the article I will outline an alternative approach of building blocks for dynamic and interrelated learning processes.

2

Reestablish the wholeness of the learning situation

The reconceptualization of the learning object

The main problem of the Lego approach for learning object is the destruction of the wholeness of the learning situation. One part of a complex learning arrangement -the learning content -is taken as the representative for the interrelated connection of all parts. The consequence is a certain overemphasis of content ("Content is king") and a fallback to the education triangle model. To distinguish this extended conception I will change the name "learning object" of this building block with regard to content to information object (IO). To reestablish the wholeness of the learning experience we would need to focus our attention of all the relevant elements of the educational situation. I call the collection of these interrelated parts the educational scenario (ES).

But how are these two essential parts of every learning situation linked together? The glue that brings them together and attaches them to each other is the learning target (LT). Nowadays educational specialists distinguish between learning goals and learning outcomes. As far as this paper is concerned I will use both terms equivalent. Therefore we have saved the abbreviation "LO" (learning object) for the whole new construction (cf. Fig. 4). Either the enhanced taxonomy of Bloom [START_REF] Bloom | Taxonomy of educational objectives; the classification of educational goals[END_REF][START_REF] Anderson | Revising Bloom's taxonomy[END_REF][START_REF] Anderson | A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives[END_REF][START_REF] Anderson | National Society for the Study of Education: Bloom's taxonomy: a forty-year retrospective[END_REF] or the approach of Marzano/Kendall [START_REF] Marzano | The new taxonomy of educational objectives[END_REF][START_REF] Marzano | American Association of School Administrators[END_REF] could be used.

It is important to understand that the same information object could be used for different learning targets. For instance, using the vocabulary of the revised taxonomy by Anderson and colleagues one information object could be targeted at different cognitive process dimensions: remember, understand, apply, analyze, evaluate, create. In that case the information object would need alternative classification elements. ─ Learning Target (LT): These days the description of learning outcomes for modules or courses is obligatory. Therefore it should be easy to collect and provide this essential information and to pack it into a searchable object. But there is still some homework to be done: The used vocabulary has to be restricted to a chosen taxonomy of learning objectives. This chosen taxonomy has to be made explicit and their vocabulary has to be used throughout the whole system, e.g. the collection of learning targets, information objects and educational scenarios have to apply the same taxonomy. ─ Educational Scenario (ES): Here is the real challenge: We do not have an agreed taxonomy of educational scenarios and we are also lacking a formal system of descriptions in order to specify the dynamic and complexity of these elements of the learning situation. In some educational communities the pattern language approach of the architect Christopher Alexander [START_REF] Alexander | Notes on the synthesis of form[END_REF][START_REF] Alexander | The timeless way of building[END_REF][START_REF] Alexander | A pattern language: towns, buildings, construction[END_REF] is discussed as a possible candidate for this kind of description [START_REF] Bagert | Pedagogical patterns: advice for educators[END_REF][START_REF] Köppe | A Pattern Language for Teaching Design Patterns (Part 1)[END_REF][START_REF] Kohls | A Pattern Language for Online Trainings[END_REF][START_REF] Bauer | Showcase of learning: towards a pattern language for working with electronic portfolios in higher education[END_REF][START_REF] Kohls | Investigations of E-learning Patterns: Context Factors, Problems, and Solutions[END_REF]. This discussion is grounded on the conception of teaching as a design science and community endeavor [START_REF] Laurillard | Teaching as a design science: building pedagogical patterns for learning and technology[END_REF]. There is also starting a lively debate about philosophical underpinnings of this approach after Alexander has published his four volume magnum opus "The Nature of Order" [START_REF] Alexander | The nature of order: an essay on the art of building and the nature of the universe. 4 Volumes[END_REF].

Summary and Outlook

Especially on the educational side of the learning object equation much work has still to be done. We need a huge collection of educational scenarios based on a (somewhat) formalized description system, which could be derived from the Alexandrian pattern language approach. The result would be a new kind of educational repository as demonstrated in figure 5. At edu-sharing.net, an interdisciplinary community of computer scientists, psychologists, educational specialists, teachers and administrators are currently con-structing and exploring the viability of the conceptual approach described in this paper.

Fig. 1 .

 1 Fig. 1. Educational Triangle

Fig. 2 .

 2 Fig. 2. Categorical Educational Model[START_REF] Baumgartner | Taxonomie von Unterrichtsmethoden: Ein Plädoyer für didaktische Vielfalt[END_REF]

Fig. 3 .

 3 Fig. 3. Lego: Uniform buildings block do not match the variety of user wants

Fig. 4 . 2 Educational

 42 Fig. 4. The parts of the new learning object


Ali Shojaee Bakhtiari

Nizar Bouguila
email: nizar.bouguila@concordia.ca

Online Learning for Two Novel Latent Topic Models

Keywords: Generalized Dirichlet, Beta-Liouville, online learning, Latent model, variational learning, count data

Latent topic models have proven to be an efficient tool for modeling multitopic count data. One of the most well-known models is the latent Dirichlet allocation (LDA). In this paper we propose two improvements for LDA using generalized Dirichlet and Beta-Liouville prior assumptions. Moreover, we apply an online learning approach for both introduced approaches. We choose a challenging application namely natural scene classification for comparison and evaluation purposes.

Introduction

In order to extract the hidden information within count data various models have been proposed in the past. The first widely used model for count data modeling was the naive Bayes model combined with multinomial distribution [START_REF] Nigam | Text classification from labeled and unlabeled documents using EM[END_REF][START_REF] Bouguila | A model-based approach for discrete data clustering and feature weighting using map and stochastic complexity[END_REF][START_REF] Bakhtiari | A novel hierarchical statistical model for count data modeling and its application in image classification[END_REF]. However several researchers proceeded with mentioning the oversimplifications and the subsequent drawbacks of the Naive Bayes assumption [START_REF] Madsen | Modeling word burstiness using the Dirichlet distribution[END_REF][START_REF] Bouguila | Unsupervised learning of a finite discrete mixture: Applications to texture modeling and image databases summarization[END_REF][START_REF] Bouguila | A generative model for spatial color image databases categorization[END_REF]. The foremost solution offered to compensate for the deficiencies of the naive assumption was considering the Dirichlet distribution [START_REF] Madsen | Modeling word burstiness using the Dirichlet distribution[END_REF][START_REF] Bouguila | Unsupervised learning of a finite discrete mixture: Applications to texture modeling and image databases summarization[END_REF] as the prior assumption for the multinomial distribution. Based on the Dirichlet assumption, several models have been developed for proper count data modeling. One model that has gained much acceptance among the research community is the latent Dirichlet allocation (LDA) model firstly proposed in [START_REF] Blei | Latent Dirichlet allocation[END_REF]. LDA model uses a Bayesian model for data generation using a variational Bayes (VB) approach for parameter inference. The majority of the models developed thus far have been based on the Dirichlet prior assumption. However, researchers began questioning the merit of Dirichlet assumption [START_REF] Bouguila | Count data modeling and classification using finite mixtures of distributions[END_REF]. The main drawback of the Dirichlet assumption is the fact that it has a strictly negative covariance matrix and therefore it inherently fails to properly model the data in which topics have positive correlation in between. The other main drawback of the Dirichlet distribution is the fact that the elements with similar mean need to have similar variance, which clearly is an oversimplification. To overcome these shortcomings research has recently been shifted towards finding models with better modeling accuracy.

Recently it has been shown that the generalized Dirichlet distribution is a good replacement for the Dirichlet distribution when using finite mixture models [START_REF] Bouguila | Clustering of count data using generalized dirichlet multinomial distributions[END_REF]. One important factor of the generalized Dirichlet is that like Dirichlet distribution it is a conjugate prior to the multinomial distribution. Generalized Dirichlet distribution also does not carry the restrictions of the Dirichlet distribution and allows more relaxed modeling capabilities. Another modeling prior that has recently attracted notice is the Beta-Liouville distribution [START_REF] Bouguila | Count data modeling and classification using finite mixtures of distributions[END_REF]. The advantage point of the Beta-Liouville distribution in comparison to generalized Dirichlet is that it requires only two more parameters to be estimated compared to Dirichlet distribution compared with the twice the parameters the generalized Dirichlet distribution requires. Based on the above facts we proceed with proposing two different latent topic models based on the LDA model but with the generalized Dirichlet and the Beta-Liouville assumptions. The former model is called latent generalized Dirichlet allocation (LGDA) and the latter is called latent Beta-Liouville allocation (LBLA). The learning of both models is performed online using the approach proposed in [START_REF] Hoffman | Online learning for latent dirichlet allocation[END_REF]. The structure of the paper is as follows. In section 2, we introduce the LGDA and LBLA models. In section 3 we shall describe the adaption of the online learning model on the two models. In section 4 we shall bring the experimental results and we will finalize the paper with conclusion.

Proposed Latent Topic Models

In this section we briefly describe the two proposed latent topic models. Both models essentially have the same generative model as the LDA:

1. Choose N ∝ Poisson(ζ). 2. Choose (θ 1 , . . . , θ d ) ∝ Dir(ξ).
3. For each of the N words w n :

(a) choose a topic z n ∝ M ultinomial(θ).

(b) Choose a word w n from p(w n |z n , β w ).

In above z n is a d dimensional binary vector of topics defined so that z i n = 1 if the ith topic is chosen and zero, otherwise. We define, θ = (θ 1 , . . . , θ d ). A chosen topic is attributed to a multinomial prior β w over the vocabulary of words so that β w ( ij) = p(w j = 1|z i = 1), from which every word is randomly drawn. p(w n |z n , β w ) is a multinomial probability conditioned on z n and Dir(ξ) is a d-variate Dirichlet distribution with parameters ξ = (α 1 , . . . , α d ).

The main inference problem of LGDA is estimating the posterior of the hidden variables, θ and z:

p(θ, z|w, ξ, β w ) = p(θ, z, w|ξ, β w ) p(w|ξ, β w ) (1)
The above equation is known to be intractable. As proposed in [START_REF] Blei | Latent Dirichlet allocation[END_REF], an efficient way to estimate the parameters of this intractable posterior is to use the vibrational Bayes (VB) inference. VB inference offers a solution to the intractability problem by determining a lower bound on the log likelihood of the observed data which is mainly based on considering a set of vibrational distributions on the hidden variables [START_REF] Jordan | An introduction to variational methods for graphical models[END_REF]:

q(θ, z|w, ξ q , Φ w ) = q(θ|ξ q ) N n=1 q(z n |φ n ) (2)
The details of the parameter estimation algorithm can de found in [START_REF] Blei | Latent Dirichlet allocation[END_REF]. In this section we proceed with introducing LGDA and LBLA subsequently.

Latent Generalized Dirichlet Allocation

The major difference between the LGDA and the LDA model is the consideration of the generalized Dirichlet assumption:

1. Choose N ∝ Poisson(ζ). 2. Choose (θ 1 , . . . , θ d ) ∝ GenDir(ξ). 3. For each of the N words w n : (a) choose a topic z n ∝ M ultinomial(θ). (b) Choose a word w n from p(w n |z n , β w ).
In above z n is a d + 1 dimensional binary vector of topics defined so that z i n = 1 if the ith topic is chosen and zero, otherwise. We define, θ = (θ 1 , . . . , θ d+1 ), where θ d+1 = 1 -d i=1 θ i . A chosen topic is attributed to a multinomial prior β w over the vocabulary of words so that β w ( ij) = p(w j = 1|z i = 1), from which every word is randomly drawn. p(w n |z n , β w ) is a multinomial probability conditioned on z n and GenDir(ξ) is a d-variate generalized Dirichlet distribution with parameters ξ = (α 1 , β 1 , . . . , α d , β d ) and probability distribution function given by:

p(θ 1 , . . . , θ d |ξ) = d i=1 Γ (α i + β i ) Γ (α i )Γ (β i ) θ αi-1 i (1 - i j=1 θ j ) γi (3)
where

γ i = β i -α i+1 -β i+1 . It is straightforward to show that when β i = α (i+1) + β (i+1)
, the generalized Dirichlet distribution is reduced to Dirichlet distribution [START_REF] Bouguila | High-dimensional unsupervised selection and estimation of a finite generalized Dirichlet mixture model based on minimum message length[END_REF]. Therefore it is understood that under certain conditions LGDA will also behave like LDA and subsequently LDA is a special case of the LGDA model. To estimate the posterior of the hidden variables of the LGDA model we use a similar VB approach as the one proposed for LDA, where q(θ|ξ q ) can be viewed as a variational generalized Dirichlet distribution, calculated once per document, q(z n |φ n ) is a multinomial distribution with parameter φ n extracted once for every single word inside the document, and Φ w = {φ 1 , φ 2 , . . . , φ N }. Using Jensen's inequality [START_REF] Jordan | An introduction to variational methods for graphical models[END_REF] one can derive the following:

log p(w|ξ, β w ) ≥ E q [log p(θ, z, w|ξ, β w )] -E q [log q(θ, z)] (4)
Assigning L(ξ q , Φ w ; ξ, β w ) to the right-hand side of the above equation it can be shown that the difference between the left-hand side and the right-hand side of the equation is the KL divergence between the variational posterior probability and the actual posterior probability, thus we have:

log p(w|ξ, β w ) = L(ξ q , Φ w ; ξ, β w ) + KL q(θ, z|ξ q , Φ w ||p(θ, z)|w, ξ, β w ) (5)
The left hand side of the above equation is constant in relation to variational parameters, therefore to minimize the KL divergence on the right-hand side one can proceed with maximizing L(ξ q , Φ w ; ξ, β w ). Up to here the formulation basically follows the LDA model. The divergence of the models begins when we proceed with assigning the generalized Dirichlet distribution as the parameter generator instead of the LDA Dirichlet assumption. The breakdown of L(ξ q , Φ w ; ξ, β w ) to maximize the lower bound L(ξ q , Φ w ; ξ, β w ) with respect to φ nl , leads to the following updating equations for the variational multinomial:

φ nl = β lv e (λn-1) e (Ψ (γ l )-Ψ (γ l +δ l )) φ n(d+1) = β (d+1)v e (λn-1) e (Ψ (δ d )-Ψ (γ d +δ d )) (6)
where Ψ is the digamma function, β lv = p(w v = 1|z l = 1) and the weighing constant e λn-1 is given by:

e λn-1 = 1 d l=1 β lv e (Ψ (γ l )-Ψ (γ l +δ l )) + β (d+1)v e (Ψ (δ d )-Ψ (γ d +δ d )) (7)
Maximizing the lower bound L with respect to the variational generalized Dirichlet parameter gives the following updating equations:

γ l = α l + N n=1 φ nl δ l = β l + N n=1 d+1 ll=l+1 φ n(ll) (8)
The above equations show that the variational generalized Dirichlet for each document acts as a posterior in the presence of the variational multinomial parameters. The same conclusion was observed in [START_REF] Blei | Latent Dirichlet allocation[END_REF] for the LDA case. This is a direct result of the conjugacy between the generalized Dirichlet and the multinomial distribution. The LGDA parameters are corpus parameters and therefore they are estimated by considering all M documents inside the corpus.

In the following, we denote L = M m=1 L m as the lower bound corresponding to all the corpus, where L m is the lower bound corresponding to each document m. Maximizing the corpus lower bound L with respect to β w(lj) delivers the following updating equation:

β w(lj) ∝ M d=1 N d n=1 φ dnl w j dn (9)
The model's parameters are derived using a Newton-Raphson method.

Latent Beta-Liouville allocation

The model that we briefly discuss in this subsection, latent Beta-Liouville allocation (LBLA), is another model developed based on the LDA model. The model assumes a Beta-Liouville as its topic generating prior. The model proceeds with generating every single word (or visual word) of the document (or the image) through the following steps:

1. Choose N ∝ Poisson(ζ). 2. Choose (θ 1 , . . . , θ D ) ∝ BL(ξ).
3. For each of the N words w n :

(a) choose a topic z n ∝ M ultinomial(θ).

(b) Choose a word w n from p(w n |z n , β w ).

In above z n is a D + 1 dimensional binary vector of topics defined so that z i n = 1 if the ith topic is chosen and zero, otherwise. We define, θ = (θ 1 , . . . , θ D+1 ), where θ D+1 = 1-D i=1 θ i . A chosen topic is attributed to a multinomial prior β w over the vocabulary of words so that β w ( ij) = p(w j = 1|z i = 1), from which every word is randomly drawn. p(w n |z n , β w ) is a multinomial probability conditioned on z n and BL(ξ) is a d-variate Beta-Liouville distribution with parameters ξ = (α 1 , β 1 , . . . , α d , β d ) and probability distribution function given by:

P (θ 1 , . . . , θ D |ξ) = Γ ( D d=1 α d )Γ (α + β) Γ (α)Γ (β) D d=1 θ α d -1 d Γ (α d ) ( D d=1 θ d ) α-D l=1 α l (1- D l=1 θ l ) β-1 (10
) where γ i = β iα i+1β i+1 . It is straightforward to show that when β i = α (i+1) +β (i+1) , the Beta-Liouville distribution is reduced to Dirichlet distribution [START_REF] Bouguila | High-dimensional unsupervised selection and estimation of a finite generalized Dirichlet mixture model based on minimum message length[END_REF]. We define, θ = (θ 1 , . . . , θ D+1 ), where θ D+1 = 1 -D i=1 θ i . The maximization of the lower bound L(ξ q , Φ w ; ξ, β w ) with respect to φ nl , leads to the following updating equations: φ nl = β lv e (λn-1) e (Ψ (γi)-Ψ ( D ii=1 γii)) [START_REF] Jordan | An introduction to variational methods for graphical models[END_REF] φ n(D+1) = β (D+1)v e (λn-1) e (Ψ (βγ )-Ψ (αγ +βγ )) [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] where Ψ is the digamma function, β lv = p(w v = 1|z l = 1) and the weighing constant e λn-1 is given by:

e λn-1 = 1 β (D+1)v e (Ψ (βγ )-Ψ (αγ +βγ )) + D i=1 β iv e (Ψ (γ d )-Ψ ( D ii=1 γII )) (13)
γ i = α + N n=1 φ ni α γ = α + N n=1 D d=1 φ nd β γ = β + N n=1 φ n(D+1) (14)
The above equations show that the variational Beta-Liouville for each document acts as a posterior in the presence of the variational multinomial parameters.

The same conclusion was observed in [START_REF] Blei | Latent Dirichlet allocation[END_REF] for the LDA case. This is a direct result of the conjugacy between the Beta-Liouville and the multinomial distribution.

One needs to consider that the LBLA parameters are corpus parameters and therefore they are estimated by considering all M documents inside the corpus.

In the following, we denote L = M m=1 L m as the lower bound corresponding to all the corpus, where L m is the lower bound corresponding to each document m. Maximizing the corpus lower bound L with respect to β w(lj) delivers the following updating equation:

β w(lj) ∝ M d=1 N d n=1 φ dnl w j dn (15)
The model's parameters are the last ones to be derived using a Newton-Raphson algorithm, also.

Online Latent Topic Models

The variational Bayes model of the LDA model and the subsequent adaption for the LGDA and LBLA are shown to coverage to a local likelihood of the actual posterior of the hidden parameters of the models. However, the main problem with the original VB approach is that it needs to consider the entire corpus beforehand for parameter estimation. This in return emerges two serious problems. Firstly, the need for the collection of the entire training corpus and secondly the computational requirements of dealing with a huge corpus. To overcome this problem the authors in [START_REF] Hoffman | Online learning for latent dirichlet allocation[END_REF] offered an online learning model that fixes the mentioned issues. The solution is based on a time dependent (time defined as the index of the part of the data given to the model in each iteration) weight:

ρ t (τ 0 + t) -κ , κ ∈ (0.5, 1] (16)
The parameter τ 0 slows down the effect of early parameter estimations. The online learning algorithm can easily be extended to cover LGDA and LBLA models as well. The steps of the algorithm are as follows.

1. In each learning interval the model performs a batch VB over the patch of the training set attributed to that interval and assigns a weight value to the patch according to 16. 2. Prior parameter estimation: Perform the Newton-Raphson algorithm over the entire corpus for t = 0 to ∞ as: ξ ← ξρ t α(ξ t ) where α(ξ t ) is the inverse of the Hessian times the gradient in respect to α of the posterior lower bound. 3. Word dictionary update: β w (t + 1) = normalize((1ρ t ) β w t + ρ t β w (t)) where β w (t) is the available estimation of the word dictionary at t-th step.

It was shown in [START_REF] Hoffman | Online learning for latent dirichlet allocation[END_REF] that the condition κ ∈ (0.5, 1] is necessary for keeping the online learning model stable.

In this section we shall proceed with applying our proposed two models, online LBLA and LGDA, on the challenging task of natural scene classification and make a comparison between the classification success rates offered by the two models versus that of the online LDA. The main idea that we use here is based on the description of scenes using visual words [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF]. This approach has emerged over the past few years and received strong interest that is mainly motivated by the fact that many of the techniques previously proposed for text classification can be adopted for images categorization [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF][START_REF] Weston | Large scale image annotation: learning to rank with joint word-image embeddings[END_REF][START_REF] Scalzo | Adaptive patch features for object class recognition with learned hierarchical models[END_REF].

For the construction of the visual words vocabulary, we need first to extract local descriptors from a set of training images. Many descriptors have been proposed in the past, but scale invariant feature transform (SIFT) descriptor [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF], that we consider here, has dominated the literature. The extracted features are then quantized through clustering (the K-Means algorithm in our case) and the obtained d clusters centroids are considered as our visual words. Having the visual vocabulary in hand, each image can be represented as a d-dimensional vector containing the frequency of each visual word in that image. In our experiment we take 7 classes from the natural scenes dataset introduced in [START_REF] Oliva | Modeling the shape of the scene: A holistic representation of the spatial envelope[END_REF]. The 7 classes chosen from the data set described in [START_REF] Oliva | Modeling the shape of the scene: A holistic representation of the spatial envelope[END_REF] At first the models were given 5 chunks of training images each containing 20 images. In this set of experiments the effect of the online learning was reduced since the small number of iterations plus the big chunks of test data quite resembled the Batch LDA and LGDA models. The results of applying the online LDA and LBLA models are brought in Fig. 2. Under the same experimental conditions we proceed with delivering the results for the LGDA model as well in Fig. 3. The optimal confusion matrix of the online LBLA model is brought 3. Optimal confusion matrix of the online LDA model applied for the scenes classification task.

Conclusion

In this work we proposed two new online learning multitopic models. We performed a series of experiments over a challenging application, natural scene classification, and we showed and compared the merits of our proposed models in comparison with online LDA. The two models show promising results when adapted for the online learning scheme and tend to surpass the online LDA model in the scope of the experiments performed.

  are coast, forest, highway, inside of cities, open country, street, and tall building, which contain 361, 329, 261, 309, 411, 293, and 356 images, respectively. Examples of images from the different considered classes are shown in figure 1.

Fig. 1 .

 1 Fig. 1. Sample images from each group. (a) Highway, (b) Inside of cities, (c) Tall building, (d) Streets, (e) Forest, (f) Coast, (g) Open country, (h) Bedroom.

4. 1

 1 Comparison between the performance of LBLA and LGDA models against LDA

Fig. 2 .

 2 Fig. 2. Comparison of the success rates of the online LBLA model against online LDA model for the natural scene classification application for a training size that equals 20 for two different extracted number of topics.

Fig. 3 .

 3 Fig. 3. Comparison of the success rate of the online LGDA model against online LDA model for the natural scene classification application over a training size that equals 20 for two different extracted number of topics.

table 1 .

 1 The optimal confusion matrix of the online LBLA model is brought in table 2 and the optimal confusion matrix of the online LDA model is brought in table3.

		C F H I O S T
	Coast (C)	216 1 86 3 50 3 2
	Forest (F)	3 242 15 53 4 51 0
	Highway (H)	40 1 69 5 6 3 15
	Inside of cities (I) 0 2 4 146 2 12 6
	Open country (O) 90 39 23 22 331 14 50
	Streets (S)	5 41 60 57 9 203 2
	building (T) 6 2 3 22 8 6 281

in

Table 1 .

 1 Optimal confusion matrix of the online LBLA model applied for the scenes classification task.

		C F H I O S T
	Coast (C)	282 4 130 5 98 7 14
	Forest (F)	9 287 24 84 24 167 3
	Highway (H)	19 2 55 13 16 6 94
	Inside of cities (I) 4 19 23 157 1 58 9
	Open country (O) 38 7 19 24 241 19 66
	Streets (S)	8 7 8 18 30 32 13
	Tall building (T) 0 2 1 7 0 3 157

Table 2 .

 2 Optimal confusion matrix of the online LGDA model applied for the scenes classification task.

Acknowledgments. The completion of this research was made possible thanks to the Natural Sciences and Engineering Research Council of Canada (NSERC).


Nizar Bouguila
email: nizar.bouguila@concordia.ca

Mohamed Al Mashrgy

An Infinite Mixture Model of Generalized Inverted Dirichlet Distributions for High-Dimensional Positive Data Modeling

Keywords: Clustering, feature selection, generalized inverted Dirichlet, mixture models, Bayesian inference, image databases

We propose an infinite mixture model for the clustering of positive data. The proposed model is based on the generalized inverted Dirichlet distribution which has a more general covariance structure than the inverted Dirichlet that has been widely used recently in several machine learning and data mining applications. The proposed mixture is developed in an elegant way that allows simultaneous clustering and feature selection, and is learned using a fully Bayesian approach via Gibbs sampling. The merits of the proposed approach are demonstrated using a challenging application namely images categorization.

Introduction

The important proliferation of digital content requires the development of powerful approaches for knowledge extraction, analysis, and organization. Clustering, in particular, has been widely adopted for knowledge discovery and data engineering. The main goal of any clustering algorithm is to partition a given data set into groups so that objects within a cluster are more similar than those in different clusters [START_REF] Topchy | Analysis of consensus partition in cluster ensemble[END_REF]. Many clustering techniques have been developed in the past and have been applied successfully on different data types (e.g. binary, discrete, continuous) extracted within various applications [START_REF] Bezdek | Approximate clustering in very large relational data[END_REF][START_REF] Chen | Spectral clustering with discriminant cuts[END_REF][START_REF] Huang | A hybrid particle swarm optimization approach for clustering and classification of datasets[END_REF]. Among these techniques, mixture models have played important roles in many areas including (but not confined to) image processing, computer vision, data mining, and pattern recognition, thanks to their flexibility and strong statistical foundations which offer a formal principled way to clustering. In particular, Gaussian mixture model has drawn considerable attention in the machine learning community and has achieved good results [START_REF] Mclachlan | Finite Mixture Models[END_REF]. However, recent concentrated research efforts have shown that this mixture model may fail to provide good generalization capabilities when the per-cluster data distributions are clearly non-Gaussian, which is the case of positive data as deeply discussed in [START_REF] Bdiri | Positive vectors clustering using inverted dirichlet finite mixture models[END_REF][START_REF] Bdiri | Learning inverted dirichlet mixtures for positive data clustering[END_REF][START_REF] Bdiri | An infinite mixture of inverted dirichlet distributions[END_REF]. The main contribution of [START_REF] Bdiri | Positive vectors clustering using inverted dirichlet finite mixture models[END_REF][START_REF] Bdiri | Learning inverted dirichlet mixtures for positive data clustering[END_REF] was the introduction of the finite inverted Dirichlet mixture model for the clustering of positive data which are naturally generated by many real-world applications. The authors have proposed a detailed approach for the learning of the parameters of this finite mixture, also. In order to handle huge number of classes and avoid over-or under-fitting problems (a.k.a. controlling variance, model selection), which is a central issue in learning-based techniques, the finite inverted Dirichlet mixture was extended to the infinite case in [START_REF] Bdiri | Learning inverted dirichlet mixtures for positive data clustering[END_REF]. This extension was based on the consideration of Dirichlet processes which have been widely used in the case of nonparametric Bayesian approaches [START_REF] Bouguila | A dirichlet process mixture of generalized dirichlet distributions for proportional data modeling[END_REF][START_REF] Bouguila | A nonparametric bayesian learning model: Application to text and image categorization[END_REF]. Despite its advantages and flexibility, the inverted Dirichlet has a very restrictive covariance structure that is generally violated by data generated from real-life applications. Thus, we propose an alternative to the inverted Dirichlet namely the generalized inverted Dirichlet (GID) that has a more general covariance structure. Our work can be viewed as a principled and natural extension to the framework developed in [START_REF] Bdiri | Learning inverted dirichlet mixtures for positive data clustering[END_REF], since we consider the GID within an infinite mixture model by taking feature selection into account. The feature selection process is formalized by introducing a background distribution, common to all mixture components, into the infinite model to represent irrelevant features. Moreover, we develop an algorithm for the learning of the resulting model using Markov chain Monte Carlo (MCMC) sampling techniques namely Gibbs sampling and Metropolis-Hastings [START_REF] Marin | Bayesian Core: A Practical Approach to Computational Bayesian Statistics[END_REF]. The paper is organized as follows: Section 2 presents our infinite mixture model. Section 3 provides empirical evaluation based on the challenging problem of images categorization. Finally, Section 4 concludes the paper.

The Model

In this section, we start by presenting the finite GID mixture model, then its infinite counterpart is developed. A feature selection approach is proposed, also.

Finite Model

Let us consider a data set

Y = (Y 1 , Y 2 , . . . , Y N ) of N D-dimensional positive vectors, where Y i = (Y i1 , . . . , Y iD ), i = 1, . . . , N . We assume that Y i follows a mixture of M GID distributions: p(Y i |Θ) = M j=1 π j p(Y i |Θ j ) (1)
where p(Y i |Θ j ) is a GID distribution [START_REF] Lingappaiah | On the generalised inverted dirichlet distribution[END_REF]:

p(Y i |Θ j ) = D l=1 Γ (α jl + β jl ) Γ (α jl )Γ (β jl ) Y α jl -1 il T ηjl il (2)
where T il = 1 + l k=1 Y ik and η jl = β jl + α jl -β j(l+1) with β j(D+1) = 0. Each Θ j = (α j1 , β j1 , α j2 , β j2 , . . . , α jD , β jD ) is the set of parameters defining the jth component, and π j is the mixing weight of the jth cluster. Of course, being probabilities, the π j must satisfy: π j > 0, j = 1, . . . , M , and

M j=1 π j = 1.
In mixture-based clustering [START_REF] Mclachlan | Finite Mixture Models[END_REF], each vector Y i is assigned to all classes with different posterior probabilities p(j|Y i ) ∝ π j p(Y i |Θ j ). It is possible to show that the properties of the GID distribution allows the factorization of the posterior probabilities as:

p(j|Y i ) ∝ π j D l=1 p ib (X il |θ jl ), where X i1 = Y i1 and X il = Y il 1+ D l=1 Y il for l > 1, p ib (X il |θ jl ) is an inverted Beta distribution with θ jl = (α jl , β jl ), l = 1, . . . , D: p ib (X il |α jl , β jl ) = Γ (α jl + β jl ) Γ (α jl )Γ (β jl ) X α jl -1 il (1 + X il ) -α jl -β jl
Thus, the clustering structure underlying Y is the same as that underlying X = (X 1 , . . . , X N ) described by the following mixture model with conditionally independent features:

p(X i |Θ) = M j=1 π j D l=1 p ib (X il |θ jl ) (3)
This means that GID mixture model has the ability to reduce complex multidimensional clustering problems to a sequence of one-dimensional ones.

Infinite model

Let Z i be a variable indicating from which cluster each vectors X i arose (i.e Z i = j means that X i comes from component j), thus π j = p(Z i = j), j = 1, . . . , M and

p(Z|P ) = M j=1 π nj j (4)
where

P = (π 1 , . . . , π M ), Z = (Z 1 , . . . , Z N ), n j = N i=1
I Zi=j is the number of vector in cluster j. It is common to consider a Dirichlet distribution as a prior for P which is justified by the fact that the Dirichlet is conjugate to the multinomial [START_REF] Bouguila | On discrete data clustering[END_REF]:

p(P |η 1 , . . . , η M ) = Γ ( M j=1 η j ) M j=1 Γ (η j ) M j=1 p ηj -1 j (5)
where (η 1 , . . . , η M ) ∈ R + M are the parameters of the Dirichlet. By taking η j = η M , j = 1, . . . , M , where η ∈ R + , we obtain

p(P |η) = Γ (η) Γ ( η M ) M M j=1 p η-1 j (6)
Because the Dirichlet is a conjugate prior to the multinomial, we can marginalize out P :

p(Z|η) = P p(Z|P )p(P |η)dP = Γ (η) Γ (η + N ) M j=1 Γ ( η M + n j ) Γ ( η M )
which can be considered as a prior on Z. We have also

p(P |Z, η) = p(Z|P )p(P |η) p(Z|η) = Γ (η + N ) M j=1 Γ ( η M + n j ) M j=1 p nj + η M -1 j (7)
which is a Dirichlet distribution with parameters (n 1 + η M , . . . , n M + η M ) from which we can show that:

p(Z i = j|η, Z -i ) = n -i,j + η M N -1 + η (8)
where

Z -i = {Z 1 , . . . , Z i-1 , Z i+1 , . . . , Z N }, n -i,j
is the number of vectors, excluding Y i , in cluster j. Letting M → ∞ in Eq. 8, the conditional prior gives the following limits [START_REF] Rasmussen | The infinite gaussian mixture model[END_REF] p

(Z i = j|η, Z -i ) = n-i,j N -1+η if n -i,j > 0 (cluster j ∈ R) η N -1+η if n -i,j = 0 (cluster j ∈ U) (9)
where R and U are the sets of represented and unrepresented clusters, respectively. The previous equation describes actually a Dirichlet process of mixtures which learning is generally based on the MCMC technique of Gibbs sampling [START_REF] Bdiri | An infinite mixture of inverted dirichlet distributions[END_REF] by generating the assignments of vectors according to the posterior distribution

p(Z i = j|Z -i , X ) ∝ p(Z i = j|Z -i ) p(X i |Z i = j, Θ j )p(Θ j |Z -i , X -i )dΘ j (10)
where Z -i represents all the vectors assignments except Z i and X -i represents all the vectors except X i . In order to obtain the conditional posterior distributions of our infinite model's parameters given the data that we would like to cluster, we need to choose appropriate priors. Here, we consider the same priors previously proposed in [START_REF] Bdiri | An infinite mixture of inverted dirichlet distributions[END_REF] for the inverted Dirichlet which is actually the multivariate case of the inverted Beta in Eq. 3. Thus, we need to parametrize the inverted Beta as following:

p ib (X il ||α jl |, µ jl ) = Γ (|α jl |) Γ (µ jl |α jl |)Γ ((1 -µ jl )|α jl |) X µ jl |α jl |-1 il (1 + X il ) -|α jl | (11)
where

|α jl | = α jl +β jl , µ jl = α jl |α jl |
, and for which we impose independent uniform and inverse Gamma priors, respectively:

p(µ jl ) ∼ U jl [0,1] p(|α jl ||σ, ̟) ∼ ̟ σ exp(-̟/|α jl |) Γ (σ)|α jl | σ+1 (12)
where σ and ̟ are hyperparameters, common to all components, representing shape and scale of the distribution, respectively, and for which we consider the following priors to add more flexibility to the model:

p(σ|λ, δ) ∼ δ λ exp(-δ/σ) Γ (λ)σ λ+1 p(̟|φ) ∼ φ exp(-φ̟) (13)
Having all our priors in hand, the calculation of the parameters posteriors given the rest of the variables becomes straightforward:

p(|α jl || . . .) ∝ ̟ σ exp(-̟/|α jl |) Γ (σ)|α jl | σ+1 Zi=j p(X i |Θ) (14)
p(µ j | . . .) ∝ Zi=j p(X i |Θ) (15)
p(σ| . . .) ∝ ̟ Mσ δ λ exp(-δ/σ) Γ (σ) M Γ (λ)σ λ+1 M j=1 exp(-̟/|α jl |) |α jl | σ+1 (16)
p(̟| . . .) ∝ ̟ Mσ φ exp(-φ̟) Γ (σ) M M j=1 exp(-̟/|α jl |) |α jl | σ+1 (17)
With these posteriors, the learning algorithm can be summarized as follows:

-Initialization.

-Generate Z i from Eq. 10, i = 1, . . . , N using the algorithm in [START_REF] Neal | Markov chain sampling methods for dirichlet process mixture models[END_REF].

-Update the number of represented components M .

-Update n j and π j = nj N +η , j = 1, . . . , M . -Update the mixing parameters of unrepresented components π U = η η+N . -Generate |µ jl | from Eq. 15 and |α jl | from Eq. 14, j = 1, . . . , M using Metropolis-Hastings [START_REF] Marin | Bayesian Core: A Practical Approach to Computational Bayesian Statistics[END_REF]. -Update the hyperparameters: Generate σ from Eq. 16 and ̟ from Eq. 17 using adaptive rejection sampling as proposed in [START_REF] Gilks | Algorithm as 287: Adaptive rejection sampling from logconcave density functions[END_REF].

Note that in the initialization step, the algorithm starts by assuming that all the vectors are in the same cluster and the initial parameters are generated as random samples from their prior distributions.

Feature Selection

It is noteworthy that the model proposed in the previous section does not take into account the fact that different features may have different weights in the clustering structure, and that some features may be noise and then compromise the generalization capabilities of the model [START_REF] Fan | Unsupervised hybrid feature extraction selection for high-dimensional non-gaussian data clustering with variational inference[END_REF]. In order to introduce feature selection in our model, it is possible to use the following formulation:

p(X i |Ξ) = M j=1 π j D l=1 ρ l p ib (X il ||α jl |, µ jl ) + (1 -ρ l )p ib (X il ||α irr jl |, µ irr jl ) ( 18
)
where Ξ = {Θ, ρ, Θ irr } is the set of all the model parameters, ρ = (ρ 1 , . . . , ρ D ), Θ irr = {|α irr jl |, µ irr jl }, and p ib (X il ||α irr jl |, µ irr jl ) is a background distribution, common to all mixture components, to represent irrelevant features. ρ l = p(z il = 1) represents the probability that the l th feature is relevant for clustering where z il is a hidden variable equal to 1 if the l th feature of X i is relevant and 0, otherwise. By introducing feature selection, the learning algorithm proposed in the previous section has to be slightly modified by adding simulations from the posteriors of |α irr jl |, µ irr jl , for which we choose the same priors considered for |α jl |, µ jl , and ρ for which we consider a Beta prior with location δ 1 and scale δ 2 common to all dimensions:

p(ρ|δ 1 , δ 2 ) = Γ (δ 2 ) Γ (δ 1 δ 2 )Γ (δ 2 (1 -δ 1 )) D D d=1 ρ δ1δ2-1 d (1 -ρ d ) δ2(1-δ1)-1 (19)
Moreover, the z i are generated from a D-variate Bernoulli distribution with parameters (ẑ i1 , . . . , ẑiD ), where ẑil =

ρ l p ib (X il ||α jl |,µ jl ) ρ l p ib (X il ||α jl |,µ jl )+(1-ρ l )p ib (X il ||α irr jl |,µ irr
jl ) denotes the expectation for z il :

p(z|ρ) = N i=1 D d=1 ρ z id d (1 -ρ d ) 1-z id = D d=1 ρ f d d (1 -ρ d ) N -f d (20)
where

f d = N i=1 I z id =1 . Then, the posterior for ρ is p(ρ| . . .) ∝ p(ρ|δ 1 , δ 2 )p(z|ρ) ∝ D d=1 ρ δ1δ2+f d -1 d (1 -ρ d ) δ2(1-δ1)+N -f d -1 (21)
Note that the feature selection process starts by assuming that all features have a probability of 0.5 to be relevant, then this relevancy value is updated during the learning iterations.

Experimental Results: Images Categorization

In this section we demonstrate the utility of our model by applying it on a challenging application namely visual scenes categorization. Moreover, we compare the proposed approach with the infinite inverted Dirichlet proposed in [START_REF] Bdiri | Learning inverted dirichlet mixtures for positive data clustering[END_REF]. Comparing our results with many other generative and discriminative techniques is clearly out of the scope of this paper. In this application, the values of the hyperparameters have been set experimentally to one. This choice has been found reasonable according to our simulations. The wealth of images generated everyday has spurred a tremendous interest in developing approaches to understand the visual content of these images. In this section, we shall focus on the challenging problem of images categorization, to validate our GID infinite mixture model, which is a crucial step in several applications such as annotation [START_REF] Benitez | Semantic knowledge construction from annotated image collections[END_REF][START_REF] Chang | Cbsa: content-based soft annotation for multimodal image retrieval using bayes point machines[END_REF], retrieval [START_REF] He | Manifold-ranking based image retrieval[END_REF][START_REF] Wang | Multi-model similarity propagation and its application for web image retrieval[END_REF], and object recognition [START_REF] Spirkovska | Higher-order neural networks applied to 2d and 3d object recognition[END_REF].

A common recent approach widely used for images categorization, that we follow in this application, is the consideration of the so-called bag of visual words generated via quantization of local image descriptors such as SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF].

We considered two challenging datasets in our experiments namely the 15 class scene recognition data set [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF] and the 8 class sport events data set [START_REF] Li | What, where and who? classifying events by scene and object recognition[END_REF]. The 15 class scene recognition data set contains the following categories: coasts (360 images), forest (328 images), mountain (374 images), open country (410 images), highway (260 images), inside of cities (308 images), tall buildings (356 images), and streets (292 images), suburb residence (241 images), bedroom (174 images), kitchen (151 images), livingroom (289 images), and office (216 images), store (315 images), and industrial (311 images). Figure 1 displays examples of images from this data set. The 8 class sports event dataset contains the following cate- gories: rowing (250 images), badminton (200 images), polo (182 images), bocce (137 images), snowboarding (190 images), croquet (236 images), sailing (190 images), and rock climbing (194 images). Figure 2 displays examples of images from this data set. We construct our visual vocabulary for each data set, from half of the available images in each data set, by detecting interest points from these images using the difference-of-Gaussians point detector, since it has shown excellent performance [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. Then, we used SIFT descriptor [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF], computed on 1. This table clearly shows that the GID infinite mixture outperforms the infinite inverted Dirichlet mixture. The results can be explained by the fact that the GID is more flexible than the inverted Dirichlet. We can clearly notice, also, that introducing feature selection improves further the results. Clustering plays a crucial role in various data mining and knowledge discovery applications. The majority of existing clustering algorithms, however, either assume that clusters follow Gaussian distributions; or are very sensitive to the presence of irrelevant features. In this paper we have proposed a new clustering algorithm devoted to positive data that is robust to irrelevant features, and identifies automatically clusters having non-Gaussian distributions. Our approach achieves this by representing the data using an infinite mixture model of GID distributions in which a feature weighting component is introduced. Feature selection is introduced in order to remove irrelevant features that may compromise the clustering process. Our simulations based on the challenging problem of images categorization have shown the efficiency of the proposed model. A potential future work could be the development of a variational approach, like the one proposed in [START_REF] Fan | Unsupervised hybrid feature extraction selection for high-dimensional non-gaussian data clustering with variational inference[END_REF], to improve the learning of our model from a computational point of view. Several other directions present themselves for future efforts. Indeed, the developed approach could be applied to many real-world problems such as 3D object recognition [START_REF] Selinger | A perceptual grouping hierarchy for appearance-based 3d object recognition[END_REF] or to the generation of SVM kernels using the methodology recently proposed in [START_REF] Bdiri | Bayesian learning of inverted dirichlet mixtures for svm kernels generation[END_REF].

(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) (o)

Fig. 1 .

 1 Fig. 1. Sample images from each group in the 15 class scene recognition data set. (a) Highway, (b) Inside of cities, (c) Tall buildings, (d) Streets, (e) Suburb residence, (f) Forest, (g) Coast, (h) Mountain, (i) Open country, (j) Bedroom, (k) Kitchen, (l) Livingroom, (m) Office, (n) Store, (O) Industrial.

Fig. 2 .

 2 Fig. 2. Sample images from each group in the 8 class sports event dataset . (a) rowing, (b) badminton, (c) polo, (d) bocce, (e) snowboarding, (f) croquet, (g) sailing, (h) rock climbing.

Table 1 .

 1 Classification performance (%) obtained for the two tested data sets using three different approaches.

	GID GID + feature selection Inverted Dirichlet
	Data set 1 (15 categories) 74.52	75.31	70.11
	Data set 2 (8 events) 73.25	74.03	70.72


Guruh Fajar Shidik

Khabib Mustofa
email: khabib@ugm.ac.id

Predicting Size of Forest Fire using Hybrid Model

Keywords: Forest fire Prediction, FCM, Back-Propagation Neural Network, Data Mining

This paper outlines a hybrid approach in data mining to predict the size of forest fire using meteorological and forest weather index (FWI) variables such as Fine Fuel Moisture Code (FFMC), Duff Moisture Code (DMC), Drought Code (DC), Initial Spread Index (ISI), temperature, Relative Humidity (RH), wind and rain. The hybrid model is developed with clustering and classification approaches. Fuzzy C-Means (FCM) is used to cluster the historical variables. The clustered data are then used as inputs to Back-Propagation Neural Network classification. The label dataset having value greater than zero in fire area size are clustered using FCM to produce two categorical clusters,i.e.: Light Burn, and Heavy Burn for its label. On the other hand, fire area label with value zero is clustered as No Burn Area. A Back-Propagation Neural Network (BPNN) is trained based on these data to classify the output (burn area) in three categories, No Burn Area, Light Burn and Heavy Burn. The experiment shows promising results depicting classification size of forest fire with the accuracy of confusion matrix around 97, 50 % and Cohens Kappa 0.954. This research also compares the performance of proposed model with other classification method such as SVM, Naive Bayes, DCT Tree, and K-NN that showed BPNN have best performance.

Introduction

Forest fire is a common natural world phenomenon. Every year millions of hectares of forests in the world are destroyed [START_REF] Alonso-Betanzos | An intelligent system for forest fire risk prediction and fire fighting management in galicia[END_REF], between 1980 -2007 at least 2.7 million hectares were burnt in Portugal [START_REF] Cortez | A data mining approach to predict forest fires using meteorological data[END_REF]. This caused severe damages to the natural environment and resulted in loss of precious human lives. Forest fire is one of the major environmental concern that affects the preservation of forests, resulting in economical and ecological damage that causes human suffering.

Referring to Elmas [START_REF] Elmas | A data fusion framework with novel hybrid algorithm for multi-agent decision support system for forest fire[END_REF], quick fire detection and response are effective ways in reducing the damages caused by forest fires. Various studies have been made in order to improve early fire prediction and detection systems that helps to develop response strategies during the fire. It means, one of the key successes of putting out forest fire is by providing an early warning detection. Early warning detection is related to accurate prediction of results based on determined parameters. There are three trending techniques that could be used in predicting forest fire such as the use of satellite data, infra red or smoke scanners and local sensors, for example, using the meteorological ones [START_REF] Cortez | A data mining approach to predict forest fires using meteorological data[END_REF].

Safi et al [START_REF] Safi | A neural network approach for predicting forest fires[END_REF] tried to overcome forest fire impacts by making prediction using data mining technique. A future event has always been considered a mysterious activity scientists trying to treat into scientific activities based on theories and models. Predictions in data mining can be used in identifying many real world problems such as financial forecasting and prediction of environmental applications or to test scientific understanding of the behaviour of complex systems or phenomena. The predictions are also used as a guide or basis for decision making [START_REF] Cortez | A data mining approach to predict forest fires using meteorological data[END_REF].

In [START_REF] Iliadis | An intelligent system employing an enhanced fuzzy c-means clustering model: Application in the case of forest fires[END_REF], based on the perspectives of forest fire, it is mentioned that several scientists around the world had utilized statistical approaches such as regression analysis, probabilistic analysis and artificial intelligence. Some data mining techniques have been applied in the domain of fire detection, for example by adopting meteorological data to predict forest fire [START_REF] Cortez | A data mining approach to predict forest fires using meteorological data[END_REF]. Back propagation neural network and the rule generation approach [START_REF] Yu | Pattern clustering of forest fires based on meteorological variables and its classification using hybrid data mining methods[END_REF], fuzzy c-means clustering application in the case of forest fire [START_REF] Iliadis | An intelligent system employing an enhanced fuzzy c-means clustering model: Application in the case of forest fires[END_REF], artificial neural network to the real word problem of predicting forest fire [START_REF] Safi | A neural network approach for predicting forest fires[END_REF], Neural Network (NN) and Support Vector Machines to predict forest fire occurrence based on weather data [START_REF] Sakr | Efficient forest fire occurrence prediction for developing countries using two weather parameters[END_REF], decision tree algorithm namely C4.5 to extract a forest fire data and classifying hotspot occurrences [START_REF] Sitanggang | Hotspot occurrences classification using decision tree method: Case study in the rokan hilir, riau province, indonesia[END_REF].

This research aims at proposing an approach for predicting the size of forest fire occurs based on meteorological and forest weather index dataset consisting of eight variables: FFMC, DMC, DC, ISI, temperature, RH, wind and rain. The size of forest fire will be classified using Back-Propagation Neural Network into three categories,i.e: No Burn Area, Light Burn and Heavy Burn. As the label (area) in datasets of size of forest fire is numerical, before classification process, the data should be clustered into three classes. We split the dataset into two part: the data with zero value and data having value greater than zero. The process of clustering two categories dataset (light and heavy burn) is done with unsupervised method FCM in label data (area) that have value greater than zero, while for No Burn Area, it is done by selecting label (area) that have value zero. We used ten fold cross validations in separating training dataset and testing dataset with shuffled and stratified sampling. Confusion matrix and Kappa is used in evaluating the performance of the model. The remaining of this paper will be organised as follows: chapter two talks about related works, chapter three talks about fundamentals of the approach, chapter four describe the research method used to predict the size of forest fire. The rest of the papers are discussion on the results and, the last chapter is conclusion and future work of this research. Satoh et al [START_REF] Satoh | A study of forest fire danger prediction system in japan[END_REF] developed a system for predicting the dangers of a forest fire. A simulation of dangers related to forest fire was developed not only using the previous weather condition, but also coupled with data on population density and some other factors. [START_REF] Cortez | A data mining approach to predict forest fires using meteorological data[END_REF] used five different data mining techniques to predict the burnt area of forest fire using Support Vector Machines (SVM) and Random Forests. With four distinct features likes spatial, temporal, Fire Weather Index components and weather variables (such as temperature, relative humidity, rain and wind), it was found that the best configuration was reached using Support Vector Machine, which is capable of predicting the frequent burnt areas due to small fire.

Cortez and Morais

A study to increase the Fuzzy C-means model intelligently using a flexible termination criteria for the clustering of forest fire was conducted by Illadis et al [START_REF] Iliadis | An intelligent system employing an enhanced fuzzy c-means clustering model: Application in the case of forest fires[END_REF]. This approach enables the algorithm to be more flexible and humanlike in an intelligent way. It also avoids possible infinite loops and unnecessary iterations.

Decision tree C4.5 algorithm is implemented to predict the location of the incident hotspots in Rokan Hilir district, Riau province, Indonesia [START_REF] Sitanggang | Hotspot occurrences classification using decision tree method: Case study in the rokan hilir, riau province, indonesia[END_REF]. The dataset consists of hotspot locations, human activity factors, and land cover types. The human activity factors include city center locations, road network and river network.

Safi et al [START_REF] Safi | A neural network approach for predicting forest fires[END_REF] applied artificial neural networks to the real world problem of predicting forest fire, using back propagation learning algorithm. Yu et al [START_REF] Yu | Pattern clustering of forest fires based on meteorological variables and its classification using hybrid data mining methods[END_REF] conducted a research investigating the nonlinear relationship between the size of a forest fire and meteorological variables (temperature, relative humidity, wind speed and rainfall) using two hybrid approaches. At first phase Self Organizing Map is used to cluster the data. Than, in second phase the clustered data were used as inputs for two different approaches, the back-propagation neural network and the rule generation approaches. Sakr et al [START_REF] Sakr | Artificial intelligence for forest fire prediction[END_REF] applied a description and analysis of forest fire prediction methods based on Support Vector Machines to predict the fire hazard level of a day, where the algorithm depended on previous weather conditions. Moreover, in [START_REF] Sakr | Efficient forest fire occurrence prediction for developing countries using two weather parameters[END_REF], Sakr et al try to reduced a set of weather parameters utilizing relative humidity and cumulative precipitation to estimate the risk of the output, to predict the occurrence of forest fire by comparing two artificial intelligence-based methods: Artificial Neural Networks (ANN) and Support Vector Machines (SVM).

Based on the above existing researches, this paper proposes an approach to predict the size of forest fire using hybrid model, between Fuzzy C-Means (FCM) clustering technique and Back-Propagation Neural Network (BPNN) classification technique in processing meteorological and forest weather index data as input.

Fundamentals

Forest Fire

Forest fire as a kind of common natural disaster possibly makes a great danger to people living in the burnt forest as well as to wildlife. Such disaster may be caused by lightning, human negligence or arson that can burn thousands of square kilometers. According to Brown and Davis [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations[END_REF], there are three types of forest fire namely: ground fire, surface fire and crown fire.

Data Mining

Data mining can be seen as a process of discovering patterns in large volume of data having meaningful information [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations[END_REF]. The process must be automatic or (more usually) semi-automatic. Among several existing methods commonly applied in data mining, in this research, clustering and classification are chosen to be implemented. Clustering technique is used to cluster the size of forest burning size area having value greater than zero into four clusters, while classification technique is to determine which type of burning size that will probably occur based on meteorological data.

Fuzzy C-Means (FCM) is one of popular fuzzy clustering techniques that used for finding similarities in data and putting similar data into several groups, has been proposed by Dunn [START_REF] Dunn | A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters[END_REF] in 1973 and then later modified by Bezdek [START_REF] Bezdek | Pattern recognition with fuzzy objective function algorithms[END_REF] in 1981. It is an approach where the data points have their membership values with cluster center, to be updated iteratively. The detail explanation of FCM algorithm, could be seen at [START_REF] Chattopadhyay | A comparative study of fuzzy c-means algorithm and entropy-based fuzzy clustering algorithms[END_REF].

Back-Propagation Neural Network (BPNN) is classification technique highly dependent on the network structure and training process that has better learning rate [START_REF] Singh | Neural network based handwritten hindi character recognition system[END_REF]. The number of input layer nodes, hidden layer and output layer in BPNN will determine the structure of the network.

Back-propagation learning process requires a pair of input vectors and the target vectors. The output vector of each input vector will be compared with the target vector. This measurement is necessary in order to minimize the difference between the output vector and the target vector.

In BPNN it begins with the initialization of weights and thresholds at random. The weights are updated in each iteration to minimize the Mean Square Error (MSE) between the output vector and the target vector, where the detail information of BPNN was explain in [START_REF] Eleyan | PCA and LDA based Neural Networks for Human Face Recognition[END_REF].

Preprocessing

There are several steps in data mining preprocessing [START_REF] Han | Data mining : concepts and techniques[END_REF], such as data cleansing, data integration, data reduction and data transformation. In this research, data transformation is used to normalize the data. Data normalization is useful for classification involving neural networks or distance measurements such as nearest neighbour classification and clustering. Beside that, it can affect to speed up the learning rate of BPNN for classification. In this research Min-max normalization is applied to perform a linear transformation on the original data [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations[END_REF], where the formula could be seen at [START_REF] Alonso-Betanzos | An intelligent system for forest fire risk prediction and fire fighting management in galicia[END_REF]. The data of eight variables or attribute used in this research will be transform in new range with min value is 0 and max value is 1.

v i = v i -min A max A -min A (N ew max A -N ew min A ) + N ew min A (1)
Where min A is existing minimum value and maxA is existing maximum values of an attribute A. vi is existing data value in attribute A that will be mapped to current data value v i in the new range [0 , 1] [N ew min A , N ew max A ].

Research Method

Fig. 1 depicts the overall process in this research, describing the position the proposed hybrid model in predicting size of forest fire.

Data Collection

Data on forest fire are collected from the study by Cortez and Morais, available in the UCI machine learning repository [START_REF] Cortez | A data mining approach to predict forest fires using meteorological data[END_REF] . The dataset contains 12 variable with their respective labels, forest fire weather index (FWI) components in Montesano Natural Park, a northeast region of Portugal. Weather observations are collected by Braganza Polytechnic Institute and integrated to the forest fire dataset. The park was divided into 81 distinct locations by placing a 9×9 grid onto the map of the park. The dataset has a total of 517 samples, from year 2000 until 2007. This research only select 8 variables to be considered: FFMC, DMC, DC, ISI, Temperature, RH, Wind and Rain.

Splitting the Dataset

In this steps, the dataset is split into two categories. The process of splitting data is conducted by selecting label dataset(Area). The label (Area) that have value zero, it means have not any total burn area size will be separate from label (Area) that have value more than zero as showed in Fig. 2. After that, all data with zero value will be categorized as data "No Burn Area". Otherwise, the label (Area) data with value more than zero will be cluster by FCM to categorized as data Light Burn or Heavy Burn.

Normalization

After splitting dataset into two categories between dataset that have value zero or more than zero in attribute label, we continue with the process of normalization. Normalization process in this research uses equation ( 1) with min max normalization. The normalization process, only transform 8 variables that will be used in clustering and classification process such as FFMC, DMC, DC, ISI, temperature, RH, wind, and rain. This process results in minimum and maximum values between [0, 1] in dataset. The sample of process before and after normalization could be seen in Fig. 3 and Fig. 4.

Fuzzy C-Means Clustering Dataset

The process of categorizing dataset into two categorise of size of fire is done in this phase. Fuzzy C-Means here will cluster the data based on eight Meteorological variables. Since FCM is unsupervised method, it will automatically categorise the dataset into two categorise by default: cluster 0 (as Light Burn) and custer 1 (as Heavy Burn).

We observed several distance similarity measurements algorithm in FCM such as Correlation Similarity, Cosine Similarity, Dice Similarity, Inner Product Similarity, Jaccard Similarity, Overlap Similarity, Kernel Euclidian Distance, Manhattan Distance, Chebychev Distance, Euclidean Distance, Canberra Distance, Dynamic Time Warping Distance to achieve the best performance of classification BPNN .

Merge Dataset

After clustering process has been done, the data that has been categorize as No Burn Area will be merged with the data that has been cluster by FCM. Therefore, after this process we will have the dataset that contain label No Burn Area, Light Burn, and Heavy Burn. As you can see in Fig. 5. The architecture of Back-Propagation Neural Network in this research uses only one hidden layer, where the learning rate has been fixed at β = 0.3 and the maximum number of iteration is α = 500. The detail steps of BPNN could be seen at [START_REF] Eleyan | PCA and LDA based Neural Networks for Human Face Recognition[END_REF]. Fig. 6 is showed the architecture of BPNN.

Result Evaluation and Discussion

Performance Measurement

After classification process, to assess the performance results of our proposed hybrid method for predicting forest burning size, we used confusion matrix [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations[END_REF] to measures accuracy of classifier can be calculated by equation ( 2) and Cohen's Kappa statistic measurement [START_REF] Byrt | Bias, prevalence and kappa[END_REF] to assess inter-rater reliability when observing categorical variables can be calculate by equation [START_REF] Elmas | A data fusion framework with novel hybrid algorithm for multi-agent decision support system for forest fire[END_REF]. The overall approach of experiment in this study is different to the existing work done by Cortez and Morais [START_REF] Cortez | A data mining approach to predict forest fires using meteorological data[END_REF] that also used same dataset. However, in their study used twelve variables which our approach used eight variables. Besides that, they only evaluate pure prediction methods such as Neural Network, SVM, Naive Bayes, Multiple Regression and Decision Trees without combining cluster methods that provide burn area prediction in numerical results without categorizing the type of result forest burning size.

Accuracy = TP + TN T P + T N + F P + F N (2) Kappa = Observed Agreement-Expected Agreement 1-Expected Agreement (3)

Conclusion

This research has proposed an alternative hybrid model capable of predicting the size of forest fire by combining Fuzzy C-Means and Back-Propagation Neural Network method. The model which incorporates meteorological and forest weather index variables (FFMC, DMC, DC, ISI, temperature, RH, wind and rain) has been shown to be successfully classify the level of burning into three categories: No Burn Area, Light Burn and Heavy Burn. The evaluation of the proposed model has showed promising results with accuracy of confusion matrix around 97.50% and Kappa 0.961. It is also found that cosine similarity method in FCM shows better performance than other similarity distance measuring algorithms under simulation. For the future work, the model will be implemented as web services and integrated with meteorological sensor to build early warning of forest fire prediction system.

Fig. 1 .

 1 Fig. 1. Research Method Outline

Fig. 2 .Fig. 3 .Fig. 4 .

 234 Fig. 2. Sample of Process Split Dataset

Fig. 5 .

 5 Fig. 5. Sample of Merge Dataset

Fig. 6 .

 6 Fig. 6. Back-Propagation Neural Network Architecture

Fig. 8 .

 8 The proposed Hybrid model for predicting the size of forest fire indicates a promising result. Compared with other methods such as SVM, K-NN and DCT Tree, the proposed method is still showing better performance, more over Naive Bayes and Random Forest have lowest performance classification with accuracy less than 74% and Cohens Kappa 0.54.

Fig. 7 .Fig. 8 .

 78 Fig. 7. Results of Accuracy Confusion Matrix Performance

Table

Table 1 .

 1 1 and Table 2, it is shown performance of Back-Propagation neural network could achieve best results with accuracy around 97.50% and index of Cohens Kappa 0.961. The best performance of proposed model are gathered with combination Performance of Hybrid Model in Stratified Sampling

	Type of Distance Similarity in FCM	Accuracy Kappa
	Correlation Similarity	95.74%	0.933
	Cosine Similarity	97.10%	0.954
	Dice Similarity	96.91%	0.945
	Inner Product Similarity	96.71%	0.936
	Jaccard Similarity	96.91%	0.945
	Overlap Similarity	91.30%	0.861
	Kernel Euclidian Distance	96.14%	0.938
	Manhatan Distance	96.13%	0.935
	Chebychev Distance	95.74%	0.930
	Euclidean Distance	96.33%	0.941
	Canberra Distance	93.42%	0.897
	Dynamic Time Warping Distance	86.26%	0.780

Table 2 .

 2 Performance of Hybrid Model in Shufled Sampling

	Type of Distance Similarity in FCM	Accuracy Kappa
	Correlation Similarity	95.75%	0.932
	Cosine Similarity	97.50%	0.961
	Dice Similarity	97.30%	0.952
	Inner Product Similarity	96.34%	0.927
	Jaccard Similarity	97.30%	0.952
	Overlap Similarity	89.56%	0.833
	Kernel Euclidian Distance	96.92%	0.951
	Manhatan Distance	96.90%	0.947
	Chebychev Distance	96.91%	0.949
	Euclidean Distance	96.92%	0.951
	Canberra Distance	92.65%	0.884
	Dynamic Time Warping Distance	84.56%	0.754

of BPNN with FCM that used Cosine Similarity. Besides that, to show the performance of BPNN, we also compare the performance of another classification method such as SVM, KNN, DCT, and Naive Bayes, that include with same clustering technique FCM in categorizing the dataset. The results comparison could be seen at

Fig 7 and