Hamid Jahankhani
email: hamid.jahankhani@uel.ac.uk

Elias Pimenidis
email: e.pimenidis@uel.ac.uk

Amin Hosseinian-Far

A Probabilistic Knowledge-Based Information System for Environmental Policy Modeling and Decision Making

Keywords: Knowledge Management System, Decision Support System, London Plan, Influence Diagram, Tacit and Explicit Knowledge

Decision making for setting new policies is a challenging process as the current policy making system is utterly flawed. A policy is introduced by the decision maker when the problem domain was fully consulted by experts in the field. Not always all the consultants and advisers agree on details or even basics of such a course of action. The need for an intelligent predictive system is emerging. Policy making on environmental issues are even shoddier as the environmental systems are habitually complex, and adaptive; and introduction of new technologies can easily affect the guiding strategies already taken. This paper outlines the principles of Knowledge Management Systems. It then reflects on Influence Diagrams' suitability for construction of such an information system through the use of the London Plan case study. An application of such a system is outlined by means of a probabilistic knowledge based IS which is developed by Influence Diagrams and can be utilized as an Environmental policy modeler and/or DSS.

Introduction to KMS

There are various applications for Knowledge Management Systems (KMS) theories including but not limited to Distributed Databases, Ontology, and Artificial Intelligence (AI). Knowledge Based systems which are one of the software tools in Intelligent Systems are the instruments to manage the knowledge [START_REF] Valente | Artificial Intelligent Methods in Operational Knowledge Management[END_REF]. The main focus would be on the Artificial Intelligence in this research.

Knowledge Management Systems can be classified into two main categories; Distributive and Collaborative aplications. Distributive knowledge management system is where database plays a vital role in shaping the knowledge base, hence there is a structured formal database involve. On the other hand the collborative knowledge management system partakes an informal internal knowledge [START_REF] Zack | Knowledge management and Collaboration Technologies[END_REF]. Some experts still argue on the defintion of knowledge itself. knowledge is summation of information, skills, experience, and personal capabilities [START_REF] Baker | Leveraging human capital[END_REF]. Other scholars also attempted to categorize and formulated knowledge and its types. Another categorization of knowledge is outlined in Fig. 1: Defining knowledge is not enough as the knowledge should be retained and gathered. There are various frameworks for knowledge acquisition. The common knowledge acquisition framework is illustrated in Fig. 2: According to this general framework, the first step is the declarative stage where the rules, standards, examples and definitions should be clearly defined. Once the standards are declared, the problem solving and encoding stage begins, still within the declarative stage. Up to this point the knowledge is still in the declarative knowledge acquisition mode. In the start of the procedural stage, compilation should be performed on the declarative acquitted knowledge. Once the errors are fixed and the compilation is tuned, then the procedural knowledge acquisition is completed [START_REF] Mccall | Use of knowledge management Systems and the Impact on the Acquisition of Explicit Knowledge[END_REF].

Furthermore, knowledge can be transformed while in use. The general categorization for knowledge transformation is internal, external and socialized knowledge (Fig. 2). The knowledge is initially in the internal form in tacit format. The tacit format indicates that the knowledge still contains areas where the use is not fully identified. The tacit knowledge within the internal domain will be transformed into explicit knowledge where the standards and usages will be clarified. By having the formal explanations and models the knowledge can be externalized. The externalization on its own does not have to be formal at all circumstances. Informal explanations and observations in the external domain based on the explicit internal knowledge can still provide a means for formalization. The set formal external knowledge which is explicit can be widely used and become a socialised knowledge format [START_REF] Kernstock | A Web-Based knowledge Management System for information technology Education[END_REF].

A dynamic system model which has the learning characteristic, complexity elaboration and non-linear features will be able to correspond to the research questions within the problem domain. The complexity, non-linearity, and adaptive characteristics are the main factors that should be considered for selection of proper tool. Fig. 3 represents different software tools for Artificial Intelligence. As it is preferred to use a predictive design one of the following should be selected for the design. Generally aritificial intelligent systems are divided into two main categories which are Knowledge Based systems and Computational Intelligence systems. Some characteristics of the sustainability can be modeled using other types of system modeling techniques; for instance using cybernetic system approach through state space method for socio-ecological system. The latter has been modeled for the climate change by Schellnhuber [START_REF] Schellnhuber | Earth system' analysis and the second Copernican revolution[END_REF].

Policitcians are having a demanding era in facing the sustainability challenges. Policy making as a rule happens after long periods of consultancies. Not always, all the consultants and polictical advisers come to a common ground in different scenrios. The use of predictive information Systems can be valuable in assisting the decision maker. These ISs have be predictive in a way that can approximate the outcomes of different decisions in a given scenrio. Modeling environmental states is quite a challenging process. The tradeoff between complexity, efficiency and accuracy of the Information System used for development of a DSS is crucial. Use of probabilistic inference is often overlooked. Influence Diagrams practices as the primary means for developing the knowledge base for complex environmental scenarios should be tested. Appraisal of Influence Diagrams and ID evaluation techniques can be tested by means of a case study research design type. The London Plan as a case study has had various discussions since 2004 [START_REF]The London Plan[END_REF]. Using microeconometric theories, the required investment for policy 4A.2 has been calculated between 40 to 80 £Billion. Since the case study has these estimates using other quantitative modeling techniques, using Influence Diagrams and probablistic inference can be validated for these kind of modelings. Bayesian networks can be utilised for development of such a knowledge base, but chance nodes are required for development of a DSS. Therefore, Influence Diagram is a fine choice for the knowledge base construction. The objectives of this research would be:

1. Knowledge management representation of the knowledge body behind the London Plan sustainability domain. The London Plan is the case study used for validation of the proposed model. 2. Identifying an appropriate quantitative approach for modeling the financial domain of The London Plan.

Probabilistic Networks for Knowledge Types

Real-life state of affairs mostly modelled as group of entities demonstrating random variables in a "probabilistic network". Clever graphical illustration of dependence and independence relations between accidental variables is a "probabilistic network". Area of random variables could, for instance, help decision makers to identify the most beneficial decision in a given situation from the basis of a decision support system.

Probabilistic networks processes and symbolize probabilistic data. Representative elements of a probabilistic network are a quantitative and a qualitative element. The qualitative component sets (conditional) belief and independence assertions along with a set of chance variables, informational fondness, and preferred relations [START_REF] Uffe | Bayesian Networks and influence Diagrams: a guide to construction and analysis[END_REF]. Graphical language visually encoded the statements of (conditional) dependence and independence, information preference, and favourite relations. On the other hand, the quantitative component identifies the potency of dependence relations by means of probability and utility theories [START_REF] Lerz | The Case for Using Probabilistic Knowledge in a Computer Chess Program[END_REF].

The illustrative depiction of a probabilistic network, explains knowledge of a problem area in a clear-cut manner [START_REF] Hosseinian-Far | Reflections on Modeling Systems' Resilience. 5th SASTech Intl. Symposium[END_REF]. The illustrative depiction is perceptive and easy to understand, making it an ideal tool for passing statement of domain knowledge among experts, users, and systems. Therefore, the formalism of probabilistic networks is becoming an ever trendier domain knowledge representation for interpretation and decision making under uncertainty [START_REF] Lerz | The Case for Using Probabilistic Knowledge in a Computer Chess Program[END_REF]. In this model, tacit and explicit knowledge are extracted using narrative extraction from plan policy reports and the discussions. The decision node is derived from the policy objectives. Therefore in this scenario, 60% CO2 reduction as the aim objective makes the decision node. The suitability of Influence Diagram is assessed as follows.

Suitability of Influence Diagrams for Environmental Policy Modeling

Despite having various modeling techniques in the field of sustainability, probabilistic inference and Bayesian networks models should be given high priority. The complexity of sustainability scenarios and understanding the systems' resilience and boundary paradox are essential when analyzing and designing the case studies [START_REF] Hosseinian-Far | Reflections on Modeling Systems' Resilience. 5th SASTech Intl. Symposium[END_REF]. Among the quantitative techniques, agents and multi agent systems are quite useful as they can bring together the components of the system using agents' characteristics. Furthermore the artificial intelligence techniques used in multi agent systems would enable the system to adapt itself with the changes faced from outside the boundary. But there is a major drawback with multi-agent systems as they are too expensive for building large Decision Support System applications. Although they can fairly fit fine in a CAS (Complex Adaptive System) methodology in theory [START_REF] Hosseinian-Far | Reflections on the Need for an Improved Quantitative Approach[END_REF], but when it comes to practice, they are not responsive to all of the components. Use of neural networks as the tool for modeling the inputs and the outputs based on a network of linked components sound very functional. But the major drawback with the neural networks is when it comes to its training. Training of the network in a sustainability scenario is almost impossible. The longitudinal study may help for the training, but long intervals and wait time is required for the testing. Hence practically, they are not usable in large complex systems where training is nearly impossible. The system's dynamic models are also other quantitative techniques for sustainability modeling. The difficult nature of developing a system using systems' dynamic, has made it again impossible for the developer to develop the model. System dynamic might be useful for modeling some small trade-offs in the environment where increase in one component would lead to a drop in another. Stella might be a good IDE for these small projects; but again, if the system becomes large with various elements in the knowledge base, analysis, design and development of the model would be unattainable [START_REF] Hosseinian-Far | A Systemic Approach to an Enhanced Model for Sustainability[END_REF].

On the other hand, probabilistic inference, e.g. influence diagrams are fine tools for modeling sustainability. The adaptively of the model is an issue with probabilistic networks as the base for the model, but that can be easily fixed by incorporating an agent based engine. The use of goal oriented knowledge management systems theories would be a good further work for this study. This research study by means of the London Plan case study validates that despite having a complex scenario; modeling with ID would be feasible. The model would be a formal model where replication, testing and validation are possible using the available algorithms. The predictive nature of the Influence Diagrams, very good graphical interface for nonexpert users, and also accurate mathematical and probability layer would be one of the finest approaches to model sustainability systems with complex nature. It still fit into the Complex Adaptive System structure if the agents are incorporated to give the adaptively feature to the system. The use of knowledge management theories would help to build the knowledge base for the model [START_REF] Hosseinian-Far | Financial Assessment of London Plan Policy 4A.2 by Probabilistic Inference and Influence Diagrams[END_REF]. Selection of a specific probabilistic network on its own should be rationale. The reason for selection of Influence Diagrams over Binominal Trees and Decision trees is explained using more details on the other two techniques; although they can be converted to each other in many scenarios.

KMS for the London Plan Case Study

The final model can be implemented as decision support system for the stakeholders. It also has the capability of being disseminated as a web based DSS tool via web services techniques. There is an overlap is between probabilistic networks and fuzzy systems. The ID uses the same theories, but incorporates the set graphical interface for better understanding and modeling the problem domain. Financial analysis of the London Plan policy 4A.x (X=1, 2, 3) is an appropriate case study for this validation, although other case studies might be used. The reason for selection of such a case study lies within the gap in discussions and analysis of the GLA plan. The London Plan Policy 4A.X proposes 60% reduction in London CO2 compared to 1990 base, using CHP (Combined Heat & Power), Energy Efficiency, and Renewable Energy Systems expending solar and wind [START_REF]The London Plan[END_REF]. The data extraction is done using a theoretical narrative extraction, and the design base of this research would be quantitative practical base. Fig. 4 outlines the high level ID for this scenario.

Reasoning in such a model is dependent on the type of scenario under analysis. In this scenario, there is no 'Explaining Away' reasoning case between the nodes. Although, there are converging connections in the node orders, but explaining away cannot be inferred in the Extended ID developed. The converging inference can be assessed when the node orders are reversed using the reversal technique. There is no successor for this decision node and therefore the decision node removal cannot be considered for evaluating this ID. The reversal and node removal including barren node removal for this influence diagram can be further works of this piece of research. Casual reasoning in the model is where the UK-GDP has statistical dependence on the London GDP and that continues to all other proceeded nodes. There is various inductive reasoning in the ID where the parent nodes are not dependent to each other, i.e. 'Total Costs'. [START_REF] Cobb | Measuring Efficiency in Influence Diagram Models[END_REF]. In addition, α is the value set by the decision maker. This formula should be looked at with caution, as the complexity and accuracy of the ID should be normalized first [START_REF]The Mathematica Book[END_REF]. Cobb's formula concerning the accuracy and complexity functions are as follows: ̂

A represents accuracy and C represents complexity. The consideration is that when the accuracy is maximized, the complexity of minimized and vice versa. N min represents 1 and N max represents 2 in Cobb's research. The Accuracy and Complexity is scaled between a minimum and a maximum in order to assess a trade-off. The complexity of the ID expressions can be calculated using LeafCount function of the Mathematica IDE. Mathematics is software developed by Wolfram which can evaluate the complexity of functions and graphs. The calculation is simple and is counting the variables of any approximation within the ID. It does not only count the variables, but also the expressions defining the function. Therefore LeafCount is a function which counts the number of words, variables and constants in an expression [START_REF]The Mathematica Book[END_REF]. The rationale behind this calculation is consideration of the memory required to process the function. The Complexity of an ID then is the summation of all individual complexities:

∑ ∑ ∑
Using the LeafCount function we can simply determine the complexity of expressions and sub-expressions of the London Plan Policy 4A.x. For instance:

In [START_REF] Valente | Artificial Intelligent Methods in Operational Knowledge Management[END_REF]:=LeafCount [399943*Num_london_propertie] Out [START_REF] Valente | Artificial Intelligent Methods in Operational Knowledge Management[END_REF]=3

In [START_REF] Zack | Knowledge management and Collaboration Technologies[END_REF]:=Level [399943 + * + Num_london_propertie, Heads  True] Out [START_REF] Zack | Knowledge management and Collaboration Technologies[END_REF]= {Multiply, 399943, Num_london_propertie} The complexity of the RoI node expression is 3. That is summed up to complexity outputs from the LeafCount function in Mathematica on other nodes. If there are combinatory expressions in the ID, then the NestList function of Mathematica, lists all the expression in combinations and then the LeafCount can be run. The complexity of the whole Extended Influence Diagram for considering the Leafcount function is done manually and the number 50 is reached.

For finding Accuracy of Influence Diagram accuracy, the mean squared error between the analytical decision rule and the ID decision rule should be calculated.

The found Accuracy and Complexity values are then traded off using the stated formulas. Cobb's minimum and Maximum (N) and be scaled using other values, but the technique remains the same. The efficiency of an ID can also be affected by the decision makers' amendments using α variable [START_REF]The Mathematica Book[END_REF].

This model also validates the approximations on predicting the financial costs of the policy which gives practical values to this research study. The result of the research indicates the estimated costs of the London plan policy concerning CO2 reduction has been within the discussed outcomes. The output of the simulation states that the London plan would cost £Billion 68.84. This number is derived after subtraction of £Billion 12.06 Billion pounds return on investment from the total costs which was £Billion 80.9. The simulations are performed within Analytica IDE, but other environments can also be utilized for simulation.

However, Analytica is a simulation-based tool, and currently cannot prescribe an optimal solution for the detection network. Although unavailable in software tools, influence diagrams with continuous variables can be found in the literature, Again, none of the works is applicable to the parallel detection network structure. Now consider a variation where we remove the continuous variables, i.e., the measurements from the sensor, and transform the local decision makers to be chance nodes.

This yields an alternative model, which is equivalent a similar structure was introduced by Heckerman to analyze the value of information for diagnosis. In this configuration, the subordinate decision makers provide evidence or local decisions [START_REF] Baye | Managerial Economics and Business Strategy[END_REF].

Conclusion, limitation and further works

There are number of limitations to this model. One the limitations of this research study would be the all-embracing complexities involved. The complexities do not only arise from the financial domains of the policy, but also the social and environmental facets as well.

Another constraint of this research project has been the consideration of only the financial facets of the policy plan. The social limitations and all environmental restraints reside outside the boundary of this study. Knowing that this project involves an open system analysis, therefore, not considering those two facts would restrain the overall result of the research work.

On the case study, there are some zones where the plan policy has not clarified fully. For instance the choice of 60% reduction by 2050 within policy has not been explained. In addition to that a comparative financial analysis has not been provided in the policy. This could be used as a framework for comparing the results of this project and other similar projects and the policy plan results.

There are some limitations in Knowledge Management Systems.These limitations in KMS are defined as:

1. Tacit knowledge cannot be easily monitored and managed, 2. Involvement of the stakeholders in a dynamic and up to date management of their knowledge, 3. Interactions between stakeholders might be limited, 4. System is not necessarily adaptive [START_REF] Nabeth | Enhancing Knowledge Management Systems with Cognitive Agents[END_REF]. Recent research on goal oriented models suggests that Knowledge Management Systems can be developed in a goal oriented friendly format. The initial proposal for goal oriented KMS was introduced in 2004, but still various research is taking place concerting this topic. In theory the goal oriented KMS would have the adaptively, innovation and replication characteristics [START_REF] Gray | Knowledge sourcing effectiveness[END_REF]. Although this is out of scope of this study, but a future work on ID model developed in this research work might be implementation of its goal oriented KMS which has the mentioned here characteristics.

Figure 1 :

 1 Figure 1: Categories of Knowledge [4]

Figure 2 :

 2 Figure 2: Knowledge Acquisition Framework and Stages [5]

Figure 3 :

 3 Figure 3: Artificial Intelligence areas [7]

Figure 4 :

 4 Figure 4: High Level Influence Diagram as the Knowledge Base for Policy 4A.x Financial Assessment


Sotirios P Chatzis

Dimitrios Korkinof

Yiannis Demiris

A Spatially-Constrained Normalized Gamma Process for Data Clustering

In this work, we propose a novel nonparametric Bayesian method for clustering of data with spatial interdependencies. Specifically, we devise a novel normalized Gamma process, regulated by a simplified (pointwise) Markov random field (Gibbsian) distribution with a countably infinite number of states. As a result of its construction, the proposed model allows for introducing spatial dependencies in the clustering mechanics of the normalized Gamma process, thus yielding a novel nonparametric Bayesian method for spatial data clustering. We derive an efficient truncated variational Bayesian algorithm for model inference. We examine the efficacy of our approach by considering an image segmentation application using a real-world dataset. We show that our approach outperforms related methods from the field of Bayesian nonparametrics, including the infinite hidden Markov random field model, and the Dirichlet process prior.

Introduction

Nonparametric Bayesian modeling techniques, especially Dirichlet process mixture (DPM) models, have become very popular in statistics over the last few years, for performing nonparametric density estimation [START_REF] Walker | Bayesian nonparametric inference for random distributions and related functions[END_REF][START_REF] Neal | Markov chain sampling methods for Dirichlet process mixture models[END_REF][START_REF] Muller | Nonparametric Bayesian data analysis[END_REF]. This theory is based on the observation that an infinite number of component distributions in an ordinary finite mixture model (clustering model) tends on the limit to a Dirichlet process (DP) prior [START_REF] Neal | Markov chain sampling methods for Dirichlet process mixture models[END_REF][START_REF] Antoniak | Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems[END_REF]. Eventually, the nonparametric Bayesian inference scheme induced by a DPM model yields a posterior distribution on the proper number of model component densities (inferred clusters) [START_REF] Blei | Variational methods for the Dirichlet process[END_REF], rather than selecting a fixed number of mixture components.

Markov random fields (MRFs) [START_REF] Orbanz | Nonparametric Bayes image segmentation[END_REF] are a classical methodology for modeling spatially-interdependent data. In essence, MRFs impose a Gibbsian distribution over the allocation of the modeled data into states (clusters), which enforces the belief that spatially adjacent data are more likely to cluster together. As the Gibbsian prior imposed by MRFs entails complex calculations that make it intractable in real-world problems dealing with large datasets, efficient approximations of the full MRF distribution are usually employed. For example, a pointwise simplification of the MRF prior based on the mean-field principle from statistical mechanics [START_REF] Zhang | The mean field theory in EM procedures for Markov random fields[END_REF] was employed in [START_REF] Celeux | EM procedures using mean field-like approximations for Markov model-based image segmentation[END_REF]. Recently, MRFs have also been used in the context of Bayesian nonparametrics yielding the infinite hidden Markov random field (iHMRF) model [START_REF] Chatzis | The infinite hidden Markov random field model[END_REF][START_REF] Chatzis | The infinite hidden Markov random field model[END_REF].

Inspired by these advances, in this paper we come up with a different approach towards clustering data with spatial interdependencies. We propose a spatially-adaptive random measure, coined the Markov random field normalized Gamma process (MRF-NGP). Our model is based on the introduction of a normalized Gamma process (NGP) controlled by an additionally postulated pointwise Markov random field imposed over the data allocation into model states, obtained by application of the mean-field principle [START_REF] Chatzis | The infinite hidden Markov random field model[END_REF]. We evaluate the efficacy of our approach considering an unsupervised image segmentation application using benchmark data.

Theoretical Background

The Dirichlet Process

Dirichlet process (DP) models were first introduced in [START_REF] Ferguson | A Bayesian analysis of some nonparametric problems[END_REF]. A DP is characterized by a base distribution G 0 and a positive scalar α, usually referred to as the innovation parameter, and is denoted as DP(α, G 0 ). Essentially, a DP is a distribution placed over a distribution. Let us suppose we randomly draw a sample distribution G from a DP, and, subsequently, we independently draw

M random variables {Θ * m } M m=1 from G: G|α, G 0 ∼ DP(α, G 0 ) (1)
Θ * m |G ∼ G, m = 1, . . . M (2)
Integrating out G, the joint distribution of the variables {Θ * m } M m=1 can be shown to exhibit a clustering effect. Specifically, given the first M -1 samples of G, {Θ * m } M -1 m=1 , it can be shown that a new sample Θ * M is either (a) drawn from the base distribution G 0 with probability α α+M -1 , or (b) is selected from the existing draws, according to a multinomial allocation, with probabilities proportional to the number of the previous draws with the same allocation [START_REF] Blackwell | Ferguson distributions via Pólya urn schemes[END_REF].

Markov random fields

We consider an alphabet Q = {1, ..., K}. Let S be a finite index set, S = {1, ..., N }; we shall refer to this set, S, as the set of sites or locations. Let us consider for every site j ∈ S a finite space Z j of states z j taking on Q. The product space Z = N j=1 Z j will be denoted as the space of the configurations of the state values of the considered sites set, z = (z j ) j∈S . A strictly positive probability distribution, p(z), z ∈ Z, on the product space Z is called a random field [START_REF] Maroquin | Probabilistic solution of ill-posed problems in computational vision[END_REF].

Let ∂ denote a neighborhood system on S, i.e. a collection ∂ = {∂ j : j ∈ S} of sets, such as j / ∈ ∂ j and l ∈ ∂ j if and only if j ∈ ∂ l ∀l, j ∈ S. Then, the previously considered random field, p(z), is a Markov random field with respect to the introduced neighborhood system ∂ if [START_REF] Geman | Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images[END_REF] p(z j |z S-{j} ) = p(z j |z ∂j ) ∀j ∈ S. The distribution p(z) of a Markov random field can be shown to be of a Gibbsian form [START_REF] Clifford | Markov random fields in statistics[END_REF]:

p(z) 1 W (γ) exp - c∈C V c (z|γ) (3)
where γ is the inverse temperature of the model, W (γ) is the (normalizing) partition function of the model, V c (z|γ) are the clique potentials of the model, and C is the set of the cliques included in the model neighborhood system. A significant problem of MRF models concerns computational tractability, as the normalizing term W (γ) is hard to compute in applications dealing with large datasets. One way to resolve these issues is the mean-field approximation [START_REF] Zhang | The mean field theory in EM procedures for Markov random fields[END_REF][START_REF] Chatzis | A fuzzy clustering approach toward hidden Markov random field models for enhanced spatially constrained image segmentation[END_REF]. It is based on the idea of neglecting the fluctuations of the sites interacting with a considered site, so that the resulting system behaves as one composed of independent variables for which computation becomes tractable. That is, given an estimate ẑ of the unknown site labels vector z, obtained by means of a stochastic restoration criterion, such as the iterative conditional modes (ICM) or the marginal posterior modes (MPM) algorithm (see, e.g., [START_REF] Geman | Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images[END_REF][START_REF] Chatzis | A fuzzy clustering approach toward hidden Markov random field models for enhanced spatially constrained image segmentation[END_REF]), we make the hypothesis [START_REF] Qian | Estimation of parameters in hidden Markov models[END_REF]

p(z) = N j=1 p(z j |ẑ ∂j ; γ) (4)
where

p(z j = i|ẑ ∂j ; γ) = exp(-c j V c (z ij |γ)) K h=1 exp(-c j V c (z hj |γ)) (5)
zij (z j = i, ẑ∂j ), ẑ∂j is the estimate of the jth site neighborhood, and the indexes c refer to the cliques that contain the jth site.

3 Proposed Approach

Model Formulation

Let us consider a set of observations Y = {y n } N n=1 , y n ∈ Y, measured over a set of sites S = {1, ..., S} on which a neighborhood system ∂ is defined. Let us denote as X = {x n } N n=1 , x n ∈ S, the sites where the observed data points {y n } N n=1 were measured. Let us introduce the latent variables {z n } N n=1 denoting the model state (cluster) where an observed data point y n measured at the location x n is assigned by our model. Motivated by the merits and the theory of the DP discussed in the previous section, to derive the sought model, we make the key-assumption, based on the mean-field-based approximation of the MRF distribution, that for any given site x n , we have available an estimate ẑ∂n of the value z ∂n (z m ) m∈∂n of the latent cluster assignment variables of the observations measured at sites in the neighborhood of site x n . Apparently, this assumption entails obtaining an initial estimate of the latent variables {z n } N n=1 for the modeled data, as discussed in Section 2.2. Further, we consider the following predictor (location)-dependent random measure

G(x) = ∞ i=1 i (x)δ Θi (6)
where

i (x) = Λ i (x) ∞ j=1 Λ j (x) (7)
the random variables Λ i follow a Gamma distribution as

Λ i |x n ∼ G(αk i (x n ; ẑ∂n ), 1) (8)
α is the innovation parameter of the process, k i (x n ; ẑ∂n ) is the probability of the nth site being assigned to the ith cluster as computed by the employed pointwise MRF distribution

k i (x n ; ẑ∂n ) p z n = i ẑ∂n ; γ = exp(-c xn V c (z ni |γ)) ∞ h=1 exp(-c xn V c (z nh |γ)) (9)
zni (z n = i, ẑ∂n ), ẑ∂n is the current estimate of nth site neighborhood, V c (•) are the employed clique potential functions, and the indexes c refer to the cliques that include the nth site, x n . The utility of the pointwise MRF distribution k i (x n ; ẑ∂n ) in our model consists in reducing the probability (discounting) of clusters that seem rather unlikely from the viewpoint of the postulated neighborhood system. We dub this random probability measure G(x) the MRF-NGP process. A proof that the normalizing constant in the denominator of ( 7) is finite almost surely is provided in the Appendix.

Variational Bayesian Inference

Let us a consider a set of observations Y = {y n } N n=1 with corresponding locations X = {x n } N n=1 . We postulate for our observed data a likelihood function of the form p(y

n |z n = i) = p(y n |θ i ) (10)
while for the latent assignment variables z n we consider

p(z n = i|x n ) = i (x n ) (11)
where the i (x) are given by [START_REF] Zhang | The mean field theory in EM procedures for Markov random fields[END_REF], with the prior over the Λ i (x) given by [START_REF] Celeux | EM procedures using mean field-like approximations for Markov model-based image segmentation[END_REF]. Regarding the likelihood parameters θ i , we impose a suitable conjugate exponential prior over them; for instance, in case of a Gaussian likelihood function

p(y n |θ i ) = N (y n |µ i , R i ) (12)
we impose a Normal-Wishart prior over the likelihood parameters

θ i = {µ i , R i }, i.e. p(µ i , R i ) = N W(µ i , R i |λ i , m i , ω i , Ω i ) (13)
Regarding the MRF temperature parameter γ, and the innovation parameter α, we choose to optimize them as model hyperparameters, as part of the variational inference procedure discussed next.

Our variational Bayesian inference formalism consists in derivation of a family of variational posterior distributions q(.) which approximate the true posterior distribution over the infinite sets {z n } N n=1 , {Λ i (x n )} ∞,N i,n=1 , and {θ i } ∞ i=1 . Apparently, under this infinite dimensional setting, Bayesian inference is not tractable. For this reason, we fix a value K and we let the variational posterior over the Λ k (x) have the property q(Λ k>K (x) = 0) = 1, ∀x ∈ S. In other words, we set k (x) equal to zero for k > K, ∀x ∈ S. Note that, under this setting, the treated model involves a full MRF-NGP prior; truncation is not imposed on the MRF-NGP prior itself, but only on the variational distribution to allow for a tractable inference procedure [START_REF] Blei | Variational inference for Dirichlet process mixtures[END_REF].

Letting W be the parameters of our model, and Ξ the entailed hyperparameters, including γ, α, and the hyperparameters of the prior over the likelihood parameters, variational Bayesian inference consists in derivation of an approximate posterior q(W ) by maximization (in an iterative fashion) of the variational free energy

L(q) = ˆdW q(W )log p(X, Y, W |Ξ) q(W ) (14)
which provides a lower bound to the computationally intractable log marginal likelihood (log evidence), logp(X, Y ), of the model [START_REF] Jordan | An introduction to variational methods for graphical models[END_REF]. The derived algorithm is in essence an expectation-maximization-like algorithm. Each iteration comprises an E-step, on which the variational posteriors over the model latent variables are computed, and an M-step, on which the variational posteriors over the model parameters are updated. Let us denote as . the posterior expectation of a quantity.

M-step This step comprises the updates of the Gamma-distributed variables

Λ i (x n ) q(Λ i (x n )) = G(Λ i (x n )|β ni , ξ ni ) (15)
where

β ni = αk i (x n ; ẑ∂n ) + q(z n = i) (16)
ξ ni =1 + 1 K j=1 Λ j (x n ) (17)
and

Λ j (x n ) = β nj ξ nj ( 18
)
as well as of the parameters θ i , for which we obtain the general solution

logq(θ i ) ∝ logp(θ i ) + N n=1 q(z n = i)logp(y n |θ i ) (19)
which is similar to the corresponding solution for models imposing simple DP priors over their cluster assignment distributions.

E-step This step comprises the updates of the posteriors q(z n = i):

q(z n = i) ∝ exp ( logΛ i (x n ) ) exp (ϕ ni ) (20)
where

logΛ i (x n ) = ψ(β ni ) -logξ ni (21)
and

ϕ nj = logp(y n |θ j ) (22)
It also consists in updating the estimates of the assignment variables ẑ = (ẑ n ) N n=1 which are used in computing the pointwise MRF priors employed in our model to regulate cluster discounting. For this purpose, we simply set

ẑn = argmax K i=1 q(z n = i) (23)
Finally, regarding the model hyperparameters Ξ, in this work we obtain estimates of the hyperparameter γ by maximization of the lower bound L(q), and we heuristically select the values of the rest of the model hyperparameters.

Experimental Evaluation

We investigate the efficacy of our approach considering an unsupervised image segmentation application. Specifically, we consider segmentation of real-world images, using a subset of the Berkeley image segmentation benchmark [START_REF] Martin | A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[END_REF]. The Berkeley image segmentation benchmark comprises real-world color images along with their segmentation maps provided by different individuals. To obtain an objective performance evaluation of the proposed algorithm, we employ the probabilistic rand index (PR index or PRI) [START_REF] Unnikrishnan | A measure for objective evaluation of image segmentation algorithms[END_REF]. The PR index counts the fraction of pairs of pixels whose labelings are consistent between a computed segmentation and the given groundtruth, averaging across multiple groundtruth segmentations to account for scale variation in human perception. In all our experiments, we use Gaussian likelihoods, and choose to impose a Normal-Wishart prior over the likelihood parameters. We compare the performance of our approach to iHMRF and DPM, both using the same likelihood function and prior over the likelihood function parameters as in the case of our model. All the evaluated algorithms are initialized by means of the k-means algorithm. Regarding the potential functions of the imposed pointwise MRFs for both the evaluated iHMRF and MRF-NGP models, we opt for a simple Potts model with a second order (8-neighbors) neighborhood system, yielding

p(z n = c|ẑ ∂n ; γ) = exp(γ l∈∂n δ(c -ẑl )) K h=1 exp(γ l∈∂n δ(h -ẑl )) (24)
for the pointwise MRF priors, where K is the truncation threshold, and δ(.) stands for the Kronecker's delta function.

Feature extraction is effected as follows: First, each image is segmented into approximately N = 1000 superpixels using the method proposed in [START_REF] Mori | Guiding model search using segmentation[END_REF]. We then compute feature vectors at superpixel level, comprising RGB and HSV color information along with the values of the Maximum Response (MR) filter banks [START_REF] Varma | Classifying images of materials: Achieving view-point and illumination independence[END_REF]. The truncation level of the variational Bayesian algorithm for all the treated models is set to K = 10.

To account for the effect of poor model initialization, which may lead model training to yield bad local optima as model estimators, we execute our experiments multiple times for each image, with different initializations each time, common for all the evaluated algorithms. The visual segmentation result is presented for 4 selected images in Fig. 1, along with the original image, one human groundtruth, and the initialization. The mean PRI results (over the executed repetitions) for the whole considered dataset are presented in Table 1. Total results across all images are presented in Table 2. Based on the obtained PRI metric results, we can conclude that the MRF-NGP performs considerably better than all the considered rival methods. Note also that small differences in the values of the PRI metric correspond to significant differences in the quality of the obtained segmentation results [START_REF] Nikou | A class-adaptive spatially variant mixture model for image segmentation[END_REF]. The illustrated segmentation results vouch for this assertion.

Conclusions

In this paper, we proposed a method for nonparametric clustering of data with general spatial interdependencies. Our method, coined the MRF-NGP, consists in postulating a normalized Gamma-process, the cluster prior probabilities of which are discounted by means of a simplified pointwise Markov random field imposed over data point allocation into clusters. As a result of this construction, the MRF-NGP imposes the belief that spatially proximate data are more likely to cluster together. To examine the efficacy of our approach, we evaluated it in unsupervised image segmentation tasks using a real-life benchmark dataset, namely the Berkeley image segmentation benchmark. We showed that it yields a considerable improvement in the obtained performance of the clustering algorithm compared to both the DPM, and the recently proposed iHMRF model.

Fig. 1 :

 1 Fig. 1: Few selected 321x481 images from the Berkeley image segmentation dataset. Left-to-right: a) Original image, b) One human groundtruth, c) Kmeans initialization, d) iHMRF, e) MRF-NGP. Top-to-bottom: a) #241004, b) #161062, c) #385028, d) #246053.

Table 1 :

 1 Obtained PRI results for the considered subset of the Berkeley benchmark.

	Image # DPM iHMRF MRF-NGP Image # DPM iHMRF MRF-NGP
	159029 0.7688 0.7727	0.7842	246053 0.5896 0.6006	0.6526
	20008 0.8376 0.8514	0.8478	239096 0.7570 0.7957	0.7871
	100075 0.7851 0.7795	0.7861	323016 0.8283 0.8436	0.8479
	301007 0.8438 0.8432	0.8460	231015 0.8019 0.8185	0.8138
	122048 0.7396 0.7520	0.7421	25098 0.8270 0.8231	0.8394
	145053 0.6189 0.6315	0.7304	8143 0.6294 0.6605	0.7011
	236017 0.5997 0.6035	0.6346	35010 0.7701 0.7854	0.8051
	170054 0.6841 0.7453	0.7628	15004 0.7561 0.7865	0.7994
	385028 0.8520 0.8393	0.8544	100080 0.8067 0.8093	0.8036
	67079 0.7344 0.7347	0.7599	161062 0.6610 0.6280	0.6742
	209070 0.6335 0.7006	0.7380	159045 0.6984 0.7315	0.7482
	27059 0.8359 0.8470	0.8669	170057 0.6948 0.7243	0.7498
	176019 0.6930 0.7470	0.7343	89072 0.7377 0.7590	0.7841
	103070 0.7164 0.7307	0.7530	175032 0.5594 0.6783	0.6686
	241004 0.8643 0.8642	0.8738	86016 0.7379 0.7664	0.7573

Table 2 :

 2 Mean and median of the PRI metric across the considered subset of the Berkeley benchmark.

	PRI(%) DPM iHMRF MRF-NGP
	Mean	73.54 75.51 77.15
	Median 73.88 76.27 77.34

Acknowledgment

This work has been partially funded by the EU FP7 ALIZ-E project (contract #248116).

Appendix

Here, we prove the almost sure finiteness of the normalizing factor

since the random variables Λ j (x n ) are non-negative, as they follow a Beta distribution.

Then, to prove that S is finite almost surely, we only need to prove that E[S] is finite. From the monotone convergence theorem, we yield

T →∞ T j=1 k j (x n ; ẑ∂n ) = 1, as the k j (x n ; ẑ∂n ) comprise prior MRF-derived probabilities of the observation at the nth site being assigned to any of the postulated model states. Hence, we have proven that S is finite almost surely.


Petr Hájek
email: petr.hajek@upce.cz

Vladimír Olej
email: vladimir.olej@upce.cz

Adaptive Intuitionistic Fuzzy Inference Systems of Takagi-Sugeno Type for Regression Problems

Keywords: Intuitionistic fuzzy sets, intuitionistic fuzzy inference systems, adaptation, regression

Recently, we have proposed a novel intuitionistic fuzzy inference system (IFIS) of Takagi-Sugeno type which is based on Atanassov's intuitionistic fuzzy sets (IF-sets). The IFIS represent a generalization of fuzzy inference systems (FISs). In this paper, we examine the possibilities of the adaptation of this class of systems. Gradient descent method and other special optimization methods are employed to adapt the parameters of the IFIS in regression problems. The empirical comparison of the systems is provided on several well-known benchmark and real-world datasets. The results show that by adding non-membership functions, the average errors may be significantly decreased compared to FISs.

Introduction

Several generalizations of Zadeh's fuzzy set theory [START_REF] Zadeh | Fuzzy Sets[END_REF] have been developed to handle imprecision and uncertainty in a better way, e.g. IF-sets [START_REF] Atanassov | Intuitionistic Fuzzy Sets[END_REF][START_REF] Atanassov | Intuitionistic Fuzzy Sets[END_REF], L-fuzzy sets [START_REF] Goguen | L-fuzzy Sets[END_REF], interval-valued fuzzy sets (grey sets) [START_REF] Sambuc | Fonctions Φ-floues. Application a Ľaide au Diagnostic en Pathologie Thyroidienne[END_REF], or interval-valued IF-sets [START_REF] Atanassov | Interval Valued Intuitionistic Fuzzy Sets[END_REF], see [START_REF] Deschrijver | On the Relationship Between Some Extensions of Fuzzy Set Theory[END_REF] for a review. It was proven that interval-valued fuzzy set theory is equivalent to IF-set theory, which is equivalent to the vague set theory. In a similar manner, FISs have also been extended as interval-valued FISs [START_REF] Gorzalczany | A Method of Inference in Approximate Reasoning Based on Interval Valued Fuzzy Sets[END_REF], interval-valued type-2 FISs (IT2FISs) [START_REF] Karnik | Introduction to Type-2 Fuzzy Logic Systems[END_REF][START_REF] Starczewski | Connectionist Structures of Type-2 Fuzzy Inference Systems[END_REF] or IFIS [START_REF] Montiel | Mediative Fuzzy Logic: A New Approach for Contradictory Knowledge Management[END_REF][START_REF] Olej | IF-Inference Systems Design for Prediction of Ozone Time Series: The Case of Pardubice Micro-region[END_REF][START_REF] Olej | Possibilities of Air Quality Modelling Based on IF-sets Theory[END_REF].

The concept of IF-sets can be viewed as an alternative approach to define a fuzzy set in cases where available information is not sufficient for the definition of an imprecise concept by means of a conventional fuzzy set. This may hold true for many real world applications. For example, the uncertainty presented in air quality evaluation was modelled using hierarchical IFISs of Mamdani type in [START_REF] Olej | Possibilities of Air Quality Modelling Based on IF-sets Theory[END_REF].

Recently, we presented a novel IFIS of Takagi-Sugeno type for time series prediction [START_REF] Olej | IF-Inference Systems Design for Prediction of Ozone Time Series: The Case of Pardubice Micro-region[END_REF][START_REF] Olej | Comparison of Fuzzy Operators for IF-inference Systems of Takagi-Sugeno Type[END_REF]. It was shown that MIN t-norm (Gödel t-norm) provided the lowest error in terms of root mean squared error (RMSE) on testing data [START_REF] Olej | Comparison of Fuzzy Operators for IF-inference Systems of Takagi-Sugeno Type[END_REF]. Furthermore, we developed IFIS with various levels of uncertainty represented by intuitionistic fuzzy index (IF-index). This class of systems presents a strong possibility to express uncertainty (in the presence of imperfect facts and imprecise knowledge) and provides a good description of object attributes by means of membership functions   x) and non-membership functions   x).

In this paper, we compare several optimization algorithms used to adapt FISs and IFISs on selected artificial and real world datasets. We will also analyze the behaviour of the IFIS when the weights of the subsystems outputs (with membership functions   x) and non-membership functions   x)) change. The rest of this paper is organized as follows. First, we briefly introduce FISs and IFIS of Takagi-Sugeno type. In this section, the difference between IFIS and related IT2FISs is mentioned. Next, we present the methods used for the adaptation of IFISs. In section 4, regression datasets are described. Finally, the results of experiments are provided and analyzed.

IFIS of Takagi-Sugeno Type

The FIS of Takagi-Sugeno type [START_REF] Takagi | Fuzzy Identification of Systems and its Applications to Modeling and Control[END_REF] is composed of the following steps. In the fuzzification process, the input variables are compared with the membership functions   x). Next, operators (AND, OR, NOT) are applied within the if-then rules. Thus, firing weight of each if-then rule is obtained. Further, the outputs of each if-then rule are generated. The output of each rule is a linear combination of input variables plus a constant term. The final output is the weighted average of each if-then rule's output.

Let xX, X={x 1 ,x 2 , … ,x i , … ,x n } be input variables defined on the universes X 1 ,X 2 , ... ,X i , ... ,X n and let y be an output variable defined on the universe Y. Then FIS has n input variables and one output variable. Then the k-th if-then rule R k , k=1,2, ... ,N, in the FIS of Takagi-Sugeno type can be defined in following form

R k : if x 1 is A 1,k AND x 2 is A 2,k AND ... AND x i is A i,k  AND ... AND x n is A n,k then y k =f(x 1 ,x 2 , ... ,x n ), i=1,2, ... ,n, (1)
where A 1,k ,A 2,k , ... ,A i,k , ... ,A n,k represent fuzzy sets and f(x 1 ,x 2 , ... ,x n ) can be a linear or polynomial function. In further considerations, we assume a linear function defined as y k =a 1,k x 1 +a 2,k x 2 + ... +a i,k x i + ... +a n,k x n +b.

The FIS of Takagi-Sugeno type was designed in order to achieve higher computational effectiveness. This is possible as the defuzzification of outputs is not necessary. Its advantage lies also in involving the functional dependencies of output variable on input variables. The output level y k of each the k-th if-then rule R k is weighted by w k =x 1 ) AND x 2 ) AND … AND x m ). The final output y of the FIS Takagi-Sugeno type is the weighted average of all N if-then rule R k outputs y k , k=1,2, … ,N, computed as follows

     N k N k k k k w w y y 1 1 .
(

) 2
Let a set X be a non-empty fixed set. An IF-set A in X is an object having the form

A{x,  (x),  (x)
R  k : if x 1 is A  1,k AND x 2 is A  2,k AND ... AND x i is A  i,k  AND ... AND x n is A  n,k then y  k = a  1,k x 1 +a  2,k x 2 + ... +a  i,k x i + ... +a  n,k x n +b  , R  k : if x 1 is A  1,k AND x 2 is A  2,k AND ... AND x i is A  i,k  AND ... AND x n is A  n,k then y  k = a  1,k x 1 +a  2,k x 2 + ... +a  i,k x i + ... +a  n,k x n +b  . (3)
The output y  of FIS  (the output y   of FIS  ) is defined in the same way as presented in equation ( 2) for firing weights w  k and w  k , respectively. The output y of the IFIS represents a combination of y  and y

  y=(1β)y  βy  , ( 4
)
where y is the output of the IFIS, y  is the output of the FIS  using the membership function   (x), y  is the output of the FIS  using the non-membership function   (x), and β is the weight of the output y  . In prior studies [START_REF] Olej | Possibilities of Air Quality Modelling Based on IF-sets Theory[END_REF][START_REF] Olej | Comparison of Fuzzy Operators for IF-inference Systems of Takagi-Sugeno Type[END_REF], it was assumed that β=.

Like IFISs, IT2FISs also represent a generalization of the FISs. The membership degree of an interval type-2 fuzzy set is an interval (also known as the footprint of uncertainty) bounded from the above and below by two type-1 membership functions, Ā(x) (upper membership function) and A(x) (lower membership function). When related to IF-sets, A(x)=  (x) and Ā(x)=  (x)+  (x). This fact implies the differences between IFISs and IT2FISs. When applying AND operator (MIN t-norm) in if-then rules, the firing intervals

w k =[ k w ,w k ] are obtained. Notice that w k =w  k = MIN(x 1 ),x 2 ), … ,x m )) but k w =MIN(x 1 )+(x 1 ),x 2 )+(x 2 ), … ,x m )+(x m )) while w  k =MIN(x 1 ),x 2 ), … ,x m ))
. Recently, IT2FISs have been adapted using Karnik-Mendel algorithm and its variants [START_REF] Wu | Enhanced Karnik-Mendel Algorithms[END_REF][START_REF] Wu | Comparison and Practical Implementation of Type-Reduction Algorithms for Type-2 Fuzzy Sets and Systems[END_REF][START_REF] Yeh | An Enhanced Type-reduction Algorithm for Type-2 Fuzzy Sets[END_REF], back-propagation algorithm [START_REF] Lee | Nonlinear System Identification Using Takagi-Sugeno-Kang Type Interval-Valued Fuzzy Systems via Stable Learning Mechanism[END_REF], fuzzy C-means algorithm [START_REF] Uncu | Discrete Interval Type-2 Fuzzy System Models Using Uncertainty in Learning Parameters[END_REF], or genetic algorithms [START_REF] Lee | Type-2 Fuzzy Neural Network Systems and Learning[END_REF][START_REF] Celikyilmaz | Genetic Type-2 Fuzzy Classifier Functions[END_REF].

IFIS of Takagi-Sugeno Type Adaptation

There has been developed many approaches to adapt FISs in the literature, e.g. neural networks, genetic algorithms, Kalman filter, see e.g. [START_REF] Jang | ANFIS: Adaptive-Network-based Fuzzy Inference Systems[END_REF][START_REF] Alcala | Learning and Tuning Fuzzy Rule-based Systems for Linguistic Modeling and their Applications[END_REF]. In this paper, we use the following algorithms to adapt IFIS (and FISs): subtractive clustering algorithm (SCA) [START_REF] Chiu | Fuzzy Model Identification Based on Cluster Estimation[END_REF], Moore-Penrose pseudo-inverse (MPPI), Kalman filter [START_REF] Jang | Fuzzy Modeling Using Generalized Neural Networks Kalman Filter Algorithm[END_REF], Kaczmarz algorithm [START_REF] Kaczmarz | Approximate Solution of Systems of Linear Equations[END_REF][START_REF] Strohmer | A Randomized Kaczmarz Algorithm with Exponential Convergence[END_REF], and gradient algorithm [START_REF] Jang | ANFIS: Adaptive-Network-based Fuzzy Inference Systems[END_REF].

The identification of IFISs consists of the following steps. First, cluster centres are found to construct the number N of if-then rules and the antecedents of the if-then rules. Second, optimize the consequents of the rules, i.e. parameters a 

1,k ,a  2,k , … ,a  n,k ,b  and a  1,k ,a  2,k , … ,a  n,k ,b  .
The number of cluster centres c is equal to the number of if-then rules N, c=N, and k=1,2, … ,N is the index of the cluster centre. It is determined automatically using a SCA. The radius of influence of a cluster r a is considered the most important parameter in establishing the number of cluster centres c. A large r a results in fewer clusters, while a small r a generates a large number of clusters and, thus, can lead to model over-fitting.

In the subtractive clustering algorithm, each data point is considered as a potential cluster centre. A measure of potential of data point x k is defined as follows

     n j k j k e P 1 x x  , (5)
where α=4/r a 2 . Thus, a data point x k with many neighbouring points x j has a high potential value P k . The data point with the highest potential represents the cluster centre of the first cluster. Then, an amount of potential from each data point is subtracted as a function of its distance from the first cluster centre . The positive constant r b represents the radius defining the neighbourhood that will have measurable reductions in potential P k .

Let each input vector x k * is decomposed into two component vectors y k * and y k * , where y k * contains first n elements of x k * (input data) and y k * contains the output component (for a multi-input and single-output system). Each cluster centre x k * is considered an if-then rule. The output y is represented by a weighted average of the output of each rule [START_REF] Atanassov | Intuitionistic Fuzzy Sets[END_REF]. Since we use a IFIS of Takagi-Sugeno type of the first order, i.e. f(x 1 ,x 2 , ... ,x m ) is a linear function, we can compute the y k * as follows

y k * =G k y k +h k , (7)
where G k is a constant vector and h k is a constant. The estimation of the parameters of the given model can be understood as least squares estimate (LSE) in the form AX=B, where B is a matrix of output values, A is a constant matrix, and X is a matrix of parameters to be estimated. Let P is the set of linear parameters and X is an unknown vector whose elements are parameters in P. Then, we seek the optimal solution of X using a LSE X*. In this process, the squared error ||AX-B|| 2 is minimized. The LSE X* can be calculated using the pseudo-inverse of X as follows

X*=(A T A) -1 A T B, (8)
where A T is a transpose of A, and (A T A) -1 A T is the pseudo-inverse of A if A T A is nonsingular. A generalization of the inverse matrix A -1 is called MPPI A + .

A sequential method of LSE (also known as recursive LSE) can be used to compute X* [START_REF] Jang | ANFIS: Adaptive-Network-based Fuzzy Inference Systems[END_REF]. This method is more efficient, especially for a low number of linear parameters. Let a i T is is the i-th input vector of matrix A and b i T is the i-th element of B. Then X can be calculated using the following formulas (online version)

X i+1 =X i +S i+1 a i+1 (b T i+1 -a T i+1 X i ), S i+1 =1/λ[S i -(S i a i+1 a T i+1 S i )/(λ+a T i+1 S i a i+1 )], (9)
where S i is the covariance matrix, λ is the forgetting factor, and X*=X n . The initial conditions are X 0 =0 and S 0 =γI, where γ is a positive large number and I is the identity matrix. A small λ shows on the fast effects of old data decay. The LSE of X can be interpreted as the Kalman filter for the process [26]

X(k+1)=X(k), Y(k)=A(k)X(k)+e, ( 10
)
where e is noise, X(k)=X k , Y(k)=b k and A(k)=a k . Therefore, the sequential method of the LSE presented above is referred to as the Kalman filter algorithm. Kalman filters have been used to optimize the output function parameters FISs of Takagi-Sugeno type [START_REF] Ramaswamy | Two Approaches for Automating the Tuning Process of Fuzzy Logic Controllers[END_REF], to extract if-then rules from a given rule base of FISs [START_REF] Wang | Extracting Fuzzy Rules for System Modeling Using a Hybrid of Genetic Algorithms and Kalman Filter[END_REF], and to tune the input membership functions [START_REF] Simon | Training Fuzzy Systems with the Extended Kalman Filter[END_REF]. Another method used to compute the LSE X* is the Kaczmarz algorithm. This algorithm is based on the following formula

k k T k T k k k k k a a a X a b X X    1 . (11)
Another possibility to adapt an IFIS is represented by gradient algorithms. Let the i-th error is defined as E i =(y i -o i ) 2 , where y i is the actual output and o i is the predicted output. Then, the total error E is given as E=ΣE i . The derivative of the overall error measure E with respect to a generic parameter α is

      n i i E E 1   . ( 12
)
The update formula for the generic parameter α is defined as

        E , where              2 E h , ( 13
)
where η is the learning rate, h is the step size and the length of each gradient transition in the parameter space. Usually, we can change the value of h to vary the speed of convergence. Some practical difficulties associated with gradient descent are slow convergence and ineffectiveness at finding a good solution [START_REF] Simon | Training Fuzzy Systems with the Extended Kalman Filter[END_REF].

The following regression datasets were selected for the modelling using IFISs: Friedman benchmark function [START_REF] Friedman | Multivariate Adaptative Regression Splines[END_REF], daily electricity energy [START_REF]Knowledge Extraction Based on Evolutionary Learning[END_REF], stock prices [START_REF] Altay | University Function Approximation Repository[END_REF], and auto MPG dataset [START_REF] Quinlan | C4.5 Programs for Machine Learning[END_REF], for details see Table 1. Friedman dataset is a synthetic benchmark dataset where the instances are generated using the following method. Generate the values of n=5 input variables, x 1 ,x 2 , ... ,x 5 independently each which uniformly distributed over [0.0, 1.0]. Obtain the value of the target variable y using the equation y=10(sin(π)x 1 x 2 )+20(x 3 -0.5) 2 +10x 4 +5x 5 +e, where e is a Gaussian random noise N(0,1).

The Energy problem involves predicting the daily average price of TkWhe electricity energy in Spain. The data set contains real values from 2003 about the daily consumption in Spain of energy from hydroelectric, nuclear electric, carbon, fuel, natural gas and other special sources of energy.

In the stock prices dataset, the data provided are daily stock prices from January 1988 through October 1991, for ten aerospace companies. The task is to approximate the price of the 10th company given the prices of the rest.

The AutoMPG dataset concerns city-cycle fuel consumption in miles per gallon (Mpg), to be predicted in terms of 1 multi-valued discrete and 5 continuous attributes (two multi-valued discrete attributes (Cylinders and Origin) from the original dataset are removed).

Datasets were divided into training and testing data in relation 1:1. This division was realized five times. The quality of regression was measured by RMSE on testing data. We conducted the experiments in Matlab Fuzzy Logic Toolbox using the adaptation algorithms available at [START_REF] Sidelnikov | Sugeno-type FIS Output Tuning[END_REF].

The parameters of FISs and IFISs are set in the following way. The initial setting of FISs and IFISs was conducted using SCA. The number N of if-then rules (and the numbers of membership and non-membership functions at the same time) depends on the choice of parameter r a in SCA primarily. In order to avoid over-fitting, we tested different values of r a ={0.1,0.2, … ,0.9}. For the IFISs, the IF-index is set on π=0.3 (medium level of hesitancy recommended in [START_REF] Olej | Comparison of Fuzzy Operators for IF-inference Systems of Takagi-Sugeno Type[END_REF]) and, therefore, the membership and non-membership functions are defined in the following way

2 2 2 ) ( 7 . 0 ) (   b x e x    , 2 2 2 )
( 7 . 0 7 . 0 ) ( )) ( 1 ( ) (     b x e x x x        . ( 14
)
The settings of FISs and IFISs parameters are presented in Table 2. In our experiments, we used the following settings of adaptation methods: -Gradient algorithm: maximum number of epochs was set to 500, step size h=0.01, step increasing rate to 1.1, and step decreasing rate to 0.9; -Kaczmarz algorithm: maximum number of sweeps was set to 10; -Kalman filter: data forgetting factor was set to λ=1.0 and its increasing factor to 1.0; -MPPI: no parameters; For comparison, FISs and IFISs with corresponding adaptation methods are shown. In the case of Friedman dataset, the dependence of RMSE (testing data) on β is depicted in Fig. 1 (SCA) and Fig. 2 (gradient algorithm). When IFIS is identified using SCA only (i.e. it is not adapted subsequently), RMSE increases with rising β for all datasets (minimum RMSE for IFIS-SCA is achieved for β=0.0). In the case of gradient algorithm it is possible to reduce RMSE when combining the outputs of both subsystems with β=0.1. For the Friedman dataset (the most complex dataset in terms of N and the numbers of membership and non-membership functions), gradient algorithm provided the best result but for the Energy dataset this algorithm did not converge and for the Stock and AutoMPG datasets it was outperformed by the Kalman filter and MPPI, respectively. For the Energy dataset, the behaviour of IFIS differs from that one for the other datasets. The lowest RMSE is achieved for the Kalman filter and MPPI with β=0.6, i.e. both subsystems (with membership and nonmembership functions) contribute with a similar magnitude.

Conclusion

In this paper, we have introduced IFIS of Takagi-Sugeno type. We have proposed several methods to adapt the parameters of this class of systems. We conclude that Kalman filter and MPPI are suitable to adapt the linear functions in the consequents of the IFIS if-then rules. However, gradient algorithm may perform better than these methods for more complex problems. The antecedents parts of if-then rules, membership functions   (x) and non-membership functions   (x) were identified using SCA. For all the tested datasets, it was possible to significantly decrease RMSE on testing data when both an appropriate adaptation method was applied and the subsystems with membership and non-membership functions were combined properly.

In further research we plan to use a wider range of benchmark datasets to test the relation between data complexity and behaviour of IFISs (the value of IF-index especially). We also suggest comparing the two related concepts, IFISs and IT2FISs of Takagi-Sugeno type on regression problems (using also other performance measures such as R 2 or MAE) to show the differences between these two generalizations of FISs. Finally, IFISs should be tested on data with additional noise since generalizations of FISs have been proven to be especially beneficial in the cases where data was corrupted by measurement or estimated noise [START_REF] Liang | Interval Type-2 Fuzzy Logic Systems: Theory and Design[END_REF].

x 1 *

 1 is the centre of the first cluster, P 1 * is the potential of x 1 * , and β=4/r b 2

Fig. 1 .

 1 Fig. 1. Relation between β and RMSE for IFIS identified by SCA on Friedman dataset.

Fig. 2 .

 2 Fig. 2. Relation between β and RMSE for IFIS adapted by gradient algorithm on Friedman dataset.

Table 1 .

 1 Description of regression datasets.

				Dataset
		Friedman	Energy	Stock	AutoMPG
	Origin	Artificial Real world Real world Real world
	Input variables	5	6	9	5
	Real / integer / nominal	5/0/0	6/0/0	9/0/0	2/3/0
	Instances	1200	365	950	392

Table 2 .

 2 Parameters of FISs and IFISs. -membership functions   (x). Concretely, if β=0.0 the output of the IFIS is determined only by the subsystem with membership functions   (x) while for β=1.0 only the output of the subsystem with non-membership functions   (x) has impact on the IFIS output. The results of the experiments for β with minimum RMSE on testing data is presented in Table3. The FISs and IFISs with adaptation achieved significantly lower RMSE (values in bold based on paired sample t-test at p=0.05) than those without adaptation (SCA). For all the datasets, using adapted IFISs resulted in the decrease in RMSEs.

	Dataset	Friedman	Energy	Stock	AutoMPG
	Type of   (x) and   (x) Gaussian Gaussian Gaussian Gaussian
	Radius r a	0.6	0.7	0.5	0.7
	N of if-then rules	26	7	12	4
	t-norm	MIN	MIN	MIN	MIN
	Experiments were realized for various weights β={0.0,0.1, … ,1.0} of the
	subsystems with non

Table 3 .

 3 RMSE and its standard deviation on benchmark datasets.

			Dataset
	Method	Friedman	Energy	Stock	AutoMPG
	FIS-SCA	1.538±0.020 0.494±0.030 2.378±0.584 3.405±0.254
	IFIS-SCA	1.538±0.020	0.494±0.030	2.378±0.584	3.405±0.254
		(β=0.0)	(β=0.0)	(β=0.0)	(β=0.0)
	FIS-gradient	1.353±0.026 7.443±1.579 1.423±0.227 3.702±0.211
	IFIS-gradient	1.332±0.032	4.776±2.776	1.402±0.219	3.684±0.195
		(β=0.1)	(β=0.5)	(β=0.1)	(β=0.1)
	FIS-Kaczmarz	1.759±0.099 0.619±0.246 1.888±0.267 3.991±0.355
	IFIS-Kaczmarz 1.581±0.159	0.505±0.034	1.708±0.340	3.880±0.554
		(β=0.3)	(β=0.1)	(β=0.2)	(β=0.1)
	FIS-Kalman	1.412±0.031 0.760±0.551 0.944±0.046 2.881±0.113
	IFIS-Kalman	1.353±0.039	0.437±0.031	0.914±0.044	2.825±0.100
		(β=0.2)	(β=0.6)	(β=0.2)	(β=0.3)
	FIS-MPPI	1.411±0.047 0.474±0.038 0.943±0.045 2.866±0.113
	IFIS-MPPI	1.352±0.049	0.435±0.027	0.913±0.044	2.821±0.101
		(β=0.2)	(β=0.6)	(β=0.2)	(β=0.3)


Fakroul Ridzuan

Bin Hashim

John J Soraghan

Lykourgos Petropoulakis

Multi-Classify Hybrid Multilayered Perceptron (HMLP) Network for Pattern Recognition Applications

This paper introduces a Hybrid Multilayered Perceptron (HMLP) based classifier known as the Multi-Classify HMLP network (MCHMLP). This network is shown to be able to enhance the performance accuracy when compared to the conventional HMLP network. The Multi-Classify HMLP network architecture is trained using a Modified Recursive Prediction Error (MRPE). This study uses three benchmark datasets in order to measure the capability of the network. The results show that the proposed Multi-Classify HMLP network provides a significant improvement over the conventional HMLP network for pattern recognition applications.

I. INTRODUCTION

Artificial neural networks (ANNs) can be used to solve problems without the need to create a model of a real biological system. ANNs pattern recognition and classification units are widely used in medicine, banking analysis, stock market prediction, telecommunication industry and many other technology and science areas. Mat-Isa et al., [START_REF] Mat-Isa | An automatic cervical pre-cancerous diagnostic system[END_REF] have shown that neural networks are able to identify the risk level in patients with pre-cancerous cervix. Neural networks are used to predict the stock market to ensure that profits can be enhanced or losses can be minimized [START_REF] Naeini | Stock market value prediction using neural networks[END_REF]. Liu et al. use neural networks as the prediction agent in measuring the earthquake tremor levels that can occur based on the identified features. In [START_REF] Liu | Extraction of if-then rules from trained neural network and its application to earthquake prediction[END_REF] ANNs have been used in aiding earthquake predictions along with the related tremors analysis. In another study, Mat-Isa et al. showed that ANNs can provide good results in identifying the shape of aggregates. The aggregates are identified and separated into groups before being used in construction and those aggregates with good shape are selected to ensure the strength of the concrete at the highest level [START_REF] Mat-Isa | Suitable features selection for the HMLP and MLP networks to identify the shape aggregate[END_REF].

Multilayered Perceptron (MLP) neural network is a conventional example of an ANN network. There are several modified versions of the MLP networks. The Hybrid MLP (HMLP) network is one such version, which has added connections to feed information from the input layer directly to the output layer. Mashor [START_REF] Mashor | Hybrid multilayered perceptron networks[END_REF] has shown an increased accuracy of results provided by the HMLP network compared to the conventional MLP networks. The ability to make predictions of the neural network is dependent on network training. In this study, modifications to the HMLP network will be made to improve the performance of the resulting network. The presented method proposes that the classification process is done repetitively in multi-stages. The output outcomes from a HMLP network in the first stage will be presented as input to another HMLP network in the second stage. Courtney et al. [START_REF] Courtney | A multistage neural network for color constancy and color induction[END_REF] in their research used a multistage neural network to measure the colour constancy and the colour induction using 'NEXUS'. Nexus [START_REF] Sajda | NEXUS: A simulation environment for large-scale neural systems[END_REF] is a large scale interactive neural simulator designed to simulate a huge number of networks with approximately 1.65 million connections. Lai et al. [START_REF] Lai | Multistage neural network metalearning with application to foreign exchange rates forecasting[END_REF] used multi-stage neural networks to predict the foreign exchange rate (FOREX). Multiple training algorithms were used to train the network before the pruning phase took place in order to produce the selected base models. A multi-stage neural network was implemented to recognise the pattern of a mammogram screen [START_REF] Zheng | Multistage neural network for pattern recognition in mammogram screening[END_REF]. Back propagation with a Kalman filter was applied as the training algorithm to perform the classification.

In the work presented in this paper three datasets, taken from the University of California -Irvine (UCI) repository [START_REF] Newman | UCI Repository of machine learning database[END_REF] are used in order to investigate whether the proposed Multi-Classify HMLP network can provide more accurate results than conventional networks. These were: the Breast Cancer Wisconsin (BCW) dataset, the Liver Disorder (LD) dataset the Pima Indians Diabetes (PID) dataset. The remainder of the paper is organized as follows. In section II the proposed method is shown in detail followed by the results and discussion in section III. Conclusions are presented in section IV.

II. MULTI-CLASSIFY HYBRID MULTILAYERED PERCEPTRON NETWORK

It is well known [START_REF] Mashor | Hybrid multilayered perceptron networks[END_REF] that modelling a linear framework using a standard nonlinear MLP network is not a good decision. For this reason a hybrid version of the MLP network was introduced known as the HMLP network. This involved adding linear connections linking the input layer directly to the output layer without going through the hidden layer, an approach which proved to significantly improve performance. Thus, the HMLP network was shown to be sufficiently competent for designing optimal structures both for linear and nonlinear systems. By directly linking the network input to the output node through a number of weighted connections, HMLP allows the formation of a linear system. Additional connections can be made as represented by the dashed line shown in Figure 2. For the output node, the output of the network is given by:

∑ ∑ ∑ = = = +         + = h i i n j n i i ik j n i i ij jk t x w b t x w F w t y 1 1 0 1 0 1 0 1 2 ) ( ) ( ) ( ˆ for h n j ≤ ≤ 1 and m k ≤ ≤ 1 (1)
where 1 ik w denotes the weights of the additional linear connection between the input and output layer, h n is the hidden node while m is the number of outputs of the network.

In this study we propose a modified version of the HMLP network that is formed by repeating the classification process in a second stage. The proposed method is affected by adding another HMLP network to the output layer of the conventional HMLP network.

Figure 3 shows the block diagram of the architecture of the proposed classification method using two HMLP networks. These networks are in cascade so that the output of the first network is the input of the second network. The first HMLP network is fed with training input data. Each data consists of several parameters representing the data features. These features correspond to the various attributes used for the classification in each case. Detailed explanation of these attributes is given in section III of the paper. Here the network was trained in order to find the best performance of the classification process. If the accuracy of the network keeps giving the same results even when the number of iterations is increased, this means the network is near or at its highest convergence point. So, by using the output of the first stage as input for the second network we could improve on the convergence limitations of first HMLP network.

The second stage network will re-classify the wrong classification held at the first stage network. The function of the second HMLP network is to automatically readjust the classification thresholds thus improving the performance obtained compared to a single HMLP.

III. BENCHMARK DATASETS

The performance of the proposed MCHMLP network using MRPE as the training algorithm is evaluated by comparing it with a multi-stage MLP network, and conventional HMLP, MLP, HRBF and RBF networks. The 3 benchmark datasets from the UCI machine learning repository [START_REF] Newman | UCI Repository of machine learning database[END_REF], i.e. the Breast Cancer Wisconsin, the Liver Disorder, and the Pima Indians Diabetes, have been used to examine the performance of the MCHMLP network. The results are then compared to the results from the aforementioned networks.

The Breast Cancer Wisconsin dataset contains a set of subjects with breast cancer and characterized by 9 attributes. The subjects have been diagnosed either with benign tumors or malignant tumors. The dataset has 699 entries with 458 benign tumors and 241 malignant tumors. The 9 attributes are: clump thickness, uniformity of cell size, uniformity of cell shape, marginal adhesion, single epithelial cell size, bare nucleoli, bland chromatin, normal nucleoli and mitoses. The dataset is divided into two groups with 400 comprising the training dataset while 299 make up the testing dataset for the neural networks.

The Liver Disorder dataset classifies the existence of liver disorder in subjects with active alcoholic activity. Blood samples are tested for each subject and, according to the results the subject is placed in group 1 or in group 2. The dataset has 345 entries and is divided into two groups with 145 entries in group 1 and 200 entries in group 2. The 6 attributes are used to classify the subjects into these groups are: mean corpuscular volume (mcv), alkphos alkaline phosphotase, alamine aminotransferase (sgpt), aspartate aminotransferase (sgot), gamma-glutamyl transpeptidase (gammagt) and drinks number of half-pint equivalents of alcoholic beverages drunk per day. For the purpose of training and testing the neural networks, this dataset is divided into two subsets with 200 entries for the training dataset and 145 for the testing dataset.

The Pima Indians Diabetes dataset is taken from female subjects with at least 21 years old of Pima Indians heritage. The subjects have been diagnosed either as healthy or as diabetics. The dataset has 8 attributes: number of pregnancies, plasma glucose concentration, diastolic blood pressure, triceps skin fold thickness, 2-h serum insulin, body mass index, diabetes pedigree, and last but not least, the age of the subject. The database has 768 entries with 268 having tested positive for diabetes. For the purpose of training and testing the neural networks, this dataset is divided into a training set with 400 entries and a testing set with 368 entries.

IV. EXPERIMENTAL RESULTS

Several types of networks have been used and compared in this study using the 3 datasets mentioned earlier. The MCHMLP along with the multi-stage MLP, the HMLP, the Hybrid Radial Basis Function (HRBF), the MLP, and the Radial Basis Function (RBF) networks were used to classify all three datasets. Training was carried out using the following techniques: the proposed MCHMLP, the multi-stage MLP, the HMLP and the MLP networks were trained using the MRPE training algorithm to obtain the data classification. The HRBF and the RBF networks were trained using the Given's Least Square (GLS) for clustering the dataset. Tables 1, 2 and 3 show the performance results of the neural networks using the three datasets. It is clear from these tables that the MCHMLP outperforms all the other networks, obtaining the highest overall accuracy for every dataset used. The performances of the multi-stage MLP, the HMLP and the MLP networks are quite good, however they are significantly lower that the performance of the MCHMLP. The results clearly show that when additional networks are added to the conventional HMLP and MLP structures the accuracy of the results increases considerably for a relatively small increase in the number of network nodes due to the introduced second stage. The RBF family of networks (RBF and HRBF) are able to classify the three datasets but no better than the MLP family.

The standard deviation in this study is used as the stopping criterion of network iteration to measure the convergence level. The MCHMLP shows the highest convergence and accuracy performance for the BCW dataset by using 15 and 6 nodes at the first and second stage networks with 2.32 and 0.93 of SD respectively. For the Liver Disorder dataset, the f network needed 9 nodes for the first and 2 nodes for the second stages. In the case of the PID dataset 12 and 4 nodes were required in each stage respectively. From the tables of the results it is obvious that the MLP family needs a larger number of nodes to reach the highest convergence when compared to the RBF family. Billings et al. in their study state the RBF family is able to perform faster than MLP networks, where the weight can be pre-fixed since they involve only linear optimization [START_REF] Billings | Generalized multiscale radial basis function networks[END_REF]. Although the RBF networks may be able to produce faster training sessions, are unable to produce as accurate results. This is well illustrated in our test cases. These results are also supported by Mat-Isa et al. who, in their research, found that the performance of the RBF family of networks cannot be better than that of the MLP family [START_REF] Mat-Isa | An automatic cervical pre-cancerous diagnostic system[END_REF].

The results in Tables 2 and3 also indicate that the MCHMLP offers superior performance however the difference in performance is not as clear as in Table 1. In the case of Liver Disorder (Table 2) and Pima Indian Diabetes (Table 2) problem occurs when there is a large overlap between two groups in the datasets. The datasets are divided into groups A and B but, parts of the data belong to either group A or group B. So, datasets with a high overlap lead to a difficult classification process compared to datasets with a small overlap. The MCHMLP network is shown to be able to obtain the best classification results under all tested conditions in this study compared to the other networks. Nonetheless, datasets with a high overlap reduce the accuracy of the classification. However, Tables 1, 2 and 3 shows the MCHMLP network is capable to generalize the datasets in training phase and performs good results in testing phase.

V. CONCLUSION

In this study, the multi-classify HMLP network trained with MRPE as the training algorithm has been proposed for pattern recognition applications. The performance of the proposed network was measured by using three benchmark datasets. The results show that the multiclassify HMLP network is able to classify the Breast Cancer Wisconsin, Liver Disorder and Pima Indian Diabetes with significant accuracy (and in some cases with very high accuracy) and, in all cases, the performance was better than any of the other networks used in this study. Moreover, in all three cases examined, the introduction of the second stage network resulted in improved standard deviation results at the second stage, a direct indication of improved convergence of the network. The additional overhead of extra nodes and increased training time was not significant.

Fig. 2 :

 2 Fig. 2: A schematic diagram of HMLP network with one hidden layer.

Fig. 3 :

 3 Fig. 3: Block diagram of the proposed network architecture.It is considered that the best possible performance has been attained when the error of the network has reached a small value and remains constant even though the number of nodes increases. At that moment it is considered that the network has obtained the highest level of convergence. Several statistical techniques are available to represent the error of system such as Mean Square Error (MSE), Sum Square Error (SSE), and Standard Deviation (SD). In this case the Standard Deviation is used to represent the error of the network.

Fig. 4 :

 4 Fig. 4: A graph of Standard Deviation variation with increasing Number of Network NodesFigure4shows that the smallest SD occurred after using 9 nodes (arrow). After this point the value of SD remains unchanged although the number of nodes increases. With the highest convergence reached, a set of test data is provided to test the HMLP network. A good trained network is indicated by the ability to represent the whole dataset at the training phase. Once the best performance has been achieved for the first HMLP its outcome is then used as input to the second stage HMLP network. The same training and testing procedures applied to the first HMLP network are also used to train and test the second stage HMLP network. In this case only one feature is represented for each data input since the first stage HMLP network provides only one feature at the output layer as outcome.Several aspects of the network training must be fulfilled in order to obtain results with the highest possible accuracy. The type of network structure must be matched

  There are many types of training algorithms for training neural networks to increase prediction. Popular training algorithms include Back Propagation (BP), Lavenberg Marquardt (LM), Bayesian Regularization (BR), Recursive Least Squares (RLS), and Recursive Prediction Error (RPE). Mashor in [6] has shown that the accuracy could be improved if some modifications were made to the RPE training algorithms. The Modified Recursive Prediction Error (MRPE) is the modified version of the RPE training algorithm which can increase the accuracy of predictions of the conventional RPE algorithm when training the HMLP network. He also stated that the MRPE training algorithm is better suited for use with the HMLP network as it resulted in more accurate predictions than other approaches.

Table 1 :

 1 Neural Networks performance using the Breast Cancer Wisconsin dataset for the training and testing phases.

		Accuracy, (%)	Number	Standard
	Network	Training	Testing	of Nodes	Deviation,
		Phase	Phase		(SD)
	MCHMLP	97.25	97.03	15(fs)	2.32(fs)
				6(ss)	0.93(ss)
	Multi-stage MLP	94.50	92.30	14(fs)	2.48(fs)
				2(ss)	0.95(ss)
	HMLP	95.00	94.66	15	2.32
	MLP	81.75	80.63	14	2.48
	HRBF	82.00	76.58	1	5.20
	RBF	75.25	73.58	1	5.92
	*first stage (fs), second stage (ss).

Table 2 :

 2 Neural Networks performance using the Liver Disorder dataset for the training and testing phases.

		Accuracy, (%)	Number	Standard
	Network	Training	Testing	of Nodes	Deviation,
		Phase	Phase		(SD)
	MCHMLP	77.00	75.86	9(fs)	4.69(fs)
				2(ss)	3.76(ss)
	Multi-stage MLP	71.00	70.34	5(fs)	6.30(fs)
				3(ss)	3.96(ss)
	HMLP	74.50	73.17	9	4.69
	MLP	69.00	67.43	5	6.30
	HRBF	68.00	60.00	2	6.50
	RBF	44.50	40.00	2	6.70
	*first stage (fs), second stage (ss).

Table 3 :

 3 Neural Networks performance using the Pima Indians Diabetes dataset for the training and testing phases.

		Accuracy, (%)	Number	Standard
	Network	Training	Testing	of Nodes	Deviation,
		Phase	Phase		(SD)
	MCHMLP	77.50	77.05	12(fs)	3.68(fs)
				4(ss)	3.06(ss)
	Multi-stage MLP	65.25	64.90	14(fs)	3.78(fs)
				6(ss)	3.74(ss)
	HMLP	73.50	72.98	12	3.68
	MLP	64.75	62.57	14	3.78
	HRBF	58.00	51.52	2	5.92
	RBF	56.00	51.52	2	4.85
	*first stage (fs), second stage (ss).