H Goh
email: hlgoh@i2r.a-star.edu.sg

J.-H Lim

C Quek

Fuzzy Associative Conjuncted Maps Network

Hanlin Goh, Joo-Hwee Lim, Member, IEEE, and Chai Quek, Member, IEEE Abstract-The fuzzy associative conjuncted maps (FASCOM) is a fuzzy neural network that associates data of nonlinearly related inputs and outputs. In the network, each input or output dimension is represented by a feature map that is partitioned into fuzzy or crisp sets. These fuzzy sets are then conjuncted to form antecedents and consequences, which are subsequently associated to form IF-THEN rules. The associative memory is encoded through an offline batch mode learning process consisting of three consecutive phases. The initial unsupervised membership function initialization phase takes inspiration from the organization of sensory maps in our brains by allocating membership functions based on uniform information density. Next, supervised Hebbian learning encodes synaptic weights between input and output nodes. Finally, a supervised error reduction phase fine-tunes the network, which allows for the discovery of the varying levels of influence of each input dimension across an output feature space in the encoded memory. In the series of experiments, we show that each phase in the learning process contributes significantly to the final accuracy of prediction. Further experiments using both toy problems and real-world data demonstrate significant superiority in terms of accuracy of nonlinear estimation when benchmarked against other prominent architectures and exhibit the network's suitability to perform analysis and prediction on real-world applications, such as traffic density prediction as shown in this paper.

Index Terms-Fuzzy associative conjuncted maps (FASCOM), fuzzy associative memory, fuzzy neural networks, Hebbian learning, Iris plant classification, multivariate data analysis, Nakanishi's nonlinear estimation tasks, neurofuzzy systems, supervised learning, traffic density prediction, two-spiral classification, unsupervised learning.

I. INTRODUCTION

F UZZY neural networks [1]- [4] are hybrid intelligent systems that combine the humanlike knowledge representation and reasoning of fuzzy logic [START_REF] Zadeh | Fuzzy sets[END_REF], with the adaptation and learning capabilities of neural networks. As such, they maximize the desirable properties of both fuzzy logic and neural networks, and become powerful tools for learning and inference. The methods to encode knowledge into fuzzy systems are common difficulties. These methods include membership function determination, fuzzy rule identification, and fuzzy rule inference. These problems may be alleviated by using brain-inspired features, such as self-organization. On the other hand, neural networks are typically black boxes and analysis of a trained network is difficult. Thus, the integration of fuzzy logic in neural networks can significantly improve the transparency of the network. It is hoped that while data prediction remains highly accurate, some form of analysis of the complex multivariate systems they represent is still possible.

A fuzzy neural network belongs to one of two paradigms: 1) linguistic fuzzy model (e.g., Mamdani model proposed in [START_REF] Mamdani | An experiment in linguistic synthesis with a fuzzy logic controller[END_REF]) and 2) precise fuzzy model (e.g., Takagi-Sugeno-Kang (TSK) model proposed in [START_REF] Takagi | Derivation of fuzzy control rules from human operator's control actions[END_REF]- [START_REF] Sugeno | Structure identification of fuzzy model[END_REF]). In the linguistic model given in [START_REF] Lin | Neural Fuzzy Systems: A Neuro-Fuzzy Synergism to Intelligent Systems[END_REF], both the antecedent and consequence are fuzzy sets, whereby for a rule if is and is then is [START_REF] Lin | Neural Fuzzy Systems: A Neuro-Fuzzy Synergism to Intelligent Systems[END_REF] where and are the input vector and output value, respectively, and represent linguistic labels, and is the number of inputs. In the precise fuzzy model given in [START_REF] Zadeh | Fuzzy logic, neural networks and soft computing[END_REF] and ( 3), the antecedent is a fuzzy set and the consequence is formed by linear equations [START_REF] Guillaume | Designing fuzzy inference systems from data: An interpretability-oriented review[END_REF], [START_REF] Casillas | Interpretability Issues in Fuzzy Modeling[END_REF] if is and is then (2)

(3)

where is the number of rules, and is the strength of . Linguistic fuzzy models focus on interpretability and lack in accuracy, while precise fuzzy models focus on accuracy and lack human interpretability [START_REF] Casillas | Interpretability Issues in Fuzzy Modeling[END_REF], [START_REF] Ang | RSPOP: Rough set-based pseudo outerproduct fuzzy rule identification algorithm[END_REF].

In this paper, we propose the fuzzy associative conjuncted map (FASCOM) architecture, which is an implementation of a linguistic fuzzy model that uses associations to learn relationships between input and output fuzzy sets. Its main objective is to provide a means for highly accurate data prediction, with some capability of interpretability of the multivariate system it models. Its design follows a line of work on fuzzy associative memories, pioneered by [START_REF] Kosko | Neural Networks and Fuzzy Systems: A Dynamical Systems Approach to Machine Intelligence[END_REF]. Though aware of its drawback in terms of space complexity, FASCOM retains the original simple concept of using a fuzzy associative matrix to represent input-output relationships. Its focus is to improve data prediction accuracy through the introduction of several new features in both data representation and its encoding process. For example, typical associative networks may result in nonperfect retrieval, due to crosstalk among nonmutually orthogonal input patterns. We proposed to alleviate this problem by conjuncting antecedent and consequence fuzzy sets, as in if is and is then is and is (4) where is the output vector and is the number of outputs. In doing so, we intrinsically form representations of multiple antecedents and consequences within the network.

By learning data obtained from past experience, fuzzy neural networks can identify fuzzy IF-THEN rules between the antecedents and consequences of a multivariate system. This is achieved by means of either unsupervised, supervised, or a combination of both unsupervised and supervised techniques.

In unsupervised approaches [START_REF] Lin | Neural-network-based fuzzy logic control and decision system[END_REF]- [START_REF] Quek | A novel approach to the derivation of fuzzy membership functions using the Falcon-MART architecture[END_REF], learning algorithms that are used to identify fuzzy rules are only dependent on the natural statistics of the data and are not concerned with the relationships between the input and the outputs. This is followed by processing involving the application of neural network techniques (e.g., clustering, self-organization) to adjust identified rules. Due to this reliance on training data, nonrepresentative data may lead to ill-defined rules and hence producing an inaccurate system model.

For supervised approaches [START_REF] Lee | A new neuro-fuzzy identification model of nonlinear dynamic systems[END_REF], [START_REF] Ishibuchi | Interpolation of fuzzy if-then rules by neural networks[END_REF], supervised learning methods (e.g., backpropagation [START_REF] Werbos | Backpropagation: Past and future[END_REF]) are used to identify rules by mapping inputs to outputs. As such, it is sensitive to the correlation between input and output data. However, a drawback for this approach is that the semantics of the fuzzy neural network remains opaque, which contradicts the original aim of combining fuzzy logic with neural networks.

FASCOM is encoded through a learning process consisting of three phases. The first phase involves an unsupervised initialization of membership functions that quantizes the input data. The second phase is a supervised Hebbian learning process that encodes synaptic weights between conjuncted fuzzy sets. The third is a supervised error reduction phase that fine-tunes the partially encoded network and completes the learning process. The entire learning process is performed in an offline batch mode manner, similar to [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF] and [START_REF] Jang | ANFIS: Adaptive-network-based fuzzy inference system[END_REF]. This is as opposed to online learning methods [START_REF] Kasabov | Evolving fuzzy neural networks for supervised/unsupervised online[END_REF], [START_REF] Kasabov | DENFIS: Dynamic evolving neural-fuzzy inference system and its application for time-series prediction[END_REF] where the models are incrementally updated with every new training instance, resulting in fuzzy rules that adaptively evolve with the sequence of training samples. On the other hand, if new data is provided to FASCOM, the entire model needs to be retrained.

The proposed architecture also bears similarity to the popular fuzzy min-max neural networks and its variants [START_REF] Simpson | Fuzzy min-max neural networks-Part 1: Classification[END_REF], [START_REF] Abe | A method for fuzzy rules extraction directly from numerical data and its application to pattern classification[END_REF], whereby hyperboxes are placed within the feature space. A learning algorithm then dynamically adjusts the quantity of these hyperboxes or modify their sizes through and expansion and contraction process, such that the overlap between classes is eliminated. However, in many real-world scenarios, data between classes may not be easily separable. As such, a gradual assessment of membership between classes is also essential, so that the classes are not mutually exclusive [START_REF] Joshi | On neurobiological, neuro-fuzzy, machine learning, and statistical pattern recognition techniques[END_REF]. This is also what FASCOM aims to achieve. However, these methods differ with FASCOM in their manner of deriving fuzzy regions. While the fuzzy min-max networks use a supervised approach to assign and adjust fuzzy sets, FASCOM uses the distribution of the data to initialize the fuzzy regions in an unsupervised manner. Supervision is only introduced in the later phases of encoding whereby an associative network is encoded.

Fuzzy IF-THEN rules provide a localized linguistic understanding that is dependent on the locations of their membership functions. But little information can be extracted about the influences of input dimension on the output. This can be somewhat achieved using various well-established methods to analyze and reduce dimensions of a multivariate system, such as principle component analysis (PCA) [START_REF] Pearson | On lines and planes of closest fit to systems of points in space[END_REF]. PCA is commonly used for dimension reduction, but fails to provide a description of the relationships between the input and output dimensions. Other dimension reduction methods have also been used in the fuzzy neural network literature [START_REF] Ang | RSPOP: Rough set-based pseudo outerproduct fuzzy rule identification algorithm[END_REF], [START_REF] Nakanishi | A review and comparison of six reasoning methods[END_REF], whereby dimension selection is based on the existence of fuzzy rules identified in the dimensions. However, while all these methods provide a general idea of which dimensions are important and which are not, they do not specify to which portions of the output the inputs exert their influences on. Interestingly, besides being able to identify fuzzy rules for data estimation, the encoded FASCOM network can also provide insights on how the influence of an input dimension varies across an output feature space.

The FASCOM architecture introduces the following key features, with their benefits explained this paper.

1) The unsupervised learning phase (Section III-A) that initializes membership functions based on uniform information density results in an improvement of the output precision of nonlinear estimation as compared to one with a proportional distribution of membership functions in the feature space. Since feature maps are organized based on the information distribution of the data in an unsupervised manner, it is a quick and effective approach for initialization as it only involves a mapping between uniformed and nonuniformed scales of the feature maps. There is evidence that this sort of neuron organization is extensively seen in the sensory maps of humans and other organisms [START_REF] Suga | Cortical computation maps for auditory imaging[END_REF]- [START_REF] Plumbley | Do cortical maps adapt to optimize information density?[END_REF]. 2) A supervised error reduction learning phase (Section III-C) that fine-tunes the network is introduced to effectively enhance the accuracy of data prediction. It also presents the influences of an input dimension on different portions of an output feature space. Conversely, we can identify the influences of different inputs on a particular portion of an output. It raises the level of understanding, in addition to fuzzy rules, by allowing the user to determine which factors (or dimensions) are the likely causes of particular ranges of the output. This is especially beneficial for nonlinear systems whereby the resultant output might be influenced differently across output space. Such analytical information can be easily extracted from the network because weights relating an input feature with various portions of the output space are learned in this phase. Unlike feature selection methods that discard noninfluential dimensions based on global statistics, our method is able to retain input dimensions that have strong influences on small portions of the output. 3) Input and output conjunctions reduce and possibly remove crosstalk in nonmutually orthogonal input patterns, as demonstrated in Sections V-A and V-B. This enables the representation of multiple antecedents and multiple consequences, as opposed to singular ones. Furthermore, it not only boosts information representation in an associative neural network, but also enhances the fuzzy-logical linguistic representation. As such, this feature is in essence a true amalgamation of neural and fuzzy concepts. Through the series of benchmark experiments, we also identified some possible limitations of high complexity in terms of space usage. Like many neural network models, there is also a requirement of parameter setting for optimal results. Based on these factors, we suggest that this architecture is especially suitable for applications that do not need real-time or online learning, but rather require fast and accurate prediction, or those needing descriptive information on relations between input and dimensions for the multivariate system.

II. FASCOM NETWORK STRUCTURE

The FASCOM architecture is a six-layer neural network as depicted in Fig. 1. The six layers are, namely, input fuzzifier (layer 1), single antecedent (layer 2), multiple antecedent (layer 3), multiple consequence (layer 4), single consequence (layer 5), and output defuzzifier (layer 6). The outputs of each layer are projected as the inputs of the next layer in a feedforward manner. Layers 1-3 represent input layers, while layers 4-6 represent output layers.

The inputs and outputs of the network are represented as vectors and , where and denote the number of input and output linguistic variables, respectively. While the output vectors are nonfuzzy, the inputs may be either fuzzy or crisp. Input fuzzification and output defuzzification are automatically accomplished in layers 1 and 6, respectively. Table I lists the description and quantity of the nodes and feature maps in each layer of the network. In biological neural circuitry, feature maps may be subsequently integrated in association areas for purposeful action or cognitive function [START_REF] Kandel | Principles of Neural Science[END_REF]. Similarly, the inputs of the network are associated with the outputs in the hybrid associative memory between layers 3 and 4.

s in layer 3 are linked to s in layer 4 via a association, with each representing a fully connected heteroassociative associative network between and . If every is connected to every , there will be a total of such associative networks. In each network, there are connections, each representing a fuzzy IF-THEN rule (i.e., if , then ). Memory between and is stored as synaptic weight

. The synaptic weights

Connections are further weighted by a modulatory weight that signifies a connection between with . Modulatory weights between an pair can be structured as a vector given by [START_REF] Mamdani | An experiment in linguistic synthesis with a fuzzy logic controller[END_REF] As a result, a connection between nodes and is weighted by resultant weight , which is the product of and , and can be computed by [START_REF] Takagi | Derivation of fuzzy control rules from human operator's control actions[END_REF] or expressed in the following as a correlation matrix

Through the processes of conjunction in layers 3 and 4, and association between the layers, this correlation matrix may end up being large. This problem of redundancy from population coding is also observed in biological systems, with only a small portion of cells in a population coded feature map responding only to stimulation of a specific region. However, it is undesirable in a computational paradigm due to its spatial redundancy. Similarly to [START_REF] Kosko | Neural Networks and Fuzzy Systems: A Dynamical Systems Approach to Machine Intelligence[END_REF], since each element within the association matrix represents a fuzzy rule, this also means that there will be a large number of fuzzy rules generated. One way to circumvent this problem is to extract a ranked list of fuzzy rules based on their computed resultant weights, thus enabling the identification and retention of those rules with higher impact and removing noninfluential fuzzy rules. Other methods to tackle this problem have also been introduced in [37]- [START_REF] Sussner | Implicative fuzzy associative memories[END_REF].

Layer 5 has a structure similar to layer 2, and consists of output maps . has output label nodes , with each node symbolizing an output linguistic label of an output variable. The total number of nodes in this layer is . Layer 6 consists of output linguistic nodes , corresponding to the size of output vector , with representing the th output linguistic variable. This layer converts fuzzy outputs from layer 5 to a crisp output.

As a convention, the output of a node or map is denoted as with subscripts specifying its origin. For example, represents the output of node and denotes the output vector of . All outputs from a layer are propagated to the corresponding inputs at the next layer with unity link-weight through an existing connection, except for the connections between layers 3 and 4 where connections are weighted by resultant weight .

A. Layer 1: Input Fuzzifier nodes in the input fuzzifier layer represent input linguistic variables such as "width," "height," etc. This layer fuzzifies the input vector using the following into a vector of fuzzy membership functions [START_REF] Sugeno | Structure identification of fuzzy model[END_REF] where is the membership function of . If a fuzzy input is presented to a node in this layer, the node simply redirects it as the output of the node. This layer first assigns as the centroid of the membership function with value 1. If no smoothing is involved, . Two smoothing functions were identified for our experiments: Gaussian defined by [START_REF] Guillaume | Designing fuzzy inference systems from data: An interpretability-oriented review[END_REF], and Laplacian of Gaussian defined by [START_REF] Casillas | Interpretability Issues in Fuzzy Modeling[END_REF] (10) [START_REF] Casillas | Interpretability Issues in Fuzzy Modeling[END_REF] where is the variance of the function. is used to model lateral inhibition in the biological neural circuitry [START_REF] Kandel | Principles of Neural Science[END_REF], [START_REF] Marr | The theory of edge detection[END_REF],

B. Layer 2: Single Antecedent

sIn layer 2, the node represents the th input label, such as "low," "medium," "high," etc., corresponding to the th input linguistic variable and grouped into . This layer receives fuzzified inputs from layer 1 and computes the fuzzy subsethood measure [START_REF] Yen | Fuzzy Logic: Intelligence, Control, and Information[END_REF], which is the degree of one fuzzy set belonging to another, for each corresponding input label. Using the minimum T-norm operator for the intersection operation, the subsethood measure of and can be approximated using [START_REF] Ang | RSPOP: Rough set-based pseudo outerproduct fuzzy rule identification algorithm[END_REF] where and are positive and negative components of , respectively, and is the membership function of input label . The output of is described by [START_REF] Kosko | Neural Networks and Fuzzy Systems: A Dynamical Systems Approach to Machine Intelligence[END_REF] An example of this computation is illustrated in Fig. 2. Based on the input with , is first computed through a smoothing of . A positive vector is derived from the subsethood measure computed separately from all five neurons with membership functions , , , , and , with only positive values of [i.e., ], while treating the negative values as 0. For example, this operation results in . Similarly, a negative vector is computed as by ignoring positive membership values and computing the subsethood measure of the absolute value of the negative ones . In practical implementation, these two vectors may be computed sequentially and stored separately. Finally, is subtracted from . This essentially combines the two vectors of fuzzy subsethood measures into a single vector as described in [START_REF] Kosko | Neural Networks and Fuzzy Systems: A Dynamical Systems Approach to Machine Intelligence[END_REF]. As such, the computed .

C. Layer 3: Multiple Antecedent

In the multiple antecedent layer, each node represents either a single antecedent, such as "if height is short," "if speed is fast," etc., or multiple antecedents, such as "if height is short and weight is light," "if speed is fast and fuel is low and temperature is high," etc. In the example shown in Fig. 3, and representing single antecedents are copies of and . In addition, representing a conjunction of labels of the linguistic variables "height" and "weight" contains four nodes that represent the four multiple antecedents, namely, "if height is short and weight is light," "if height is short and weight is heavy," "if height is tall and weight is light," and "if height is short and weight is heavy." If s are exhaustively conjuncted to form s, the number of s generated will be , where is the number of input dimensions. This means that the number of s grows exponentially with the number of s; as a result, the number of s generated by exhaustive conjunction of all s will be significantly larger with every subsequent dimension introduced. The space complexity for the number of is , which poses a limitation on high-dimensional inputs.

To handle data sets with high dimensionality, instead of exhaustively conjuncting all combinations of s, the s are selectively conjuncted into s. One simple way to do this is to limit the number of preceding s that can be combined into each . If the threshold for the number of s per is , the number of s generated will be . For example, in a data set with ten input dimensions (i.e., ten s), the exhaustive conjunction method will produce 1023 s. However, if the number of preceding s is limited to two, then the number of s will be 55. This step was introduced in the experiment described in Section V-D3.

The output of each node is known as the firing strength, and results from conjunctions of . Using the algebraic product T-norm operator, is given by [START_REF] Lin | Neural-network-based fuzzy logic control and decision system[END_REF] where takes on a link weight factor of 1 if there is a connection between and , and is otherwise 0. Equation [START_REF] Hayashi | Construction of fuzzy inference rules by NDF and NDFL[END_REF] describes the output of as a vector of firing strengths (15)

D. Layer 4: Multiple Consequence

A node in layer 4 may represent either a single consequence, for example, "then age is young," "then driving is very dangerous," etc., or multiple consequences, such as "then age is young and diet is healthy," "then driving is very dangerous and apply the brake a little," etc. Multiple consequences may be especially useful in applications requiring multiple simultaneous outputs, or data associations with a one-to-many or many-to-many relationship.

Because these nodes represent the "then" clause of a fuzzy IF-THEN rule, when is linked with , a fuzzy IF-THEN rule is formed between them (i.e., if , then ). This rule can be abbreviated as . Rules are weighted by the resultant weights , which are determined through the learning processes (details will be described later in Sections III-B and III-C), with rules that have higher weights being more significant than those with lower weights. During recall, the output of a node is its activation level , as shown in [START_REF] Yager | Modeling and formulating fuzzy knowledge bases using neural networks[END_REF], and obtained via the process to be described in Section IV (16)

E. Layer 5: Single Consequence

In layer 5, represents output labels such as "light," "medium," "heavy" of the corresponding output linguistic variable . Each node is represented by a membership function similar to the nodes in layer 2. Equation ( 17) defines a node's output as the maximum activation level of consequence nodes connected to it [START_REF] Quek | A novel approach to the derivation of fuzzy membership functions using the Falcon-MART architecture[END_REF] where is a link weight factor of 1 when there exists a connection between nodes and , and is 0 otherwise.

F. Layer 6: Output Defuzzifier

Layer 6 consists of nodes that represent output variables such as "speed," "temperature," etc. The function of this layer is to convert the fuzzy set outputs of layer 5 into crisp outputs. Three defuzzification schemes are predominant in the literature [44], namely, the center of area, center of maxima, and mean of maxima. A suitable defuzzification scheme , described in [START_REF] Lee | A new neuro-fuzzy identification model of nonlinear dynamic systems[END_REF], is applied for every [START_REF] Lee | A new neuro-fuzzy identification model of nonlinear dynamic systems[END_REF] In the subsequent set of experiments, the center of area defuzzification scheme defined in [START_REF] Ishibuchi | Interpolation of fuzzy if-then rules by neural networks[END_REF] was used [START_REF] Ishibuchi | Interpolation of fuzzy if-then rules by neural networks[END_REF] where is the aggregated output membership function.

III. THREE-PHASE LEARNING PROCESS

Similarly to [START_REF] Quek | The POP learning algorithms: Reducing work in identifying fuzzy rules[END_REF], FASCOM's three-phase learning process consists of three phases. The first phase initializes membership functions for all input and output labels based on uniform information density. Next, Hebbian learning computes the synaptic weights of connections between nodes in layer 3 and those in layer 4. The connections between the input and output conjuncted maps are fine-tuned through a supervised error reduction final phase. As illustrated in Fig. 4, the output layers 4-6 assume the same functional processes as the input layers 1-3, of fuzzification (Section II-A), subsethood measure computation (Section II-B), and conjunction formation (Section II-C), respectively.

To avoid confusion, an alternate symbol is used to represent the output of a node or a map that has been derived via a reverse propagation process (i.e., fuzzification for layer 6, subsethood computation for layer 5, and conjunction formation for layer 4). For example, represents the fuzzified output of and symbolizes the vector of firing rate of nodes in .

A. Unsupervised Membership Function Initialization Phase

A linguistic label is represented by a neuron. In the experimentations, three types of membership functions for these neurons were identified, namely, discrete, rectangular, and Gaussian. The first two offer a crisp representation of feature space, while the third provides a fuzzy representation. We adopt a three-stage approach for membership function initialization: 1) calculation of centroids, 2) initial allocation of neurons, and 3) equalization for uniform information density.

Cyclic and scalar maps are used to represent the two different natures of fuzzy linguistic variables. Examples of cyclic variables include the visual submodalities of hue and orientation. Swindale [START_REF] Swindale | How different feature spaces may be represented in cortical maps[END_REF] suggested that cyclic variables may have an advantage that the maps representing these variables can wrap around without compromising continuity and edge effects. For scalar maps containing Gaussian fuzzified neurons, it is necessary to remove edge effects by padding the two ends of the represented space.

1) Computing Centroids of Membership Functions: The number of neurons in an input or output map representing labels of a linguistic variable is first fixed. The lower the separability between data classes is, the higher requirement for acuity will be, thus increasing the required to model the data. A brief discussion on this is presented in Section V-B.

For a neuron with centroid , its membership function is determined by for scalar variables for cyclic variables [START_REF] Werbos | Backpropagation: Past and future[END_REF] where and are, respectively, the maximum and minimum values of represented by the map.

2) Initial Uniform Neuron Allocation: The rectangular function for a neuron defined in [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF] results in a crisp representation of information, identical to that of the classical set theory for otherwise [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF] where is the center of the membership function for neuron in linguistic variable , and defines the distance of the rectangular membership function from its center. The discrete membership function is a special case of rectangular membership functions, where membership functions do not overlap. This occurs when . Unlike crisp neurons, fuzzy neurons allow for a gradual assessment of membership functions and are biologically similar to the overlapping nature of receptor fields on the periphery receptor sheets. In particular, the Gaussian function, defined in [START_REF] Jang | ANFIS: Adaptive-network-based fuzzy inference system[END_REF], fits the sensitivity curve of the receptive field of a retinal ganglion cell [START_REF] Koch | Passive denritic trees[END_REF] (22

)
where is the variance of the Gaussian function.

3) Uniform Information Density Equalization:

There is a disproportion in neuron allocation [START_REF] Kandel | Principles of Neural Science[END_REF] within various sensory maps in the brain that depends on the usage or sensitivity requirements of a species or an individual. For example, in the auditory cortex of a bat, the region corresponding to the frequency of its biosonar is especially large [START_REF] Suga | Cortical computation maps for auditory imaging[END_REF]. This is also observed in other sensory systems, such as the visual [START_REF] Rovamo | Cortical magnification factors predicts the photopic contrast sensitivity of peripheral vision[END_REF], [START_REF] Azzopardi | The overrepresentation of the fovea and adjacent retina in the striate cortex and dorsal lateral geniculate nucleus of the macaque monkey[END_REF] and somatosensory [START_REF] Kuffler | From Neuron to Brain: A Cellular Approach to the Function of the Nervous System[END_REF] systems, and adapted through evolution and individual experiences [START_REF] Das | Plasticity in adult sensory cortex: A review[END_REF]. It was suggested in [START_REF] Plumbley | Do cortical maps adapt to optimize information density?[END_REF] that the disproportion is optimized by uniform cortical information density across the map for efficient information transmission. In FASCOM, membership functions are allocated based on the varying information density of the training data across feature space. This disproportion in allocation [Fig. 5(d . With more neurons allocated to the areas with higher information concentration, the output will be more responsive and precise. A similar concept to handle the sensitivity requirements of a fuzzy control system was proposed in [START_REF] Kong | Adaptive fuzzy systems for backing up a truck-and-trailer[END_REF] to allow for a finer fuzzy control of a truck reversing into a dock, when the truck is nearer to the dock.

B. Supervised Hebbian

) 23
where which is the maximum possible absolute change in synaptic weight defined in [START_REF] Simpson | Fuzzy min-max neural networks-Part 1: Classification[END_REF]. Equation ( 24) then updates by aggregating it with the current (24)

2) Modeling Learning Mechanisms: is determined by the learning mechanisms of synaptic plasticity, forgetting, and long-term facilitation [START_REF] Squire | Memory : From Mind to Molecules[END_REF], from which the following three concepts governing the behavior of learning were derived.

1) The effects of synaptic plasticity, forgetting, and their resultant combination are approximated as constants through time.

2) The effect of long-term facilitation is initially high, but decays as memory strengthens. When memory stabilizes, the effect of facilitation is insignificant.

3) The initial effect of long-term facilitation is very much greater than that of the resultant effect of synaptic plasticity and forgetting. The amount of facilitation is dependent on the current strength of memory (i.e.,

), whereby the larger is, the stronger the memory. Since facilitation decreases as memory strengthens, we describe in (25) the effects of the three learning mechanisms as a decaying sigmoid function [START_REF] Simpson | Fuzzy min-max neural networks-Part 1: Classification[END_REF] where is the initial effect of facilitation, is the resultant time constant component from the effects of synaptic plasticity and forgetting and and define the slope of the sigmoid function. Typically, . In cases where the training data is noisy, it is suggested that the value of be lowered. Keeping and constant, the amount of time required for memory development is affected by and , such that as increases or decreases, the memory will take a longer time to be developed. It is noted however that, if is set too low or is set too high, the learning process may be undesirable. Preferably, . When suitable values are selected for the parameters, the three learning mechanisms can be effectively modeled using the sigmoid function shown in Fig. 6. The sigmoid function governs how the maximum possible absolute change in synaptic weight is limited by the current absolute synaptic weight (i.e., strength of memory) between two neurons and . This process of modifying synaptic weights is performed for all training samples, and eventually, depending on the quality of the training data, the learning process will result in some memories being strong like long-term memory and possibly some being weak like short-term memory.

C. Supervised Error Reduction Phase

After deriving the synaptic weights in the supervised Hebbian learning phase described in Section III-B, the final phase in the learning process reduces output error and fine-tunes the network. This phase discovers the influence of an input conjuncted map on a consequence node and stores it as modulatory weight . As shown in Fig. 7, modulatory weights form asymmetrical connections between input and output conjuncted maps such that influence of may differ in magnitude for different portions and output conjuncted map

. The value of refers to the amount of influence on and their meanings are listed in Table II.

TABLE II MEANING OF VALUES OF MODULATORY WEIGHTS

The flowchart illustrated in Fig. 8 explains the supervised error reduction algorithm, while Table III lists the symbols used in this section.

The algorithm first initializes modulatory weights to 1.0 using [START_REF] Abe | A method for fuzzy rules extraction directly from numerical data and its application to pattern classification[END_REF]. For every training input-output pair , the temporary modulatory weights are assigned by [START_REF] Joshi | On neurobiological, neuro-fuzzy, machine learning, and statistical pattern recognition techniques[END_REF]. Memory is then recalled using the recalling process explained in Section IV. An initial error for each output node is then computed via [START_REF] Pearson | On lines and planes of closest fit to systems of points in space[END_REF]. Next, using only one input conjuncted map in each iteration, memory is again recalled. A new error for each node is then computed using [START_REF] Suga | Cortical computation maps for auditory imaging[END_REF]. With this, we apply (30) to derive the change in error with respect to the initial error. Using this information, the temporary modulatory weights for the training instance are modified using [START_REF] Azzopardi | The overrepresentation of the fovea and adjacent retina in the striate cortex and dorsal lateral geniculate nucleus of the macaque monkey[END_REF]. After processing all s for all training samples, the modulatory weights are updated with [START_REF] Kuffler | From Neuron to Brain: A Cellular Approach to the Function of the Nervous System[END_REF]. This entire process is then repeated until a terminating condition is met (31) [START_REF] Kuffler | From Neuron to Brain: A Cellular Approach to the Function of the Nervous System[END_REF] The modulatory weights are updated using a modulatory factor defined in [START_REF] Das | Plasticity in adult sensory cortex: A review[END_REF] based on three factors: 1) the change in squared error, 2) the number of preceding , and 3) the current epoch count with respect to the maximum number of epochs. The change in modulation decreases with time and is more sensitive for s with fewer preceding . The effects of the change in squared error on the modulatory weights are as follows.

1) Squared-error increases ( ) imply that plays an insignificant role, and is reduced. where is a parameter representing the initial average learning rate, is a variable affected by the number of preceding defined in [START_REF] Plumbley | Do cortical maps adapt to optimize information density?[END_REF], and is the epoch factor defined in [START_REF] Nakanishi | A review and comparison of six reasoning methods[END_REF] if otherwise [START_REF] Plumbley | Do cortical maps adapt to optimize information density?[END_REF] where , and represents a link weight factor of 1 when a connection exists between and , and 0 otherwise [START_REF] Nakanishi | A review and comparison of six reasoning methods[END_REF] where and are the current and maximum number of epochs, respectively. The two terminating conditions for the algorithm are as follows:

1) maximum number of epochs reached (i.e., ); 2) when the total change in error reaches zero or falls below a low threshold (i.e., ).

IV. RECALLING PROCESS

In the recalling process shown in Fig. 9, an input is first presented to layer 1 of an encoded network. The network will propagate this input to obtain the firing rates for nodes in layer 3 (see Sections II-A-II-C). Memory recall from the hybrid associative memory is then performed and the activation levels for nodes in layer 4 are obtained. Finally, based on output functions, the estimated output is produced at layer 6 (see Sections II-D-II-F). This section focuses on the local processes of each neuron in the memory recall stage. Assuming neuron is an node in layer 3 and neuron is a node in layer 4. Neuron transmits a signal based on its firing strength defined by for otherwise [START_REF] Kandel | Principles of Neural Science[END_REF] where is the output of , is a propagation factor to control network stability, and and are predefined positive and negative thresholds. Neuron then receives these output signals and updates its activation level based on [START_REF] Junbo | A learning rule for fuzzy associative memories[END_REF] where is the resultant weight between and , and is the number of nodes connected to .

V. EXPERIMENTS AND MULTIVARIATE ANALYSIS

Seven experiments were conducted to perform multivariate analysis and evaluate the performance of nonlinear estimation. The experiments performed were: 1) the exclusive-OR (XOR) problem, 2) the two-spiral problem, 3) iris plant classification, 4) Nakanishi's nonlinear estimation tasks (three experiments), and 5) highway density prediction and analysis.

A. The Exclusive-OR Problem

The exclusive-OR (XOR) problem is a simple nonlinear classification problem with nonmutually orthogonal inputs. This experiment's objective is to observe the behavior of the supervised error reduction phase (Section III-C).

The initial network shown in Fig. 10(a) was designed with two s, each partitioned using two Gaussian membership functions, and an with discrete partitioning using two rectangular membership functions. The supervised Hebbian learning phase and the supervised error-reduction phase were then performed. Based on the modulatory weights between the three input conjuncted maps and consequence node as shown in Fig. 11, the modulatory weights stabilized at for after six epochs, implying that only , representing conjunctions of and , influenced the output. and do not influence the output. An identical behavior was observed between and s. After learning, the network discarded and . In the encoded network shown in Fig. 10(b), four connections remain between and . These four connections represent four equally weighted fuzzy rules corresponding to the expected behavior of XOR logic as follows:

1) if is "low" and is "low," then is "low"; 2) if is "low" and is "high," then is "high"; 3) if is "high" and is "low," then is "high"; 4) if is "high" and is "high," then is "low." The XOR problem was correctly classified (Fig. 12) mainly due the existence of conjunctions formed in . Without , the classification output obtained for all input combinations would be ambiguous (i.e.,

). This shows the importance of layer 3 for removing crosstalk in nonmutually orthogonal input patterns.

B. The Two-Spiral Problem

The classical two-spiral problem [START_REF] Lang | Learning to tell two spirals apart[END_REF] is another classical nonlinear classification problem that involves the nonlinear classification of two intertwined spirals, each being labeled as a separate class. As shown in Fig. 13 where , and for the training set or for the test set. A configuration similar to that of the XOR problem (described in Section V-A) was used. However, as opposed to the XOR data, the data for the two-spiral problem higher acuity to model due to lower class separability. The number of Gaussian membership functions needed would thus be bigger for this data set.

It was determined empirically that the most suitable numbers of Gaussian membership functions were 12 and 16 for input maps and , which fuzzify the inputs and , respectively. After encoding, the modulatory weights were discovered to be for and . Thus, similarly to the XOR experiment, the output was only dependent on the conjunction of inputs. Before the encoding process, there were 660 connections between layers 3 and 4 in the network. After training, 137 positively weighted fuzzy IF-THEN rules were retained. This dramatic increase in the number of rules discovered, as compared to the four discovered for the XOR experiment, was due to the differences in the differences in separability of the data.

FASCOM was able to classify the two spirals across the 2-D space (Fig. 14) and is comparable to the other state-of-the-art architectures benchmarked against in Table IV.

C. Iris Plant Classification

The classification of iris plant uses a popular data set [START_REF] Fisher | The use of multiple measurements in taxonomic problems[END_REF] consisting of four input attributes (sepal width and length, and petal width and length) measured off iris plants to determine their subspecies (Iris setosa, Iris versicolor, and Iris virginica). The data set consists of 150 samples, with 50 samples per iris subspecies. More detailed properties and analysis of this data set were recently described in [START_REF] Singh | DCT-Yager FNN: A novel Yagerbased fuzzy neural network with the discrete clustering technique[END_REF]. The FASCOM network was initialized with the four input maps ( to ) representing sepal width, sepal length, petal width, and petal length, respectively. They contain four, four, eleven, and eleven Gaussian membership functions, respectively. The output map was partitioned into three discrete sets, with each set representing an iris plant subspecies (setosa, versicolor, and virginica).

In our experiments, we initialized the data set into two different configurations. Configuration A used 100 randomly selected samples for training and the other 50 for testing. Configuration B consisted of 50 randomly selected training samples, while the remaining 100 samples formed the testing set. Six different prediction runs were performed on different randomized instances for the two configurations, with the classification results averaged across all six runs and presented in Table V. As concluded from the results listed in Table V, FASCOM managed to outperform all other benchmarked architectures for both configurations of this classification experiment.

The total number of antecedent-consequent connections formed by the network was 5808. Subsequently, after training and pruning off the negatively and zero-weighted connections, the number of resultant fuzzy

D. Nakanishi's Nonlinear Estimation Tasks

In the previous two experiments (Sections V-A and V-B), the data contain no noise because our objective was to observe the behavior and outcome of learning. However, noise is present in the next three experiments that use the Nakanishi data set [START_REF] Nakanishi | A review and comparison of six reasoning methods[END_REF]. The data set consists of three examples of real-world nonlinear estimation tasks, namely: 1) a nonlinear system, 2) the human operation of a chemical plant, and 3) the daily stock price data of a stock in a stock market.

The objectives of these experiments is to perform multivariate analysis, comparing the similarity and differences of this analysis with other existing methods, as well as to benchmark data estimation on noisy data against other state-of-the-art architectures. Two measures were used for evaluation of data estimation, namely, the mean squared error (MSE) and the Pearson productmoment correlation coefficient (R). The nonlinear estimation results were evaluated and compared against results obtained from Hebb-RR [START_REF] Liu | A novel generic Hebbian orderingbased fuzzy rule base reduction approach to Mamdani neuro-fuzzy system[END_REF], SVM [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF], RSPOP-CRI [START_REF] Ang | RSPOP: Rough set-based pseudo outerproduct fuzzy rule identification algorithm[END_REF], DENFIS [START_REF] Kasabov | DENFIS: Dynamic evolving neural-fuzzy inference system and its application for time-series prediction[END_REF], POP-CRI [START_REF] Ang | POPFNN-CRI(S): Pseudo outer product-based fuzzy neural network using the compositional rule of inference and singleton fuzzifier[END_REF], ANFIS [START_REF] Jang | ANFIS: Adaptive-network-based fuzzy inference system[END_REF], [START_REF] Chiu | Fuzzy model identification based on cluster estimation[END_REF], Mamdani [START_REF] Mamdani | An experiment in linguistic synthesis with a fuzzy logic controller[END_REF], [START_REF] Mamdani | Applications of fuzzy algorithm for control of a simple dynamic plant[END_REF], EFuNN [START_REF] Kasabov | Evolving fuzzy neural networks for supervised/unsupervised online[END_REF], Turksen [START_REF] Turksen | Interval valued fuzzy sets based on normal forms[END_REF], and Sugeno (P-G) [START_REF] Sugeno | Structure identification of fuzzy model[END_REF]. As shown in Table VI, FASCOM obtained the best results for all three nonlinear estimation tasks.

For the three tasks, the number of fuzzy IF-THEN rules discovered after retaining only the positive connections were 149, 161, and 678, respectively.

Task A: A Nonlinear System: The aim of this task is to model a nonlinear system given by [START_REF] Liu | The fuzzy associative memory of max-min fuzzy neural networks with threshold[END_REF] Here, four input dimensions are used to predict the value of the 1-D output.

The supervised error reduction phase discovered the influence of individual input conjuncted maps on the output , across output space. As an example, the varying influence of two input conjuncted maps representing and across output space is shown in Fig. 15. It can be observed that significantly influenced the output across the entire output feature space, especially for outputs of around 2.0, 2.5, and 4.75. On the other hand, only influences the low outputs or below 1.5 and its effects are reduced for the other portions of output space.

For a more generalized description, the mean of the influences is computed across the output feature space and illustrated in Fig. 16. s representing , , , and were generally enhanced across the output space. On the other hand, it is clear that those s representing , , and exerted no influence on the outcome of and could be discarded. The effects of all other s were reduced. This generalized outcome is similar to the feature selection method in [START_REF] Nakanishi | A review and comparison of six reasoning methods[END_REF], which discarded the inputs and , as well as that of [START_REF] Ang | RSPOP: Rough set-based pseudo outerproduct fuzzy rule identification algorithm[END_REF], which discarded input and partially discarded input . However, FASCOM was more specific and descriptive in that it was able to provide information of how the influences of inputs and conjunctions of inputs vary across output space. From another perspective, FASCOM identifies the significance of various inputs for a particular portion of the output space.

2) Task B: Human Operation of a Chemical Plant: This task involves the human operation of a chemical plant, whereby there are five input dimensions representing monomer concentration , charge of monomer concentration , monomer flow rate , and local temperatures inside the plant , as well as one output dimension representing the set point for monomer flow rate . The supervised error reduction phase identified influences of each input conjuncted map across the output space, from which the s representing , , , , , , and were deemed noninfluential to output prediction and were discarded. As a comparison, the feature selection method in [START_REF] Nakanishi | A review and comparison of six reasoning methods[END_REF] discarded inputs , , and , and [START_REF] Ang | RSPOP: Rough set-based pseudo outerproduct fuzzy rule identification algorithm[END_REF] discarded inputs , , and . The that was most enhanced had a modulatory weight of 3.361, and represented the conjuncted input

. From this, we deduce that the set point for monomer flow rate depends mainly on the combination of monomer flow rate and monomer concentration (i.e.,

). Additionally, as seen from Table VI, FASCOM significantly outperformed all other architectures for this nonlinear estimation task, with a reduction of MSE by 66.3% over the state-ofthe-art. The precision of FASCOM's output estimation is displayed in Fig. 17.

3) Task C: Daily Stock Price Data of a Stock in a Stock Market: In this task, ten inputs are used to predict the price of a stock . The inputs represent the past and present moving averages over a middle period , past and present separation ratios with respect to moving average over a middle period , present change of moving over a short period , past and present changes of price , past and present separation ratios with respect to moving average over a short period , and present change of moving average over a long period . As mentioned in Section II-C, high-dimensionality data induces high spatial complexity. Exhaustive conjunction on this data set with ten input dimensions would result in 1023 s. This increases training time to impractical levels. To handle this problem, a selective approach was performed to reduce the number of s generated by limiting the number of preceding s that would combine to form each to two. This results in only 55 ICMs generated and greatly reduces both the spatial resources required and the training time back to practical levels.

For this task, the supervised error reduction phase discovered that inputs corresponding to , , , , , , and had extremely low influences and their effects were significantly reduced.

s representing and were discovered to be the most influential ones. This means that the present separation ratio with respect to moving average over a middle period , present separation ratio with respect to moving average over a short period , and present change of price are the main indicators for predicting daily stock prices. As a comparison, Nakanishi et al. [START_REF] Nakanishi | A review and comparison of six reasoning methods[END_REF] discarded inputs , , , , , , and , and [START_REF] Ang | RSPOP: Rough set-based pseudo outerproduct fuzzy rule identification algorithm[END_REF] discarded , , , , and , and partially discarded , , , and . Interestingly, although there are some differences in the outcome, was not discarded for all three methods. This means that all three methods considered to be of importance for output prediction. We further found that when comparing the past and present values of the same indicators, (i.e., , , , ), with the exception of the moving average over a middle period indicator , the present indicators generally have greater impact as compared to the past ones.

The good prediction results for this task (Table VI) suggest that similarly to [START_REF] Ang | Stock trading using RSPOP: A novel rough set based neuro-fuzzy approach[END_REF] and [START_REF] Tan | Maximizing winning trades using a rough set based other-product (RSPOP) fuzzy neural network intelligent stock trading system[END_REF], FASCOM can be used to perform prediction of the stock market. In addition, because of the natural cause-and-effect interpretation of associations and the descriptive nature of input-output dimension relationships, FASCOM may be suitable for understanding the dynamics of a stock market.

4) Justification of Learning Phases: Besides multivariate analysis and nonlinear estimation, the three Nakanishi nonlinear estimation tasks were also used to analyze the effects of three different experimental setups of the learning process:

1) using all three phases in the learning process; 2) omitting unsupervised membership function initialization (phase 1); 3) omitting supervised error reduction (phase 3). For each nonlinear estimation task, the resultant MSE produced by each of the three initializations was computed and normalized with respect to the highest MSE produced. As observed from Fig. 18, for all three tasks, the comparison between the three different experimental setups shows that the inclusion of both phases 1 and 3 produced an MSE that is significantly lower than when either one is omitted. This shows that both phases 1 and 3 are crucial in reducing the MSE of the output estimation and justifies the existence of both phases within the learning process.

E. Highway Traffic Density Prediction and Analysis

The raw traffic data [START_REF] Tan | Feasibility of predicting congestion states with neural networks[END_REF] were collected at site 29 located at Exit 5 along the east bound direction of the Pan Island Expressway (PIE) in Singapore, using loop detectors embedded beneath the road surface (Fig. 19). Data spanning a period of six days from September 5 to 10, 1996, for three straight lanes, were considered for this experiment. The purpose of this experiment is to model the traffic flow trend at the site, and subsequently produce predictions of the traffic density of each lane at time , where 5, 15, 30, 45, 60 min. From this, we can analyze the performance of data prediction and perform multivariate analysis on real-world noisy data.

The data set has four input dimensions, with one representing the normalized time, and three others representing the traffic density of each of the lanes.

The raw data records the traffic density at 1372 instants with intervals of 5 min between successive instants. After processing the data for forward projections of 5, 15, 30, 45, 60 min, there are 1371, 1369, 1366, 1363, and 1360 paired samples for each set. To evaluate the performance of predicted traffic density, three cross-validation groups (CV1, CV2, and CV3) of training and test sets were formed. Each cross-validation group has 550 training samples each, and 821, 819, 816, 813, and 810 test samples, respectively.

The MSE was computed and averaged across all prediction runs, resulting in "Avg MSE." Also, the "Var" indicator reflecting the consistency of predictions over different time intervals across the three lanes was computed as the change in the mean of Pearson product-moment correlation coefficient from 5 min to 60 min expressed as a percentage of the former. This was then averaged across all three lanes to produce "Avg Var." From the example in Fig. 20, we observe that the prediction accuracy decreases as the time interval increases. The results of traffic density prediction were compared to Hebb-RR [START_REF] Liu | A novel generic Hebbian orderingbased fuzzy rule base reduction approach to Mamdani neuro-fuzzy system[END_REF], SVM [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF], RSPOP [START_REF] Ang | RSPOP: Rough set-based pseudo outerproduct fuzzy rule identification algorithm[END_REF], POP-CRI [START_REF] Ang | POPFNN-CRI(S): Pseudo outer product-based fuzzy neural network using the compositional rule of inference and singleton fuzzifier[END_REF], DENFIS [START_REF] Kasabov | DENFIS: Dynamic evolving neural-fuzzy inference system and its application for time-series prediction[END_REF], GeSoFNN [START_REF] Tung | GenSoFNN: A generic self-organizing fuzzy neural network[END_REF], and EFuNN [START_REF] Kasabov | Evolving fuzzy neural networks for supervised/unsupervised online[END_REF]. FASCOM significantly outperformed all other architectures based on the results shown in Fig. 21, with the lowest "Avg MSE" (0.098) relative to "Avg Var" (19.0%) as compared to other architectures. The results indicate that the output prediction by FASCOM is both highly accurate and consistent over different time intervals. This is desirable.

By analyzing the influence of the input conjuncted maps across the output, it could be deduced that the prediction of traffic densities for a particular lane is mostly dependent on normalized time combined with the current density of that lane, as well as normalized time combined with the density of an adjacent lane. The influence of the density of lanes decreases with distance. Interestingly, the above factors are more distinctive for outputs predicting high traffic density. However, performance comes with a price. On the average, FASCOM extracted 42.3 fuzzy IF-THEN rules from the data. The average number of rules the other architecture extracted was: 8.1 by Hebb-RR, 14.4 by RSPOP, 40.0 by POP-CRI, 9.7 by DENFIS, 50.0 by GeSoFNN, and 234.5 by EFunN. It should also be noted that FASCOM is spatially more complex than most of the other architectures because of the deliberate use of exhaustive conjunctions and associations for information representation and input-output relation. Finally, FASCOM's learning phase can only be performed in an offline batch mode manner, while some other methods, such as EFuNN [START_REF] Kasabov | Evolving fuzzy neural networks for supervised/unsupervised online[END_REF] and DENFIS [START_REF] Kasabov | DENFIS: Dynamic evolving neural-fuzzy inference system and its application for time-series prediction[END_REF], can perform learning in an online and incremental manner. These methods have the added advantage of being able to update its model and evolve the fuzzy rules with every new training instance. However, offline methods remain suitable for many real-world scenarios.

VI. CONCLUSION

This paper proposed the FASCOM network. It is encoded through a offline batch mode learning process consisting of three phases-one unsupervised phase followed by two supervised phases. Based on biologically inspired uniform information density, the unsupervised initialization of membership functions in the first phase led to more efficient representation and transmission of information. The supervised phase of Hebbian learning was responsible for identifying weights for fuzzy rules. The third supervised phase of error reduction not only helped fine-tune the network, but also it provided insights of how input dimensions influenced the various portions of the output. In layers 3 and 4, the representation of multiple antecedents and consequences resulted in the reduction in crosstalk for nonmutually orthogonal input patterns.

Experiments performed demonstrated that the above technical innovations led to significant improvement in the results in terms of accuracy of nonlinear estimation, despite the problem of high space complexity. The positive experimental results on real-world data (Sections V-D and V-E) lead us to believe that FASCOM could be applied to a wide variety of real-world problems for either classification (e.g., medical prognosis), regression (e.g., time-series forecasting), or multivariate analysis (e.g., financial data analysis). Future work would also involve addressing limitation of the high space complexity experienced in layer 3 and layer 4. He joined Institute for Infocomm Research (I2R) and its predecessors, Singapore in October 1990. He is currently the Department Head of the Computer Vision & Image Understanding Department, with staff strength of 50 research scientists and engineers, at I2R, Singapore. He is also the Co-Director of Image Perception, Access and Language (IPAL), a French-Singapore Joint Lab (UMI 2955, January 2007-December 2010) and was bestowed the title of "Chevallet dans l'ordre des Palmes Academiques" by the French Government in 2008. He has conducted research in connectionist expert systems, neural-fuzzy systems, handwriting recognition, multiagent systems, and content-based retrieval. He has ten patents (awarded and pending) and published more than 120 refereed international journal and conference papers in his research areas.

Hanlin

Fig. 1 .

 1 Fig. 1. Structure of the FASCOM network.

Fig. 2 .

 2 Fig. 2. Computing the fuzzy subsethood measure from a LoG(1) smoothed input x .

Fig. 3 .

 3 Fig. 3. Example of projecting layer 2 to layer 3.

Fig. 4 .

 4 Fig. 4. Learning process (highlighted in gray).

  )] can be achieved by the following four steps: 1) histogram construction based on data points of training data [Fig. 5(a)]; 2) Gaussian smoothing of histogram [Fig. 5(a)]; 3) histogram equalization to modify the scale of the domain [Fig. 5(b)]; 4) using the equalized scale in step 3 to map to the initial neuron allocation [Fig. 5(c)]

Learning Phase 1 )

 1 Fundamental Hebbian Rule: Modified during learning by adopting the Hebbian rule [49], the synaptic weight stores memory between two neurons and . Assuming and are nodes in an and , respectively, the change in synaptic weight can be computed using (23) based on their firing strengths and if and otherwise (

Fig. 5 .

 5 Fig. 5. Steps to achieve membership function allocation based on information density. (a) Steps 1 and 2: insert and smooth data. (b) Step 3: histogram equalization. (c) Step 4: mapping of scale. (d) Final membership function allocation.

Fig. 6 .

 6 Fig. 6. Effects of learning mechanisms on learning. (a) Effects of long-term facilitation. (b) Combined effects of synaptic plasticity and forgetting.

Fig. 7 .

 7 Fig. 7. Modulatory weights form an asymmetrical (ICM to C ) relationship between input and output conjuncted maps.

Fig. 8 .

 8 Fig. 8. Supervised error reduction algorithm.

Fig. 9 .

 9 Fig. 9. Recalling process (highlighted in gray).

Fig. 10 .

 10 Fig. 10. Encoding transformation for the XOR problem.

Fig. 11 .Fig. 12 .

 1112 Fig. 11. Changes in modulatory weights between input conjuncted maps and consequence node C .

Fig. 13 .Fig. 14 .

 1314 Fig. 13. Two-spiral problem.

Fig. 15 .

 15 Fig. 15. Two examples showing the varying influence of an input across the output space.

Fig. 16 .

 16 Fig. 16. Mean influence of each input computed across the output space.

Fig. 17 .

 17 Fig. 17. Output estimation for task B has high precision.

Fig. 18 .

 18 Fig. 18. Comparison between the three different experimental setups for the three Nakanishi estimation tasks.

Fig. 19 .

 19 Fig. 19. Photograph of site 29 where the traffic data was collected.

Fig. 20 .

 20 Fig. 20. Traffic density prediction of lane 1 using CV1. (a) Prediction at = 5 min. (b) Prediction at = 15 min. (c) Prediction at = 30 min. (d) Prediction at = 45 min. (e) Prediction at = 60 min.

Fig. 21 .

 21 Fig. 21. FASCOM outperforms other benchmarked architectures for highway traffic density prediction.

  Goh received the B.Eng. degree in computer engineering and the M.Sc. degree in bioinformatics from Nanyang Technological University, Singapore, in 2007 and 2009, respectively. He is currently a Research Officer with the Computer Vision and Image Understanding Department, Institute for Infocomm Research, Singapore, and a member of Image Perception, Access and Language (IPAL), a French-Singaporean joint laboratory. His research interests include fuzzy neural networks, machine learning, computer vision, and image processing.Joo-Hwee Lim (M'07) received the B.Sc. (Hons I) and M.Sc. (by research) degrees in computer science from the National University of Singapore, Singapore, in 1989 and 1991, respectively, and the Ph.D. degree in computer science and engineering from the University of New South Wales, Sydney, Australia, in 2004.

TABLE I DESCRIPTION

 I AND QUANTITY OF NODES AND FEATURE MAPS IN THE NETWORK

	for a	association form a correlation matrix
	defined in
	. . .	. . .	. . .	. . .	. . .
	. . .	. . .	. . .	. . .	. . .

TABLE III LIST

 III OF SYMBOLS USED IN SUPERVISED ERROR REDUCTION PHASE

TABLE VI BENCHMARKING

 VI RESULTS ON NONLINEAR ESTIMATION FOR THE NAKANISHI ESTIMATION TASKSIF-THEN rules obtained was 257.83 when averaged across the six runs.

Chai Quek (M'96) received the B.Sc. degree in electrical and electronics engineering and the Ph.D. degree in intelligent control from Heriot-Watt University, Edinburgh, Scotland, in 1986 and 1990, respectively.

He is an Associate Professor and a member of the Centre for Computational Intelligence, formerly the Intelligent Systems Laboratory, School of Computer Engineering, Nanyang Technological University, Singapore. His research interests include intelligent control, neurocognitive architectures, artificial intel-ligence (AI) in education, neural networks, fuzzy systems, fuzzy-rule-based systems, and genetic algorithms and brain-inspired neurocognitive applications in computational finance and biomedical engineering.

Dr. Quek is a member of the IEEE Technical Committee on Computational Finance.


Sheng Gao
email: gaosheng@i2r.a-star.edu.sg

Joo-Hwee Lim

SELECTING REPRESENTATIVE AND DISTINCTIVE DESCRIPTORS FOR EFFICIENT LANDMARK RECOGNITION

Keywords: scene recognition, Markov chain, classification accuracy, PageRank, sample selection

To have a robust and informative image content representation for image categorization, we often need to extract as many as possible visual features at various locations, scales and orientations. Thus it is not surprised that an image has a few hundreds or even thousands of visual descriptors. This raises huge cost of computation and memory. To eliminate the problem, we can only select the most representative and distinctive descriptors and discard the other non-informative features when training the image category models. This paper will present a Markov chain based algorithm to learn a measure of the descriptor importance in order to weigh the degree of representativeness and distinctiveness. From the measures the descriptor selection algorithm is derived. The presented approach starts from constructing a graph with each node being a descriptor to characterize the pair-wise descriptor similarity and then the PageRank algorithm is exploited to estimate the stationary distribution of the graph whose values are the indicator of the descriptor importance. We evaluate the proposed approach on the STOIC-101 landmark dataset. Our experiments demonstrate the Markov chain based descriptor selection can select the most informative descriptors to distinguish the landmarks. Even with the large reduction of the size of descriptors, the classification accuracy is still competitive or overcomes compared with the system without any descriptor selection.

INTRODUCTION

In image categorization such as generic object recognition, scene recognition, etc., we often need to extract sufficient visual descriptors for characterizing the image category. To retain as much as possible information and to have a robust representation for scales, rotation, transformation, clutter, etc., the descriptors are always extracted from the various locations, scales and orientations. It is typical to find an image is characterized by a few hundreds or even thousands of visual features. For example, in the paper the average size of SIFT descriptors [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF] is ~655 for a 320x240 image. Therefore, for thousands of images in the training set, there will be millions of descriptors. The large size of features not only raises huge cost of computation and memory but also may cause the learned image category models bias due to the dependency among the descriptors.

In most of learning algorithms, it is often assumed that the samples (image or descriptor) are drawn independently and uniformly from the sample space. For instance, in the bag-of-words based image representation, when we build a visual codebook using the k-means clustering on the pooled descriptors, we assume each descriptor is independent and it is uniformly sampled, i.e. each descriptor having equal contribution in the objective function of k-means clustering. However, the assumption has the defects in practice. For example, in [START_REF] Jurie | Creating efficient codebooks for visual recognition[END_REF] the authors discuss that the cluster centers of k-means are drawn irresistibly towards denser regions of the sample distribution and thus the clusters tend to be tightly clustered near the dense regions and sparsely spread in the sparse. The method in [START_REF] Jurie | Creating efficient codebooks for visual recognition[END_REF] addresses the defect using an undersampling framework combined with on-line clustering and mean-shift. Besides of the undersampling method, the sample bias can be tackled using the unlabeled data (e.g. [START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF]) to correct the sample distribution of the training set. In the approach, the unlabeled data is used to weigh the samples in the training set so that the sample distribution of the training set and the test set will match. The approach breaks the uniform assumption about the training samples.

In the paper, we present a Markov chain based algorithm to learn a measure of the descriptor importance in order to weigh the degree of representativeness and distinctiveness and then apply the measure to choose the most representative and distinctive descriptors for landmark recognition. Firstly, a graph is constructed based on the pair-wise similarity analysis among the descriptors. The graph node is a descriptor while the node links to the other are sought using the k-nearest neighbor method. After the graph is ready, secondly, we exploit the PageRank algorithm to estimate a stationary distribution of the graph [START_REF] Brin | The anatomy of a large-scale hypertextual web search engine[END_REF] and the PageRank scores are the indicator of the descriptor importance in the dataset. Based on the descriptor importance distribution, we can derive the informative measure for weighing the degree of representativeness and distinctiveness of the descriptors for characterizing the landmarks. Two schemes are developed in the paper for descriptor selection (See Section 2): one is to measure the degree only using the category-specific descriptor importance and another is to measure the degree combing the category-specific importance, which is estimated from the descriptors from one category, and category-free importance, which is estimated from the descriptors from all categories. The selected descriptors can be used directly for classification, for instance, using the k-NN classifier, or be used as a pre-stage for some complex training algorithms (See Section 3).

The proposed method is a novel application of Markov chain theory to weigh the importance of descriptors based on the pair-wise dependency analysis of descriptors and the link structure analysis of the graph. The estimated distribution of the degree of representativeness and distinctiveness of descriptors has many applications. For example, we can do the sample selection based on the measure. But this selection is one time rather than multiple iterative steps like the undersampling in [START_REF] Jurie | Creating efficient codebooks for visual recognition[END_REF]. It is thus cheap. The learned distribution of descriptors can be integrated into training the classifier like in [START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF]. But it is superior to [START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF] in that the weights are estimated only from the training set without the need of the extra unlabelled dataset, which is often the test set.

It is well-known that the Markov chain is applied for measuring the popularity of web documents, e.g. PageRank in Google. We find that there are many research works on extending PageRank to other applications such as feature selection [START_REF] Ienco | Using PageRank in feature selection[END_REF], word sense disambiguation [START_REF] Mihalcea | Unsupervised large-vocabulary word sense disambiguation with graph-based algorithms for sequence data labeling[END_REF], and visual image search [START_REF] Jing | VisualRank: applying PageRank to large-scale image search[END_REF], etc. In these applications, the link structures are not natural available, which is quite different from the web documents where the links are ready. So the first step is to generate the link structure by analyzing the similarity among the elements. After that the similar operation is done as scoring and ranking the web pages.

In the paper, we only evaluate the descriptor selection for landmark recognition on the STOIC-101 dataset, which consists of 101 attractive landmarks in Singapore and 3,539 images totally.

The paper is organized as follows. In Section 2, we will present the proposed Markov chain based descriptor selection algorithm. In Section 3, we report the experimental results on multiclass landmark recognition. Finally, we conclude our findings in Section 4.

MARKOVE CHAIN BASED VISUAL DESCRIPTOR SELECTION

In millions of descriptors extracted from the training images, it is expected that high dependency among the descriptors should exist. It is not effective and efficient to use all of them to train the image category models. In fact, it is not necessary. We can only select the most representative and distinctive visual descriptors in training the image category models. The question is how to measure the quantity of representativeness and distinctiveness. In the section, we introduce the Markov chain based algorithm for measuring the importance of descriptors in characterizing the image category content. Then the descriptor selection method can be derived from the measures.

Weighing Importance of Visual Descriptors

Similar to PageRank [START_REF] Brin | The anatomy of a large-scale hypertextual web search engine[END_REF], where the random surfer visiting the webpage document is modeled by Markov chain, here we use the Markov chain to model the visual descriptor selection. In the chain, each state is a visual descriptor and its links are sought by comparing the pair-wise descriptor similarity and being determined based on the K-nearest neighbor method. If we assume the set has N descriptors and adjacency matrix,

( ) ( ) ( ) 1 2 , , , N Q q x q x q x = ⎡ ⎤ ⎣ ⎦ L . ( ) q ⋅ is
a probability of a descriptor being visited by the sampler. The distribution is the eigenvector of the following equation as,

T Q P Q = (1)
In practice, a residual probability, d, is often added into Eq.( 1). So the stationary distribution is the solution of modified Eq. (1) as, ( )

1 1 T d Q I d P Q N - = ⋅ + -⋅ (2),
where I is N-dimensional vector filling with 1 in all elements. Eq. ( 2) is a penalized version of Eq. (1), i.e. a uniform prior probability is added in each node. Its solution is found using the iterative algorithm used in PageRank [START_REF] Brin | The anatomy of a large-scale hypertextual web search engine[END_REF]. The stationary distribution is a good measure of descriptor's importance in the database. The descriptor's importance measures how many descriptors are pointed to it and are similar to it. If the descriptor has a lot of links, it implies that the descriptor is representative and it encodes richer information than the other descriptors with the lower importance. Thus, rather than using all descriptors, we can summarize the content of the dataset by only retaining the most informative descriptors whose importance values are higher than the threshold.

Measuring Representativeness and Distinctiveness

As the estimation of descriptor importance in a dataset is solved, we can exploit the measure in order to derive a quality to weigh the degree of representativeness and distinctiveness of a visual descriptor for charactering the content of image category. Based on the measure, we can tell which descriptor is representative for a category and which one is distinctive for discriminating one category from another.

We derive the idea from the weighing of the word terms in the text document using the tf-idf method, i.e. term frequency -inverse term frequency, for information retrieval [15]. Term frequency is the probability of a term occurred in a document while inverse term frequency is a measure of the importance of the term across the documents, which is calculated from the number of documents containing the term and the total document number in the set. The former is a measure of representativeness of the term and the latter is a measure of distinctiveness or discrimination of the term. The higher the term frequency is in a document, the more representative the term is for the document. Similarly, the term which has the higher inverse term frequency is more distinctive and holds much more capability of discrimination to distinguish one document from another.

In the context of image categorization, the document may be an image or an image category which have a lot of visual descriptors to characterize their content. In the paper, we would like to measure the descriptors in the category, i.e. a category as a document. The term refers to the visual descriptor. We first run PageRank on the pooled visual descriptors from one category, saying the i-th category, and obtains the category-specific importance vector,

( ) ( ) ( ) 1 2
, , ,

N i i i i i i i i Q q x q x q x ⎡ ⎤ = ⎣ ⎦ L , where j i
x is the j-th descriptor in the i-th category and i N is the size of descriptors. ( ) j i i q x is the popularity of the term j i x in the i-th category and will be treated as the term frequency. After the term frequency is obtained, we then run PageRank on the pooled descriptors from all categories to get the category-free importance vector, ( ) ( ) ( ) ( ) ( ) ( )

1 1 2 1 1 2 2 1 1
, , , , , , , , ,

N N N c C C Q q x q x q x q x q x q x ⎡ ⎤ = ⎣ ⎦ L L L L . .
Each element is the term frequency of the descriptor in the whole dataset. Its inverse simulates the inverse term frequency. Similar to tf-idf, we derive a measure from the categoryspecific importance and the category-free importance as in Eq. (3) ( ) ( )

i i i j i j j w q x q x = (3)
The value of j i w weighs the degree of representativeness and distinctiveness for the descriptor, j i x .

Visual Descriptor Selection

As the measure of representativeness and distinctiveness of descriptors is ready, the descriptor selection is cheap. For example, we can simply choose top-K descriptors, which have highest values in Eq. ( 3), for each category to summarize the category content. Due to the descriptor selection, the size of visual descriptors is decreased and huge computation cost is reduced. It speeds up the model training and it allows us to evaluate more complex and precise models for image categorization classification. More interestingly and importantly, sample selection will not worsen the classification accuracy, even with the half reduction of the training set. In contrast, we observe the accuracy is improved. It will be shown in Section 3.

EXPERIMENTS

In the section, we will evaluate the effect on the performance of the proposed visual descriptor selection algorithm in the image landmark recognition. The landmark dataset, STOIC-101, includes 101 landmarks in Singapore and has totally 3,539 images with the size of 240x320. The landmark images are captured in large variations of scale, clutter, and lighting condition. Some details can refer to [START_REF] Lim | Scene recognition with camera phones for tourist information access[END_REF][START_REF] Li | Outdoor place recognition using compact local descriptors and multiple queries with user verification[END_REF]. The dataset is further split into two parts: the sub-set A, i.e. SA, has 3,090 images and the sub-set B, i.e. SB, has 449 images. 128-dimensional SIFT descriptors are extracted using the detector developed by Low [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF]. Then the eigenspace is estimated using the principle component analysis (PCA) on the SA set, which has 2,023,507 visual descriptors, and 56 eigenvectors are retained. So the 128-dimensional SIFT descriptor is mapped into the 56-dimensional vector.

Effect of Size of Selected Descriptors

We first study the effect of the size of selected descriptors on the classification accuracy. We use SB as the training set and SA as the test set. Rather than recognizing 101 landmarks, we only select 5 landmarks due to the memory and computation cost. Thus the small training set (SB_T) and test set (SA_E) are constructed using corresponding images in the 5 landmarks. It results in the SB_T having 23 images with 15,120 descriptors and the SA_E having 152 images with 99,248 descriptors. We investigate how the classification accuracy will be affected when the descriptors in the training set is decreased. The k-NN classifier is adopted for simplicity and we make the decision based on the voting.

Two sample selection schemes are discussed. The first is to select informative descriptors for a landmark only based the category-specific importance, and the second is to select descriptors based on the combination of the categoryspecific importance (See Eq. ( 2)) and the category-free importance (See Eq. ( 3)). Then the selected categoryspecific descriptors are used as the template exemplars in the k-NN classification. The baseline is the system without descriptor selection. The number of selected descriptors per landmark is 500 (2,500), 1,000 (5,000), and 2,000 [START_REF] Brin | The anatomy of a large-scale hypertextual web search engine[END_REF]315), respectively. The number in the parentheses is the total size of selected descriptors for 5 landmarks (Note: some categories may have descriptors less than the threshold). We draw their classification accuracies in Fig. 1 (Blue bar: first scheme; Red bar: second scheme, Yellow bar: baseline).

From the figure, we can conclude that: 1) combing the category-free importance with the category-specific importance obviously improves classification accuracy; The improvement is significant when the sample size is smaller, e.g. 34.7% relative accuracy improvement achieved for 500 descriptors per landmark and 10.4% for 1,000 descriptors per landmark; 2) As the number of selected descriptors is increasing, the classification accuracy is improving; 3) both schemes works better than the baseline system without sample selection; even with 500 descriptors per landmark, we can achieve 63.82%, which is much better than the baseline system having only 59.21% ; 4) the recognition speed is significantly increased due to the large reduction of sample size, e.g. ~1s for 500 descriptors vs. ~6s for the baseline. Categoryspecific Categoryfree Baseline

Fig. 1 Classification accuracy for 2 selection schemes in various size of selected descriptors and the baseline 3.2. Large-scale Evaluation Now we will report the evaluation results of the effect on the performance of the selected descriptor size in the whole 101 landmarks. The training set is the SA having 3,090 images and the test set is the SB having 449 images. Totally there are 2,023,507 descriptors in the training set, ~20,034 per landmark. The bag-of-words method is used to derive one high-dimensional vector per image. Then we apply the MC MFoM learning algorithm to train a linear classifier for maximizing macro-averaging F1 measure [START_REF] Gao | A MFoM learning approach to robust multiclass multi-label text categorization[END_REF]. We estimate the landmark-specific visual codebook having 32 keywords per landmark and then the 101 landmark-specific codebooks are pooled into a universal codebook having 3,232 visual keywords. The universal codebook is used to quantize the descriptors which results in a normalized 3,232-dimensional histogram. The MC MFoM based classifier is trained on the bag-of-words image representation.

Considering about 2 millions of descriptors in the training set, calculating the category-free importance of descriptors will consume huge memory and computation cost. In the paper, we only report the results on the first scheme (See sub-section 3.1) of descriptor selection. The size of selected descriptors is 1,000, 5,000 and 10,000 per landmark based on the category-specific descriptor importance. Then the visual keywords are learned from the selected descriptors using k-means clustering. The baseline system is trained without any descriptor selection. Table 1 summarizes their classification accuracy. 1 in the largescale evaluation. In case of only keeping about half size of descriptors, i.e. 10,000, the classification accuracy still overcomes the baseline. The accuracy is improving with the increasing size of selected samples.

CONCLUSION

In the paper we presented a Markov chain based algorithm to learn the importance of descriptors by analyzing the pairwise similarity and building a graph to characterize the dependency among the descriptors. The descriptor importance is estimated using the PageRank algorithm. We study the application of category-specific importance and category-free importance in measuring the degree of representativeness and distinctiveness of descriptor and the application in descriptor selection for landmark recognition. Our experiments on the small and large scale sets demonstrate that the proposed approach not only reduces the size of descriptors in the training set, but also improves the classification accuracy due to informative features being selected while the noisy features being compressed.

A

  , to characterize the structure of descriptor relations. The jump from one state to another is controlled by the transition probability matrix, x N N P , where the ( ) i,j -th element measuring the probability jumping from the i-th state to the j-th state. The stationary distribution of the Markov chain is the probability distribution,

Table 1

 1

		Classification accuracy on 101 landmarks
		1,000	5,000	10,000	Baseline
	Acc(%)	62.14	74.83	76.84	75.95
	Similar findings are observed from Table