Tharam S Dillon
email: tharam.dillon7@gmail.com

Yi-Ping Phoebe Chen

Elizabeth Chang
email: elizabeth.chang@unsw.edu.au

Mukesh Mohania

Conjoint Mining of Data and Content with Applications in Business, Bio-medicine, Transport Logistics and Electrical Power Systems

Introduction

Digital information within an enterprise consists of (1) structured data and (2) unstructured content. The structured data includes enterprise and business data like sales, customers, products, , accounts, inventory and enterprise assets, etc. while the content includes contracts, reports, emails, customer opinions, transcribed calls, online inquires, complements and complaints. Further, cutting edge businesses also using GPS tracking or surveillance monitors as well as sensor technologies for productivity, performance and efficiency measures, and these are provided by outsourcers etc. Similarly in the Biomedical area, resources can be structured data say in Swiss-Prot or unstructured text information in journal articles stored in content repositories such as PubMed. The structured data and the unstructured content generally reside in entirely separate repositories with the former being managed by a DBMS and the latter by a content manager frequently provided by an outsourcer or vendor [76]. This separation is undesirable since the information content of these sources is complementary. Further, each outsourcer or vendor keep the data on their own Cloud, and data are not sharable between the vendor systems, and most vendor system were not integrated with the enterprise systems, and leaves the organization to consolidate the data and information manually for data analytics. Effective knowledge and information use requires seamless access and intelligent analysis of information in its totality to allow enterprises to gain enhanced critical insights. This is becoming even more important, as the proportion of structured to unstructured information has shifted from 50-50 in the 1960s to 5-95 today [START_REF] Brodie | Computer science 2.0: A new world of data management[END_REF]. Unless we can effectively utilize the unstructured content conjointly with the structured data, we will only obtain very limited and shallow knowledge discovery from an increasingly narrow slice of information. The techniques developed in our research will then be used to address significant issues in three application areas, but potential applications with significant impact are much more extensive.

Aims and Underlying Issues

We develop a methodology and techniques for deriving deep knowledge from structured and unstructured information conjointly. This methodology would be useful in several Business Intelligence and Advanced Analytics utilizing Data and Content (AADC), information integration applications, such as managing customer attrition, targeted marketing, fraud detection and prevention, compliance, and customer relationship management as well as a number of fields which involve information of high complexity such as Bioinformatics, Transport Logistics, Business and Electrical Power Systems (EPS). The broad aims of this research are to: a) use structured data to disambiguate text segments and link them to records for AADC investigations b) use ontologies to disambiguate and annotate content segments to permit AADC investigations c) develop methods of AADC for conjoint analysis of linked structured and content elements d) apply these techniques for investigation of business problems, biomedical problems, electrical power system problems and logistics and transportation problems.

To achieve the aims (a) to (c), we have a number of sub-aims that develop techniques for:

1. cleansing and filtering the noisy unstructured data with respect to structured data; 2. text annotation to enrich the unstructured data semantically; 3. fuzzy matching and searching over structured data based on annotated values for deriving the correlation between data and content; 4. discovering the linkages between data and content; 5. representations of the conjoint information that is suitable for AADC; 6. allowing new insights in the form of business intelligence and deep knowledge from the integrated data and content together.

The applications of the techniques developed (aim (d)) help to make better decisions and to better understand behaviors in the field of business, biomedicine, electrical power systems and logistics. In regards to aim (d), we study the following:

1. in business applications: customer experience and focused cross/up-selling and personalized marketing using the data and 'the voice of customer'; 2. in biomedical area: Use of structured information from Protein Databases such as Swiss-Prot and epidemiological medical databases together with information from journal articles in PubMed to determine their relationship to specific conditions and obtain diagnostic knowledge for specific diseases; 3. In transport logistics area: movement of people, goods and services tracking for productivity, security and safety; trust, reputation and quality of service (QoS); track and trace of sub-contractor's performance (against SLA); track and trace fuel and vehicle performance, track and trace carbon emission, etc.

4. in Electric Power Systems: (i) Determination of risk events to profitability related to contractual terms for supply through bi-level contracts, in conjunction with structured data such as inflows, maintenance, failure and spot prices; (ii) Refinement of contractual terms and conditions and (iii) Remedial actions to manage and mitigate potential risk conditions.

Advanced Analytics (AADC) is an automated or semi-automated process for eliciting novel embedded knowledge, patterns and associations from data and information repositories, that is useful and understandable [START_REF] Fayyad | From data mining to knowledge discovery: An overview[END_REF]. Thus information could be structured data and unstructured information such as freeform text. Furthermore the distribution of data both in an output class or category could be relatively flat or peaky with a multi-modal distribution.

Structured discrete data works well with (i) Association Rule Mining Techniques [START_REF] Feng | Inter-transactional association rules for multidimensional contexts for prediction and their application to studying meteorological data[END_REF][START_REF] Tan | Sequest: Mining frequent subsequences using DMA strips[END_REF], (ii) Decision Tree Methods [START_REF] Zhou | Theoretical and practical considerations of uncertainty and complexity in automated knowledge acquisition[END_REF][START_REF] Zhou | A statistical-heuristic feature selection criterion for decision tree induction[END_REF], and (iii) Rule Search Methods [START_REF] Hadzic | Using competitive learning between symbolic rules as a knowledge learning method[END_REF]. Effective methods for continuous structured data include Neural Net Based Rule extraction methods [START_REF] Sestito | Knowledge acquisition of conjunctive rules using multilayered neural networks[END_REF][START_REF] Sestito | Automated knowledge acquisition[END_REF]. Association Rules are of the form X => Y (s, c) where X and Y are sets of items and s, c are support and confidence respectively [START_REF] Agrawal | Fast algorithms for mining association rules[END_REF]. Association Rule Mining has since been extended to include efficient Apriori-like mining methods [START_REF] Agrawal | Fast algorithms for mining association rules[END_REF], query mining [START_REF] Tsur | Query flocks: A generalization of association-rule mining[END_REF], constraint based rule mining [START_REF] Srikant | Mining association rules with item constraints[END_REF][START_REF] Lakshmanan | Optimization of constrained frequent set queries with 2-variable constraints[END_REF], mining correlations and causal structures [START_REF] Silverstein | Scalable techniques for mining causal structures[END_REF] and interesting associations [START_REF] Ramaswamy | On the discovery of interesting patterns in association rules[END_REF] and mining associations from semi-structured data [START_REF] Wang | Discovering structural associations of semistructured data[END_REF][START_REF] Feng | An XML-enabled association rule framework[END_REF][START_REF] Zaki | XRules: An effective structural classifier for XML data[END_REF]. An XML document possesses a hierarchical document structure, and this is frequently modelled using a labelled ordered tree.

Our research group [START_REF] Feng | An XML-enabled association rule framework[END_REF] initiated an XML-enabled association rule framework. It extends the notion of associated items to XML fragments or subtrees so that one finds associations among ordered trees rather than simple items as in classical association rules. Work has been proposed for mining XML documents [START_REF] Wang | Discovering structural associations of semistructured data[END_REF][START_REF] Feng | An XML-enabled association rule framework[END_REF][START_REF] Zaki | XRules: An effective structural classifier for XML data[END_REF][START_REF] Yang | Efficient mining of XML query patterns for caching[END_REF], including a collection of semi-structured objects [START_REF] Feng | An XML-enabled association rule framework[END_REF]. To help with mining XML documents and other structures, frequent sub tree mining algorithms are being developed [START_REF] Asai | Discovering frequent substructures in large unordered trees[END_REF][START_REF] Zaki | Efficiently mining frequent trees in a forest: Algorithms and applications[END_REF][START_REF] Termier | Treefinder: A first step towards XML data mining[END_REF] which focus on extraction of different types of tree patterns for different applications. Unstructured information has no schema to describe its structure, unlike structured data. We will confine our attention to textual information, and particularly to text mining which has concentrated on two issues, namely (1) categorization of textual documents, and (2) information extraction from a document to elicit a collection of facts or named entities from text documents. The most common approach to text categorization involves three phases:

1. text pre-processing; 2. encoding key information about the document using a feature vector, which is used with the knowledge discovery technique in (3) below; 3. a classification technique or a cluster technique to categorize the document.

Encoding represents the document by a vector of features such as word or phrase or clause, weighted by an importance factor. Classification techniques used include nearest neighbor classification, decision trees, support vector machines, naive Bayes Classifier and clustering methods include SOM, K-means or statistical measures such as regression. Information Extraction essentially involves extracting factual information or named entities from textual data of a certain type. Powerful entity extraction methods include support vector machines, hidden Markov Models [START_REF] Bikel | An algorithm that learns what's in a name[END_REF], Random Fields Methods [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting and labeling sequence data[END_REF] and Maximum Entropy Models. An application in bioinformatics [START_REF] Kim | Introduction to the bio-entity recognition task at JNLPBA[END_REF], found rather poor results showing there remain challenging issues for some domains.

Significance of Work

Currently when deriving business intelligence by knowledge discovery from structured data sources one can often answer the questions related to patterns of what is happening. To answer questions related to why it is happening it will be necessary to derive conjoint mining of content (unstructured/) together with structured data. Thus, for example, a bank examining its database for patterns of customers who have decided to cancel their credit cards could through Data Mining techniques determine some of the features of the customer who is cancelling their credit card. e.g. cobranded cards, branches they belong to, their addresses, length of holding card etc. They would not be able to obtain information from the database of the number and nature of complaints, requests the customer may have made which would be contained in unstructured content repositories holding emails, transcribed phone call information, etc. In order to carry out this conjoint mining it is necessary to link segments of text from the content repository with individual records of interest in the structured data base. This is the problem of semantic integration between structured data and content (unstructured). Considerable work has been done for semantic integration between different sources of structured data. Some work has been done on semantic integration of structured data sources and XML data (semi structured) for the purpose of querying [START_REF] Bergamaschi | Semantic integration of heterogeneous information sources[END_REF][START_REF] Mcbrien | A semantic approach to integrating XML and structured data sources[END_REF]. Very limited work has been done on semantic integration of unstructured data and structured data sources [START_REF] Chakaravarthy | Efficiently linking text documents with relevant structured information[END_REF][START_REF] Bhide | LIPTUS: Associating structured and unstructured information in a banking environment[END_REF], and all of these concentrate on semantic integration for querying these diverse data sources (whether they involve XML data or Unstructured Content). Almost no work to the best of the author's knowledge has been carried out on semantic integration of unstructured content and structured data in a form suitable for deep knowledge discovery through AADC. The information integration approaches above also assume that the schema of the different data sources is available but this is not always possible as enterprises frequently outsource supporting processes to different vendors. For example, Customer Contact Centers are generally outsourced to third parties, who maintain the unstructured information in isolation from the enterprise structured data creating information silos where the schema is not always known. Therefore, we need to develop efficient techniques for correlating the data (structured) and (unstructured) content conjointly, and this task has many technical challenges.

The first issue is that the unstructured data is typically noisy in nature. For example, unstructured data received from different contact channels, like calls, SMS, emails, is very noisy. The problems of cleaning this are somewhat different to that of cleaning structured data. We need to clean this data before we can correlate it with the structured data. The domain of noisy text correction is comparatively new, and we use new techniques for cleansing drawing on the field of automatic spelling corrections [START_REF] Kukich | Techniques for automatically correcting words in text[END_REF] and use of structured data and various ontologies to provide reference information to clarify terms in the text and fill in missing values and terms. The second issue is we need to discover the semantic knowledge/features from the text data. Typical information extraction tools or annotators take plain text as input and identify named entities and simple relations (e.g. works-for) and other text mining annotations based on a data dictionary or a gazetteer approach. For example, given a dictionary of product names, one can identify the product name in the text document. However, the current annotation systems, like UIMA, cannot annotate the documents based on concept, ontology and hierarchy. Therefore, we will extend the UIMA type techniques for annotating the documents based on domain ontology/concept. The third issue is to carryout semantic integration of content and structured data. This requires discovery of the entities in the text data based on the semantic knowledge, which can be matched with the structured entities for data correlation. One of the challenges is that no explicit identifiers of the entity, such as a unique transaction number, may be available in the document. Additionally, the document is noisy, so that a term in the text does not exactly match the corresponding attribute of the entity in the structured data. For instance a customer may mention a different transaction amount in her email or spell her name differently from that in the database. This naturally affects recall. It also affects precision when the noisy and partial information in a document leads to an incorrect entity being identified. We propose the use of a new fuzzy matching algorithm that uses an information theoretic basis. This fuzzy matching algorithm helps overcome the problem of the lack of precise information that permits exact matching. The fourth issue is to develop an intermediate representation suitable for Data Mining. This intermediate representation should be capable of capturing the embedded structure in content as well as the values. It should be in a form suitable to enable the use of conjoint Data Mining techniques. We propose an XML based approach for doing this as it enables one to represent domain information in a more meaningful and specialized way. The fifth issue is the development of algorithms to carry out Data mining conjointly from the content and structured data.

There are also frequently complex and important relationships between information that is stored in the form of structured data and content. To date the ability to find patterns, knowledge and relationships in unstructured text mining is aimed at document classification or entity extraction or simple associations between entities. The ability to find patterns or knowledge relationships between entities that might exist within unstructured textual documents is severely limited. For instance, in the biomedical area one may wish to find the chains involved in metabolic pathways. Furthermore existing techniques for document classification or entity extraction flatten the information structure using feature vectors, losing any structure other than the immediately preceding or following words that might be implicitly embedded or emergent within the document. These implicit or emergent structures may represent how the text is arranged under headings (which are semantically meaningful) or reflect chains of argumentation. Thus when grading an essay, the grader does not only look for the presence or absence of certain words or phrases but also the logical validity of the organization and the structure and clarity of the presentation; otherwise inserting the required words in a semi random fashion could give a good grade. More sophisticated AADC techniques should be capable of detecting such implicit or emergent structures. The approach adopted here, which converts the unstructured text into a semi structured intermediate form such as XML or RDF, will allow better representation of implicit structures. The extension of XML Mining and RDF Mining techniques [START_REF] Bergamaschi | Semantic integration of heterogeneous information sources[END_REF] previously developed by the present authors allows one to investigate the presence of patterns and associations between tree structured items (sub trees within the embedded structures), graph structured items (sub graphs) and sequences within the conjoint data and content instead of just the values of attributes or terms. Correlation of data and content conjointly enables the discovery of interesting relationships and analytics that involve predicates and groupings and their arrangements in combinations. To obtain valuable insights, it is important to find useful associations among concepts which could be dimensions from content and from structured data. The applications in the fields of business, biomedicine and electric power systems and logistics allow one to discover innovative new insights that would be difficult to obtain without the use of conjoint AADC from content and data. The problems tackled in this proposal are very hard and complex problems that are becoming critical with the large proportion of content as well as data that is currently being generated. We develop novel and ground breaking techniques to address each of these issues, which are highly innovative that have the potential to produce a paradigm shift in business intelligence and deep knowledge discovery.

Approach And Methodology

Data and content are stored in repositories that are isolated from each other. Hence, the problem of semantic integration of data and content must be addressed in a form suitable for AADC. One also needs representations and techniques for AADC conjointly from data and content. To resolve these issues, we use two base ontologies: (i) a static concept relation ontology and (ii) an event, transformation ontology which captures the key types of transformations and events allowed. These provide a conceptual framework that enforces an agreement on the organization of information, without losing any of the flexibility of allowing people to express and view parts in their own familiar expression language. An ontology, which is a shared conceptualization of some domain [START_REF] Gruber | A translation approach to portable ontology specifications[END_REF][START_REF] Aberer | Semantics of a Networked World[END_REF], captures and represents the key concepts, relationships and constraints which permits coherent understanding of the meaning of shared information.

The base ontologies will ensure a common ground for understanding content. One way to restrict the scope of disambiguation of particular content within the base ontology is by creating sub-ontology or specialized ontology (also known as Ontology Commitment [START_REF] Jarrar | Formal ontology engineering in the DOGMA approach[END_REF], Ontology Version [START_REF] Klein | Ontology versioning and change detection on the web[END_REF], Materialized Ontology View [START_REF] Wouters | A practical approach to the derivation of a materialized ontology view[END_REF][START_REF] Wouters | Ontologies on the MOVE Database Systems for Advanced Applications[END_REF] appropriate for the category of information being considered. Next, we consider the approaches used for the sub aims.

Unstructured Data Cleansing

The unstructured content is generally noisy, the extent of which is different for each problem e.g. for biomedicine, the unstructured content in journal papers is cleaner. Content like transcribed calls or emails in customer contact centers is very noisy.

We need to clean this information. Processing SMS and email requires different data cleansing, e.g., removing spam messages, disclaimers, promotional material, and previous historical exchanges by the customer can be removed using heuristics for the domain. Even the body of the message is very noisy, using incomplete product name, spelling mistakes, added binary characters, etc. We will use two different approaches to this, (1) which borrows techniques from automatic spelling and grammar checkers [START_REF] Kukich | Techniques for automatically correcting words in text[END_REF] and (2) text cleansing methods which use structured data, and various ontologies (eg. Word Net) to provide reference data to clarify terms in the text and fill in missing values and terms, and deal with term variation arising from synonyms and acronyms. Spelling error correction is related to exact and approximate pattern matching respectively. Spell checking techniques involve non-word error detection and spell correction involves isolated-word or context-dependent error correction. The task involves three steps: (i) morphological analysis to identify a word-stem from a full word-form; (ii) isolating the misspelled words using techniques such as dictionary lookup and ngram analysis; and (iii) offering a list of suggested correct spellings using one or more of six techniques, such as minimum edit distance, similarity key techniques, rulebased techniques, n-gram-based techniques, probabilistic techniques, and neural networks [START_REF] Kukich | Techniques for automatically correcting words in text[END_REF]. When doing this, we compare it to structured terms in the database or synonyms provided from WordNet.

Unstructured Data Annotation

In a document, we distinguish between:

1. Entity Extraction; 2. Identification of a relationship between two entities; 3. A network of relationships between entities; 4. Associations between several entities; 5. Associations between groups of entities such as ones arranged in subtexts and/or subsections.

Most work is on Named Entity Extraction (NER) which finds terms (words or phrases) for a specific named entity. Measures used to judge the efficacy of the algorithms are precision P (classification accuracy), recall R (coverage) and the F-score (harmonic mean of precision and recall). NER methods include probabilistic methods such as Hidden Markov [START_REF] Bikel | An algorithm that learns what's in a name[END_REF][START_REF] Zhou | Recognizing names in biomedical texts: A machine learning approach[END_REF] or Conditional random fields [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting and labeling sequence data[END_REF], rule based methods [START_REF] Tanabe | Tagging gene and protein names in biomedical text[END_REF] or Lexicon methods. Problems in NER are due to the same word or phrase referring to different entities, or many synonyms and/or abbreviations referring to an entity. To resolve these, we use an ontology for the domain of interest which maps these terms to concepts and relationships, recasting it as named concept recognition (NCR). Unlike a lexicon, which defines various items, an ontology has definitions for concepts and their relationships. There are several text to XML Annotators such as UIMA [START_REF] Ferrucci | UIMA: An architectural approach to unstructured information processing in the corporate research environ-ment[END_REF], GLOSS [START_REF] Kaye | The gloss system for trans. from plain text to XML[END_REF], and XI [START_REF]XI: Open-source conver. of legacy text files to XML[END_REF]. We will use the UIMA (Unstructured Information Management Architecture) [START_REF] Ferrucci | UIMA: An architectural approach to unstructured information processing in the corporate research environ-ment[END_REF] back-end that uses both statistical and rulebased annotators for text. Typically such annotators use a data dictionary or a gazette for information extraction from text of named entities (persons, organizations) and simple relations between terms (works-for). To enhance their capability, we will extend the UIMA techniques for annotating the documents based on domain ontology/concept. This permits disambiguation of terms through concept relationships in the same segment of text. We use Annotators to extract relevant tokens from a document and map them to a small subset of the attributes for determining matches in structured data. E.g. by using NER in an annotator one can extract names from a document, and match them against the customer and product name attributes of the transaction table.

One could also extract chunked text such as noun clauses by using a part of speech tagger for matching. This allows us to determine a score for an entity in a document. The highest scoring entity or best matching one can be found without computing explicit scores for all entities. Performing fuzzy match on each extracted token results in a ranked list of possible entities. Entities and relationships determined can be used to define the XML profiles for those documents. The Extraction Methods can be used to determine the values for each tag for an instance document. We use a combination approaches namely (1) ontology-based [START_REF] Embley | Ontology-based extraction and structuring of information from data-rich unstructured documents[END_REF][START_REF] Alani | Automatic ontology-based knowledge extraction from web documents[END_REF] ones that extract terms from text and map them to concepts in an ontology to give semantics and (2) ontology-driven [START_REF] Handschuh | Cream: Creating relational metadata with a component-based, ontology-driven annotation framework[END_REF][START_REF] Vargas-Vera | Mnm: Ontology driven semi-automatic and automatic support for semantic markup[END_REF][START_REF] Gaizauskas | Protein structures and information extraction from biological texts: The pasta system[END_REF] ones that make active use of an ontology to guide or constrain the analysis.

Fuzzy Matching and Data and Content Correlation

Discovering the business insights conjointly requires correlation between data and content.. How can we link information from a text document (TD) with structured data in a database (DB), i.e. find the best matching entities from the DB for the given TD. We filter the annotated TD to retain only the relevant terms while the DB is considered as a set of entity instances and their associated related information. These DB entities are represented for matching as a collection of entity microschematas. We consider a single-type entity identification problem. We define the microschemata, as a rooted tree with the base table as the root and the related tables as the non-root nodes. If any tables have a foreign key relationship in the schema, their nodes are linked by edges. Each row in the base table is identified as an entity, having its own attribute values e.A j . Here, an entity is an instance (a row in the base table) rather than a class level abstraction. The entity row is connected to the appropriate rows in the related table through foreign keys. We also have a collection {di} of TDs (the content) that have references to the entities. Each TD d has a set of terms {ti}. The TD consists of sentences. One or more sentences taken together will be referred to as a segment s. Let e be the central entity for this TD, then each term ti may correspond to some attribute e.A j of entity e. For instance, a TD about a transaction entity refers to the customer name, shop name, date attributes of a specific transaction. Given the terms in a TD, our goal is to identify the central entity (e.g. the Customer A/C#) from the structured table . No explicit identifiers of the entity, e.g. a customer number, may be available in the TD. Also noise in the TD means that t i does not exactly match the corresponding attribute of e. This may lead to (i) not identifying the entity associated with the piece of content -poor recall or (ii) incorrectly identifying a wrong entity with the piece of content-poor precision. We want to link a given segment of the given document with an entity in the DB. There may also be information related to mul-tiple entities in the given segment and we will need to identify these. First we define the microschemata for the structured DB, which is a rooted tree with the base table as the root and the related tables as the non-root nodes. The TD is filtered to retain noun phrases using a part of speech parser and annotated using the annotation techniques referred to above. If necessary "semantic integration within text document" techniques [START_REF] Li | Semantic integration in text: from ambiguous names to identifiable entities[END_REF] can be used to identify terms which refer to the same concept. One next annotates the term using the annotation techniques (say with UIMA) or alternatively using database look ups to identify the column it occurs in. The key idea for matching a term is to determine the information content contained in a term in predicting the entity it refers to and we use an information theoretic formulation for this purpose. From Information theory [START_REF] Reza | An introduction to information theory[END_REF], for a finite probability distribution p i (i = 1,…, m), the entropy is given by

  i m i i m p p p p p H    1 2 2 1 log = ,..., ,
This measure of information content can be considered to be the uncertainty of the occurrence of a term corresponding to the particular entity. Let us assume that the term is contained in the contexts of n(t) distinct entities and there are N entities in total. Hence the probability of occurrence of a given entity e if t i is present is p(t i ) = n(t i )/N. Hence the associated information content

                  N t n N t n t e I i i i ) ( log * , 2
Given that t i occurs f(t i ) times in a particular segment d, the information content associated with t i occurring in the segment d linked to entity e is

                               N t n N t n t f d e I i i d ti i 2 log * * ,
A larger value for I indicates a greater predictive capability. A matching cache that contains a collection of pairs (e,t) (i.e term t is contained in entity e) is populated using two queries, one which returns the set of entities containing the term t and another which returns the set of terms contained in the contexts of the entity e. This avoids repetition of the queries for the same term in a new segment. This cache can be used with the expression above to produce a ranked list of entities that match a segment of text. An alternative to the above approach is to use the Zhou and Dillon Symmetrical Tau [START_REF] Zhou | A statistical-heuristic feature selection criterion for decision tree induction[END_REF] to produce the ranked list. The unstructured text is annotated and represented as an XML document which is matched and merged with the structured data into XML documents using mapping between the concepts. These will be matched using a combination of semantic concept matching, online dictionaries, thesauri, schemas, and structure/related information, and extensions using an ontology. Then both documents will be adjusted to use common concept labels. We then find the common knowledge segments using our U3 mining algorithm [START_REF] Hadzic | U3 -mining unordered embedded subtrees using TMG candidate generation[END_REF], as in [START_REF] Hadzic | Tree mining application to matching of heterogeneous knowledge representations[END_REF]. The additional information from unstructured content in XML corresponds to the unmatched knowledge segments and is used to augment the XML repository.

Techniques for Conjoint Data and Content Mining

To obtain valuable insight, it is important to find useful associations among concepts, from content (unstructured) and from structured data conjointly. It is useful to pre-identify valuable relationships. E.g. identify the top five products that experienced a sharp increase in complaints, common features of the customers and the nature of their complaints, products receiving the most inquiries and the profile of the inquirer and their reasons for the inquiry. Answering such questions give the enterprise timely insights about the customers' concerns. We note that structured data and content (unstructured documents) each have a very different representation. It is important for Integrated Data Mining to get a common intermediate form and we have chosen XML, as it has been successfully used for exchanging data between heterogeneous data sources, can capture the essentials of unstructured textual information, and it allows for mining of values and structures. We have the information held in a structured database with entities, i.e. E = {e 1 ,..., e n } which we represent in its XML form as: {Ex = (e x1 ,..., e xn ), and an unstructured document repository U = {u 1 ,..., u m }.

The corresponding XML representation of the unstructured repository is Ux = {ux1,..., uxn); An extended XML representation transaction is defined which consists of EX = {<exi, ux11> ... <exn, uxn>} where the tuple <e xi , u xi > consists of a concatenation of the XML representations of the two categories of information (E x and U x ).

We used this approach for data records to obtain inter-transactional association rules [START_REF] Tan | Sequest: Mining frequent subsequences using DMA strips[END_REF]. As all data is conjointly represented in an XML document, the problem now becomes one of using the powerful XML mining algorithms to tackle mining of collections of these augmented XML documents. Our recent work has demonstrated the feasibility of conjoint mining of structured databases and XML repositories [START_REF] Pan | Conjoint data mining of structured and semistructured data[END_REF]. An XML-enabled framework for mining of association rules in XML repositories was first presented in [START_REF] Feng | An XML-enabled association rule framework[END_REF] where the rules extracted are more powerful than traditional ones in expressing association relationships at both a structural and semantic level. To extract such rules the most difficult task is to find all the frequent sub trees from an XML database. This is known as the frequent sub tree mining (FSM) problem and is defined as: Given a tree database T db a minimum support threshold (σ) find all subtrees that occur at least σ times in T db [START_REF] Feng | An XML-enabled association rule framework[END_REF]. Being able to mine all different subtree types using different support definitions is particularly important when we work on an XML representation of textual information, since these concepts can be repeated within many fragments of text and there exist different relationships among the concepts in the text, given the flexibility in its representation and expressiveness. The present authors have developed amongst the most powerful algorithms for mining XML document repositories and tree structured data.

The current work builds on this considerable body of expertise. The performance bottlenecks in FSM algorithms are candidate generation and counting, and this is often affected by the ability to effectively represent the document structure. Our work in the FSM field is characterized by a Tree Model Guided (TMG) [START_REF] Hadzic | U3 -mining unordered embedded subtrees using TMG candidate generation[END_REF][START_REF] Tan | Tree model guided candidate generation for mining frequent subtrees from XML[END_REF] candidate generation approach. This non-redundant systematic enumeration model uses the underlying tree structure of the data to generate only valid candidates which conform to the underlying tree structure of the data. We proposed the so called Dictionary, Embedding List (EL) [START_REF] Rust | What you don't know about customerperceived quality: The role of customer expecta-tion distributions[END_REF] and the Recursive List (RL) structures [START_REF] Hadzic | U3 -mining unordered embedded subtrees using TMG candidate generation[END_REF] whose purpose is to capture the structural aspects of a document and allow for efficient access to necessary information. The RL is a more compact representation of the EL that reduces the memory and serves as a global lookup list and also encodes the embedding relationships of the subtrees to be mined. To enable efficient counting we use the Vertical Occurrence List (VOL) [START_REF] Tan | Tree model guided candidate generation for mining frequent subtrees from XML[END_REF][START_REF] Hadzic | Mining unordered distance-constrained embedded subtrees[END_REF] which stores a representation of a subtree encoding together with coordinates. Using the TMG framework with the above representation structures we have presented FSM algorithms for mining of following subtrees (under any support definitions): ordered induced [START_REF] Tan | IMB3-Miner: Mining induced/embedded subtrees by constraining the level of embedding[END_REF] and embedded [START_REF] Tan | MB3-Miner: Mining embedded subtrees using tree model guided candidate generation[END_REF][START_REF] Tan | L Feng MB3-Miner: efficiently mining eM-Bedded subTREEs using Tree Model Guided candidate generation[END_REF], ordered [START_REF] Tan | Razor: Mining distance-constrained embedded subtrees[END_REF] and unordered [START_REF] Hadzic | Mining unordered distance-constrained embedded subtrees[END_REF] distance-constrained embedded, unordered induced [START_REF] Hadzic | UNI3 -efficient algorithm for mining unordered induced subtrees using TMG candidate genera-tion[END_REF] and embedded [START_REF] Hadzic | U3 -mining unordered embedded subtrees using TMG candidate generation[END_REF]. We have also extended TMG for sequence mining [START_REF] Tan | Sequest: Mining frequent subsequences using DMA strips[END_REF]. An important aspect of this process is Trust [START_REF] Alhamad | SLA-based trust model for cloud computing[END_REF][START_REF] Chang | Trust and reputation relationships in service-oriented environments[END_REF] 5

Applications to the Electric Power Industry

The deregulated market allows power consumers to purchase power from different generation companies (gencos) who price the power based on the system Marginal Price [START_REF] Szkuta | Electricity price short-term forecasting using artificial neural networks[END_REF]. To maintain a competitive position, gencos form bilateral contracts with their clients (particularly large ones). These provide the clients with a guarantee of their required energy at a defined cost over a long period (say 5 years). These contracts are in textual form with possibly different terms for each client and are stored in a content repository. The bilateral contracts are legally binding and the genco has to ensure that it can meet the required demand from all the different clients with contracts and other customers who purchase power from the genco as needed. It may also have contracts for supply to it from other gencos. Thus, the genco has to deliver the expected Ensured Energy (EE) to meet its obligations. Otherwise there are many quite severe penalties which are normally staged according to proportion of power not delivered. Uncertainty, in a hydro thermal system being able to meet this EE, is caused by scheduled maintenance, unplanned outages arising from equipment breakdown, power from hydro plants being uncertain due to uncertain inflows, uncertainty in noncontract demand. Historical information related to these factors is stored in structured data bases together with historical spot prices for electrical energy. These will be used to produce forecasts for several of these factors and develop schedules for others such as hydro scheduling [START_REF] Sjelvgren | Optimal operations planning in a large hydro-thermal power system[END_REF] and scheduled maintenance which are stored in structured DBs. This has led to approaches using the structured data to assess the risk especially the loss of load probability and expected unserved energy in the case of no complex contractual terms [START_REF] Dillon | Stochastic optimization and modelling of large hydro-thermal systems for long term regu-lation[END_REF].

However what is needed is a risk assessment and management approach including textual contract terms. The profitability of a company will be impacted by any inability to meet the ensured energy. In this event the genco would try either [START_REF] Brodie | Computer science 2.0: A new world of data management[END_REF] to purchase the extra energy at spot prices which are generally much higher than from its own supplies (and may exceed the contract price) or [START_REF] Fayyad | From data mining to knowledge discovery: An overview[END_REF] to not fulfil the required demand under some contracts, or (3) try to put in place different contracts with other suppliers for the duration of any energy shortfalls, or (4) establish potential new contracts with new clients, or (5) renegotiate existing terms in existing contracts. To make such decisions, requires the genco to extract business intelligence conjointly from (i) the content repository containing information on the different contracts with its clients and suppliers (perhaps with notes on the feasibility of term variations) and (ii) the structured information in the different databases on the different factors. By linking and analyzing this information one can find associations which could result in risky situations and also determine potential remedial actions. Examples could be maintenance patterns, inflow levels (say in a dry year and failure rates which result in energy deficits leading to non-fulfilment of contractual obligations). This would alert the genco early of the need to purchase power from other gencos using supply contracts and refrain from certain client contracts.

Applications to Business Problems

We propose a new approach for consumer expectation-based market segmentation through conjoint mining of content and data. Consumer expectation has long been considered as an important satisfaction determinant that represents market demand and shapes the consumer behaviors [START_REF] Oliver | A cogni. model of the antecedents and conseq. of satisfaction decisions[END_REF][START_REF] Rust | What you don't know about customerperceived quality: The role of customer expecta-tion distributions[END_REF]. Hence, customer segmentation based on their expectations is of great significance for firms to predict dynamics of targeted markets. However, consumer expectations data are often unobservable and prohibitive to collect using traditional market research methods such as interviews, self-reported surveys, experiments, etc. Conjoint mining provides a new means by which marketers are able to understand the 'minds' of millions of consumers on a daily basis without having to physically interact with consumers (e.g. shadowing, field experiment.) or explicitly soliciting opinions (e.g. interviews, surveys) from them. Consequently, data reliability and model validity bears more substantial rigor than previous approaches. Using conjoint mining techniques, consumer expectation can be inferred from customer satisfaction data gleaned from online customer reviews which include both structured and unstructured data, e.g. the reviews on Epinions.com hold numerical ratings towards each product attribute (e.g. 'battery life' for digital cameras, 'memory capacity' for MP3 players) as well as free text comments from consumers on their opinions and experiences.

Conjoint mining allows the use of structured data to reveal latent customer expectations based on unobservable concomitant variables such as consumer preferences, taste, values and the use of content to mine consumer opinions in free text, to (1) extract product features from the reviews, and (2) obtain consumer affects and sentiments towards these. We develop algorithms which augment opinion mining methods used in the interactive data analysis from [START_REF] Ku | Mining opinions from the web: Beyond relevance retrieval[END_REF] and the overall processes of opinion tracking from [START_REF] Glance | Deriving marketing intelligence from online discussion[END_REF]. To discover heterogeneous expectations towards different brands, comparison-based algorithms [START_REF] Jindal | Identifying comparative sentences in text documents[END_REF] will be leveraged and redeveloped. The Web usage mining [START_REF] Büchner | Discovering internet marketing intelligence through online analytical web usage mining[END_REF], sentiment-based algorithms [START_REF] Turney | Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews[END_REF] and opinion holder identification algo-rithms [START_REF] Kim | Determ. the sentiment of opinions[END_REF] will be integrated in order to make markets segments 'actionable' for managers and produce a custom score function to classify the sentiment [START_REF] Kim | Determ. the sentiment of opinions[END_REF].

Product extraction will augment the method in [START_REF] Drenner | Insert movie refer. here: A system to bridge conversation and item-oriented web sites[END_REF] and borrow some ideas from entity-based search engines [START_REF] Cheng | Entityrank: Searching entities directly and holistically[END_REF] to mark items from free texts. Both product and feature extraction will allow business analysts to annotate text with an ontology and allow the unstructured customer opinions/ complaints to be linked with structured customer data in internal DBs or Customer Relationship Management systems.

Application in Transport Logistics industry

It is important to understand that today's transport logistics providers spent 50% of their time on managing the physical mess and 50% on managing the related information mess [START_REF] Chang | Transport Logistics, the Grand Challenges[END_REF]. Here, intelligent transportation has enabled vehicle to driver, vehicle to vehicle and vehicle to infrastructure communications and emerging intelligent infrastructure that provides embedded un-manned situation awareness 24/7 that enables greater mobility, security and safety.

It is also important to realise that over the years, the Transport Logistics sector has generated and accumulated much more valuable economic information than Facebook. This informs us on Big data impacts on global financial movement and Financial forecast including financial forecasts. Logistics professionals around the world know that they are no longer just transport and logistics operators, they are required to be "Data Experts" or at least to have Data Experts in their organization. Our ARC (Australia Research Council) Logistics Industry Partners in New South Wales and Queensland have been pushing their data to the Cloud since 2009 with vendor support. However, this Big Data has not been fully utilised, due to the lack of availability of co-joint data and content mining technology.

Further, many manufactured items, goods or assets today utilizing the Internet of Things are already Internet enabled, they have capability to talk to Internet, talk to each other, talk to logistics providers and talk to logistics infrastructure. This has sped up the automated people, goods and asset movement in logistics, transportation, warehouse and distribution [START_REF] Chang | Transport Logistics, the Grand Challenges[END_REF] sector.

Intelligent Tracking powered by co-joint data and content mining is the core technology that is needed in transport logistics industry today. Tracking movement of people, goods and services in the entire logistics network, tracking quality of services, service providers performance, through entire life cycle of supply chain and asset management, track and trace of data and information shared over the logistics alliances, coalition partners and joint forces, situation awareness and ambient intelligent, for productivity, security and safety. Intelligent tracking powered by conjoint predictive analytics with real time data and in real time environment is a major challenge for all modern transport logistics providers. We have been working with our industry partner to adopt conjoint predictive analytics and co-joint mining for monitoring, visualisation, sharing, control and management of physical mess (goods and assets) and information mess (data) as well as business processes for their Business Intelligence in-cluding maximisation of human, transport and infrastructure performance and minimisation of the costs and security risk.

The co-joint data and content mining on Big Data in transport logistics sector including the combined RFID and wireless sensors data on the goods and assets handling, warehousing and transportation, GPS, GPRS and position location system for transport vehicle and shipment tracking, Surveillance Systems for Operator Performance and situation awareness, provenance of Goods and Asset tracking. The co-joint data and content mining are also needed for Inter-and intra-logistics partners transactions data monitoring, customers based tracking of trades data, smart phone, bluetooth, and black-box (on heavy vehicles and ships vessels) communication and even logistics social networks to support auto and semi-automated physical flow and information flow which enables business intelligent.

We use transport logistics ontology to help manage the Big data by defining the meaning of data through adding context that gives information on the data. Our works include Ontologies, RDF annotations and contexts. We carry out mining and visualization of big data both relational data (warehouse data or 3PL data) and complex data includes tree structured data (Geo-data), XML documents (procedures and workflows), unstructured textual data (smart phone notification and web data), image data (positions and locations), multimedia data (surveillance data), graphical data (Asset tracking data).

One of our biggest challenges in the co-joint data mining has been the assurance that the Big data are from trusted sources, the data services for Big data are trusted such as Clouds, and the Quality of Data, especially in the automated environment utilising Internet of Things and Cyber-Physical Systems. If the wrong decision is made based on the poor data set, it could result in major financial losses, high casualties and possible terrorist attack through the use of transport.

Applications to biomedical applications

Existing biomedical information is distributed across a large number of information resources and is heterogeneous in its content, format and structure. This hinders effective information retrieval. Targeted searches are very difficult with current search engines as they look for the specific string of letters within the text rather than its meaning. Use of highly expressive knowledge models such as ontologies enables the machines to view the text as meaningful expressions. This increases the semantics and forms the basis of a more efficient approach to finding the right information. An ontology can be used for creating metadata by semantic annotation of text through three steps: tokenization (splitting the sentences into tokens), matching the tokens against the ontology terms and matching the tokens against the ontology relationships until the best fit is found. New web pages created can be annotated automatically during their creation process. This semantic annotation allows machines to access web content, understand it, retrieve and process the data automatically rather than only display it. In our research work , we have developed a number of ontologies, such as Protein Ontology [START_REF] Sidhu | Protein ontology: vocabulary for protein data[END_REF], Human Disease Ontology [START_REF] Hadzic | Medical ontologies to support human disease research and control[END_REF] and Mental Health Ontology [START_REF] Hadzic | Towards the mental health ontology[END_REF]. The ontologies can be used to annotate target information in content sources and enable intelligent retrieval of specific information, analysis of it and linking with the existing pool of knowledge. E.g., protein and bibliographical reference data available via Swiss-Prot can be linked with the related publications from PubMed and with the epidemiological data in medical databases. Conjoint content and data mining of the linked content/data provides quality knowledge that can help build effective prevention and intervention strategies. Thus the presence of the protein, PSA, at a given level or given form (free or complex) at a certain age or ethnicity or lifestyle, might be indicative of a certain probability of the existence of cancer. Conjoint mining of the two structured databases and the textual information in PubMed will help with the discovery of such knowledge. There may be situations which coincide with some ambiguity or inconsistency. This will help researchers identify what requires further investigation.

Conclusions

The paper presents a methodology for conjoint mining of structured and unstructured information. The potential impact in industry of use of BI and AADC can be inferred from a 2003 IDC study of 40 US and European companies that use predictive analytics KDD who achieved a median Return of Investment of 145%, achieved higher investment levels and yielded higher overall returns over five years. These improvements occurred in just effectively utilizing the 5% of information available as structured data. This effect would be considerably amplified if one could in an integrated fashion exploit the remaining 95% of content as well as the 5% of structured data. This research, by developing an integrated approach for BI and AADC of data and content, will provide a competitive edge in handling such information. This integrated knowledge discovery techniques could improve policy formation and effectiveness by Government and non-Government in such areas as compliance by companies, reduction of aberrant behavior in areas such as health benefits allocation, pension entitlements etc. It will provide an intellectually rigorous approach to underpin the trend in electronic document handling.


Yiannis Kokkinos

Konstantinos G Margaritis

Breaking ties of plurality voting in ensembles of distributed neural network classifiers using soft max accumulations

Keywords: Neural Networks, distributed computing, ensemble classifiers, majority voting, plurality voting

An ensemble of distributed neural network classifiers is composed when several different individual neural networks are trained based on their local training data. These classifiers can provide either a single class label prediction, or the normalized via the soft max real value class-outputs that represent posterior probabilities which give the confidence levels. To form the ensemble decision the individual classifier decisions can be combined via the well known majority (or plurality) voting that sums the votes for each class and selects the class that receives most of the votes. While the majority voting is the most popular combination rule many ties in votes can occur, especially in multi-class problems. Ties are usually broken either randomly where the unknown instance is assigned randomly to one of the tied classes or using the class proportions where the tied class with the largest proportion wins. We present a tie breaking strategy that uses soft max confidence accumulations. Every class accumulates a vote and a confidence for this vote. If a tie occurs then the tied class with the maximum confidence sum wins. The proposed tie breaking in the voting process performs very well in all cases of different data distributions on various benchmark datasets.

Introduction

Ensembles of neural network classifiers [START_REF] Hansen | Neural network ensembles[END_REF][2][3] [START_REF] Zhou | Ensembling neural networks: many could be better than all[END_REF] are very popular tools in pattern recognition. Ensembles methods [START_REF] Kuncheva | Combining Pattern Classifiers: Methods and Algorithms[END_REF][6] [START_REF] Seni | Ensemble Methods in Data Mining[END_REF] are well known for their high accuracy and robustness since the combined predictions of several classifiers outperforms their individual predictions. In addition to the generalization performance, for which they are preferred, ensemble methods are usually the only choice available for other reasons like: (i) reduction of computational complexity by partitioning the data set into several smaller sub-sets, training different estimators on these sub-sets, and combining their predictions, (ii) modularity, by building modular systems that work on different input spaces, (iii) distributed learning from physically distributed data repositories as well as Peer-to-Peer systems where gathering large volumes of data to a single location is unfeasible, (iv) their native parallelism and data scaling via the divide-andconquer approach.

Distributed data collections can usually lay either in physically distributed database systems, in which case a small number of locations hold large volumes of data, or in Peer-to-Peer systems, where a large number of locations have typically small volumes of data. In any case the individual neural network classifiers are independently constructed in parallel, based on their local training data. After this training phase is finished, the classification predictions are combined via several schemes [START_REF] Kittler | On Combining Classifiers[END_REF] in which the most popular one is majority voting [START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF][10] [START_REF] Ho | Decision combination in multiple classifier systems[END_REF] [START_REF] Lam | Application of majority voting to pattern recognition: An analysis of the behavior and performance[END_REF] or its most frequently used version the plurality voting combining scheme. In plurality voting (for reviews see [START_REF] Kuncheva | Combining Pattern Classifiers: Methods and Algorithms[END_REF] [8] [START_REF] Lam | A theoretical analysis of the application of majority voting to pattern recognition[END_REF][11] [START_REF] Lam | Application of majority voting to pattern recognition: An analysis of the behavior and performance[END_REF]) a classification of an unlabeled instance is performed according to the class that obtains the highest number of votes (most of the votes). The strict version of (majority) voting needs the agreement of more than half of the participants to reach a decision, although this scheme can work for binary class problems only. Thus in reality, to cover the multi-class problems, the commonly used voting process is plurality voting, which selects the candidate class that receives the most votes. That is why usually one do not distinguish between plurality voting and majority voting, and the term "majority voting" is used even if the underlining criterion is plurality voting. This method is also known as the basic ensemble method [START_REF] Rokach | Ensemble-based classifiers[END_REF].

However there may be more than one class that receives the largest number of classifications (or votes) by the ensemble members. Hence a tie can occur. In real life cases a tie-breaking chairperson can resolve this problem [START_REF] Brams | Voting Procedures[END_REF]. In classification problems such ties are usually broken arbitrary by randomly selecting one of the tied classes [START_REF] Kuncheva | Combining Pattern Classifiers: Methods and Algorithms[END_REF]. This random tie-breaking receives justification by the fact that all possible ways that other voters can vote are equi-probable. In the case of strictly binary class datasets one can just use an odd number of classifiers. In the case of k-nearest neighbour classifiers a nearest neighbour tie-break can be used. Another common strategy is choosing the class that is selected most often among the tied classes or tied classifiers [START_REF] Woods | Combination of Multiple Classifiers Using Local Accuracy Estimates[END_REF]. This strategy predicts the class based on the largest class proportion among the ties classes. Another similar strategy can use the frequencies of predicted classes that occur during the training phase.

In this work we utilize confidence (soft max) accumulation as a method of breaking the ties in majority voting. Our motivation was the fact that a neural network classifier outputs, by default, the predicted class label accompanied by the confidence of this prediction given by the soft max function. Hence, we have tried to exploit further this extra information to improve the tie-breaking process and we have find out that it usually performs much better. Thus, the proposed tie-breaking method requires no additional computing resources other than those of simple majority voting. It works with classifiers, like neural networks, than in principle can accompany their classprediction with a level of confidence, by using the soft max function. Every neural network classifier simply sends the predicted class label and its confidence conf for this class-prediction. Thus every prediction constitutes a pair {label, conf}. When a tie occurs in the voting process and two or more classes get the same number of votes then the proposed tie-breaking method sums up the received confidences for each of these tied classes. The class that accumulates the maximum confidence sum wins.

The ensemble of distributed regularization networks

We employ Regularization Neural Networks [START_REF] Poggio | Regularization algorithms for learning that are equivalent to multilayer networks[END_REF] [16] [START_REF] Evgeniou | Regularization Networks and Support Vector Machines[END_REF] for implementing the Neural Network classifiers. Regularization Networks are well known for using the real training data points as centers for their hidden neuron kernels. This is valuable when data features have discrete values, like in cases of image processing, computer vision and data mining. Using the real training data as kernel centers is a common strategy in kernel methods that can capture the data closeness approximate of any underlined problem distribution. These qualities elevate such type of stable kernel based learning methods to state of the art in modern machine learning [START_REF] Kashima | Recent Advances and Trends in Large-scale Kernel Methods[END_REF].

We assume that each data location holds Input Layer Hidden Layer Output Layer . . .

k 2 k 3 k Np . . . k 1 f B W A,B,C f C f A

Fig. 1. An ensemble of Regularization Neural Networks distributed in several locations

The Regularization Network input neurons are as many as the number of data features. The hidden neurons are as many as the number of the training instances. A location p has N p training instances. The output neurons are as many as the classes. The hidden-to-output layer weights w are computed for each class-output by solving a regularized risk functional. Assume three classes A, B and C then for the paradigm in fig. 1 the class output estimations of classifier f p () for an unknown x will be given by:

f p,A (x) = ∑ = p N n 1 w A,n • k(x n , x) (1a) f p,B (x) = ∑ = p N n 1 w B,n • k(x n , x) (1b) f p,C (x) = ∑ = p N n 1 w C,n • k(x n , x) (1c)
The class-output with the maximum value is chosen for the predicted class. The kernel function k(•,•) can be any symmetric, positive semi-definite function, and the most commonly used is the Gaussian kernel k(x n , x)=exp(-||x n -x || 2 /σ 2 ) centered at a particular x n point. The local kernel matrix K with entries K ij = k(x i ,x j ) and size N p ×N p contains the information regarding the high density regions inside classes and the separating planes in between classes. To find the weights w in the Regularization Network (RN) [START_REF] Poggio | Regularization algorithms for learning that are equivalent to multilayer networks[END_REF] [16] [START_REF] Evgeniou | Regularization Networks and Support Vector Machines[END_REF] the learning problem is stated as the minimization on a Reproducing kernel Hilbert space (RKHS) of a regularized risk functional which has the usual data error term plus a regularization term that embodies the stabilizer:

     ∑ = ∈ Np n p H f N K 1 1 min arg (y n -f p (x n )) 2 +    2 K p f γ (2)
The data term is scaled proportionally to the number of data points, and γ > 0 is the regularization parameter. The minimization solution set is unique and the weight vectors w A , w B , and w C for the three class paradigm that correspond to a regularization network classifier p are given by solving the linear systems [START_REF] Poggio | Regularization algorithms for learning that are equivalent to multilayer networks[END_REF] [1] [START_REF] Evgeniou | Regularization Networks and Support Vector Machines[END_REF] in eq. 3:

w A = (K + N p γ I) -1 y A (3a) w B = (K + N p γ I) -1 y B ( 3b
)
w C = (K + N p γ I) -1 y C ( 3c
)
where I is the identity matrix, K is the local kernel matrix (of size N p ×N p ) and y A , y B and y C are the vectors (of size N p ) that hold the desired labels for each class A, B and C respectively, which in the hot encoding are 1 for labels of the same class points and 0 for the others. The rescaled regularization parameter N p γ is usually small and can be found during training via cross validation. In the experimental runs we use a typical value N p γ = 0.1 for all the local Regularization Networks.

For an unknown x the class-output responses f p,A (x) f p,B (x) and f p,C (x) of each neural network f p are normalized, so that they fall in the unity interval [0,1] and thus to reflect posterior probabilities. These posteriors produced by the softmax function are interpreted as confidence [START_REF] Bishop | Neural Network for Pattern Recognition[END_REF]. For instance the softmax confidence for the A class is:

h p,A (x) = exp(f p,A (x)) / (exp(f p,A (x))+exp(f p,B (x))+exp(f p,C (x))) (4)
Eq. 4 gives theoretically the Bayesian posterior probability (see also [START_REF] Bishop | Neural Network for Pattern Recognition[END_REF] for proof) as approximated by the regularization network. Thus each classifier outputs the predicted class together with the confidence for this class.

Combining predictions using majority voting

In a distributed parallel environment the combination of the decisions is straightforward. The majority voting is used in classification tasks where the individual classifiers produce a single class label, where in this case each classifier "votes" for a particular class, and the class with the majority vote on the ensemble wins. This means that for an unknown x each classifier f p (x) outputs one integer number, given by the max argument of its class-outputs f p,m (x), that designates the most probable class label. With M classes and L classifiers the correct class is the k th class that collects the largest number of votes, given by: class

(x) = ( ∑ = L p M k 1 max arg { if k = ( M m 1 max arg = f p,m (x) ) then 1 else 0 } ) (5)
The simplest method to handle ties relays on a First Labelled basis which in case of a tie favours the class that was labelled earlier (or first). The drawback is that in a tie the class 1 (the first) will never lose, while the class M (the last) will never win.

The most commonly used tie-breaking method during the voting process is to break the ties arbitrary by Randomly Selecting one of the tied classes [START_REF] Kuncheva | Combining Pattern Classifiers: Methods and Algorithms[END_REF]. If a tie occurs this method choose a random class among the tied ones.

Differently from the random selection another way is to check which of the tied classes has higher prior probability and accordingly make the decision. Usually the class proportions given by the number of examples in every class (for the three class problem these are N A /N, N B /N and N C /N) can representing the priors. Hence this method uses Class Proportions, and the ties are going in favour of the largest proportions.

Another way for counting priors is to monitor how many times a class occurs during the training process. Thus if class A and class B receive the same number of votes but the class B has higher frequency of occurrence, meaning that is predicted more often that the class A during the training of all the classifiers, then B is selected. Hence this method uses Class Frequencies during training.

A variant of the previous method is to count the correct predictions during the training of all classifiers and use these Class Correctly Frequencies as priors.

Using neural networks each classifier can afford to send a pair {label, conf}, where label is the predicted class m of the unknown x and conf is the soft max confidence of this prediction. The proposed method in this paper when a tie occurs is to sum the confidences for the received votes of the tied classes and select the one class that accumulates the Maximum Confidence sum. Therefore, with L classifiers which output their prediction m and their soft max confidence h p,m (x) when classifying x in class m, the correct class is the k th class that collects the largest confidence sum (eq. 6):

class(x) = ( ∑ = L p M k 1 max arg {if k = ( M m 1 max arg = f p,m (x) ) then h p,m (x) else 0} ) (6)
To provide an example consider the running paradigm in fig. 1, which illustrates a three-class problem with 5 neural network classifiers where each one can produce a pair {label, conf}. Assuming that for an unknown x the classifier f 1 () votes for class A with confidence 0.7 which gives a pair {A, 0.7} while f 2 () gives {A, 0.8}, f 3 () gives {B, 0.5}, f 4 () gives {B, 0.6} and f 5 () gives {C, 0.7}. A tie occurs for the classes A and B with 2 votes each. In this case the class A wins with sum of confidences 1.5.

Experimental simulations

The classification performance of every tie-breaking method for majority voting are measured on a number of publicly available real-world benchmark problems which are downloaded from the UCI machine learning data repository [START_REF] Frank | UCI Machine Learning[END_REF] (http://archive.ics.uci.edu/ml). The specific details of these datasets are illustrated in table 1, where they are ordered by the number of their classes. The groups of experiments aim at comparing the error rates of the proposed tiebreaking method and to make direct comparisons against the others. The experimental design is as follows:

(i) A dataset is randomly split into a training set (80%) and a test set (20%) with stratification (ii)

The training set of size N, is divided equally into L disjoint partitions (N p =N 1 =N 2 =…=N L ) which are distributed to the different locations, (iii) every location p holds N p data and builds a Regularization Neural Network classifier f p () based on its own local examples, (iv)

The classifiers are combined and the ensemble is tested on the test set. (v)

The procedure is repeated 30 times for each benchmark dataset and each ensemble population. We measure the classification error rate which counts the number of incorrectly classified examples and divides by the test set population number.

The comparison results of the error rates and the standard deviations for each dataset and each tie-breaking method are illustrated in table 2. In table 2 the dataset name is in the first column. The second column has the number of classifiers in the ensemble. The third column illustrates the tie-breaking using the simplest "first labelled" method. The fourth column shows the results for the widely used method of breaking ties arbitrary by randomly selecting one of the ties classes in the voting. The fifth column is the tie-breaking by selecting among the tied classes the one with the largest "class proportion", namely the largest number of training examples. The sixth column uses "class frequencies" and selects among the tied classes the one class that had the maximum occurrence during training process of the classifiers (the one that appears most frequently). The seventh column corresponds to the tie-breaking that uses "class correctly frequencies" which selects the class among the tied classes that had the maximum number of correctly occurrences during the training process. The last column is the proposed method that sums also the received confidence of those votes and selects the tied class with the maximum confidence sum. Note that in many cases there are very small differences of the proposed method with the others. The differences depend only on the number of ties since the voting algorithm is otherwise the same. This simply means that not many ties where encountered during the testing.

We run several experiments and in most of the cases the proposed method of using confidence produces better results than the other tie-breaking strategies. In general, one can notice from table 2 that the differences of the methods in the comparisons are not substantial when not many ties occur. However when several ties occur during the voting process then the difference of the proposed method as compared with the others becomes considerable. In some multi-class datasets like the Vowel the proposed tie-breaking method has 4 units less error rate than that of the other methods.

Conclusions

Majority or plurality voting is the most popular decision combination rule in neural network ensembles. In many cases, regardless of number of classes and neural networks in the ensemble a tie in voting occurs. Usually such ties are broken arbitrary. We present a tie-breaking technique that uses confidence. Since each neural network classifier can afford to send by default the predicted label and the confidence for this prediction (via the soft max function), the proposed method receives pairs of {predicted class, confidence} for every unknown x. Thus every class accumulates a voting sum and a confidence sum. If a tie occurs then the tied class with the maximum confidence sum wins. When compared with other tie-breaking methods, like random class selection, or class selection using prior class probabilities, the proposed tie breaking performs very well in all cases of different data distributions on various benchmark datasets.

1 =

 1 N p training examples and there are L different data locations. Thus location p can train a neural network classifier f p () based on its local training set{x n , y n } p N n . Such an ensemble composed of Regularization Networks is illustrated in fig. 1 for the three-class case.

Table 1 .

 1 Benchmark dataset details (ordered by class population)

	Dataset Name	instances Features Classes
	Sonar	208	60	2
	Wisconsin (Diagnostic) 683	9	2
	Diabetes Pima Indians	768	8	2
	Spambase	4601	57	2
	Phoneme	5404	5	2
	Wine	178	13	3
	Vehicle Silhouettes	846	18	4
	Ecoli	336	7	5
	Page Blocks	5473	10	5
	Glass	212	9	6
	Dermatology	358	34	6
	Satellite Image	6435	36	6
	Yeast	1479	8	9
	Optical Digits	5620	64	10
	Vowel Context	990	11	11
	Spectrometer	531	100	24
	Letter	20000	16	26

Table 2 .

 2 Error rates for the majority voting tie-breaking methods.

	Dataset	Ensemble	First	Random	Class	Class	Class	Use max
		size	labeled	selection	Propor-	Fre-	Cor-	Confi-
					tions	quencies	rectly	dence
	Sonar	10	25.58	26.35	27.36	27.36	27.36	23.25
			±6.33	±6.47	±6.34	±6.34	±6.34	±5.50
	Wisconsin	20	4.50	4.28	4.50	4.50	4.50	4.28
			±1.62	±1.57	±1.62	±1.62	±1.62	±1.55
	Diabetes	20	24.95	25.08	24.95	24.95	24.95	24.27
			±2.42	±2.32	±2.43	±2.43	±2.43	2.54
	Spambase	40	9.64	9.66	9.65	9.65	9.65	9.54
			±0.83	±0.91	±0.95	±0.95	±0.95	±0.90
	Phoneme	40	18.42	18.54	18.42	18.42	18.42	18.56
			±1.15	±1.29	±1.15	±1.15	±1.15	±1.25
	Wine	10	2.61	2.52	2.43	2.34	2.43	2.34
			±2.25	±2.50	±2.45	±2.29	±2.45	±2.39
	Vehicle	20	31.03	31.55	31.50	31.48	31.77	30.86
			±2.59	±2.48	±2.37	±2.47	±2.30	±2.63
	Ecoli	10	14.25	14.49	14.58	14.58	14.58	13.94
			3.26	3.21	3.36	3.36	3.36	3.47
	Page Blocks	40	6.37	6.33	6.37	6.37	6.37	6.28
			±0.35	±0.32	±0.35	±0.35	±0.35	±0.37
	Glass	10	39.55	39.55	39.70	39.70	40.00	39.40
			±5.33	±5.72	±5.73	±5.87	±5.62	±5.03
	Dermatology	10	3.12	3.56	3.12	3.60	3.12	3.51
			±1.88	±2.27	±1.88	±2.18	±1.88	±2.35
	Satellite Image	40	12.85	12.89	12.89	12.85	12.85	12.81
			±0.54	±0.55	±0.58	±0.56	±0.56	±0.86
	Yeast	20	39.79	39.73	39.79	39.87	39.87	39.65
			±1.95	±2.02	±1.95	±1.90	±1.90	±1.97
	Optical Digits	40	2.87	2.83	2.88	2.88	2.88	2.76
			±0.41	±0.40	±0.40	±0.41	±0.39	±0.40
	Vowel Context	10	24.17	23.95	24.17	24.66	23.85	19.12
			±4.16	±4.45	±4.16	±4.90	±3.85	±4.32
	Spectrometer	10	55.66	56.16	56.16	56.19	56.31	54.95
			±4.37	±4.02	±4.64	±4.72	±4.35	±4.49
	Letter	40	11.51	11.44	11.47	11.60	11.45	11.04
			±0.59	±0.61	±0.57	±0.59	±0.60	±0.62

Data Repository [http://archive.ics.uci.edu/ml] CA: University of California, (2014)