Victoria Lush
email: lushv1@aston.ac.uk

Jo Lumsden
email: j.lumsden@aston.ac.uk

Lucy Bastin
email: l.bastin@aston.ac.uk

Visualisation of Trust and Quality Information for Geospatial Dataset Selection and Use: Drawing Trust Presentation Comparisons with B2C e-Commerce

Keywords: Geospatial Data Quality and Trustworthiness, Trust Visualisation, Trust Indicators

The evaluation of geospatial data quality and trustworthiness presents a major challenge to geospatial data users when making a dataset selection decision. Part of the problem arises from the inconsistent and patchy nature of data quality information, which makes intercomparison very difficult. Over recent years, the production and availability of geospatial data has significantly increased, facilitated by the recent explosion of Web-based catalogues, portals, standards and services, and by initiatives such as INSPIRE and GEOSS. Despite this significant growth in availability of geospatial data and the fact that geospatial datasets can, in many respects, be considered commercial products that are available for purchase online, consumer trust has to date received relatively little attention in the GIS domain.

In this paper, we discuss how concepts of trust, trust models, and trust indicators (largely derived from B2C e-Commerce) apply to the GIS domain and to geospatial data selection and use. Our research aim is to support data users in more efficient and effective geospatial dataset selection on the basis of quality, trustworthiness and fitness for purpose. To achieve this, we propose a GEO label -a decision support mechanism that visually summarises availability of key geospatial data informational aspects. We also present a Web service that was developed to support generation of dynamic GEO label representations for datasets by combining producer metadata (from standard catalogues or other published locations) with structured user feedback.

Introduction

To address issues of geospatial data quality, international organisations, initiatives, and working groups such as the International Organisation for Standardization (ISO) [1], the Open GIS Consortium (OGC) [2], INSPIRE [3], and many more, are actively working to establish, improve and extend geospatial data and metadata standards. Despite the detailed recommendations of standardisation bodies, and despite the existence of formal metadata standards such as ISO 19115:2003, data quality information is, however, often not communicated to users in a consistent and standardised way [START_REF] Griraa | Towards a collaborative knowledge discovery system for enriching semantic information about risks of geospatial data misuse[END_REF]. While standardisation efforts have significantly improved metadata interoperability, an increasing choice of metadata standards poses a number of unresolved questions: Which standards are best to follow? How much metadata to provide? How to make metadata 'useful' and not just 'usable'? [START_REF] Brown | Usability of geographic information: current challenges and future directions[END_REF]. Since metadata standards are mostly focused on data production rather than potential data use and application, a typical metadata document is not sufficient to effectively communicate dataset fitness for purpose to users from a variety of domains and expertise levels [START_REF] Griraa | Towards a collaborative knowledge discovery system for enriching semantic information about risks of geospatial data misuse[END_REF][START_REF] Boin | What communicates quality to the spatial data consumer?[END_REF]. Geospatial data users are presented with an increasing choice of data available from various data portals, repositories, and clearinghouses [START_REF] Griraa | Towards a collaborative knowledge discovery system for enriching semantic information about risks of geospatial data misuse[END_REF]. This means that the intercomparison of dataset quality and the evaluation of a dataset's fitness for purpose can present a major challenge for geospatial data users. Over the past decade, many researchers and scholars have attempted to address the challenge of communicating geospatial data fitness for purpose information, proposing a more 'user-centric approach' to geospatial metadata [START_REF] Pôças | Evaluating the fitness for use of spatial data sets to promote quality in ecological assessment and monitoring[END_REF]. Researchers argued the case for enriching metadata records with: references to relevant literature (citations information); less formal opinions from the data producers; expert opinions of data quality; and user feedback regarding previous data use [START_REF] Comber | Approaches for providing user relevant metadata and data quality assessments[END_REF]. Recent reviews, however, suggest that these recommendations have not yet been put into practice, with no practical means for collating and searching user-focused metadata, added to which many metadata records that are available are incomplete [START_REF] Griraa | Towards a collaborative knowledge discovery system for enriching semantic information about risks of geospatial data misuse[END_REF][START_REF] Pôças | Evaluating the fitness for use of spatial data sets to promote quality in ecological assessment and monitoring[END_REF][START_REF] Lush | Visualisation of quality information for geospatial and remote sensing data: providing the GIS community with the decision support tools for geospatial dataset quality evaluation[END_REF][START_REF] Lush | Developing a geo label: providing the GIS community with quality metadata visualisation tools[END_REF].

Trust significantly influences our decision making. In the field of Business to Consumer (B2C) e-Commerce, trust is considered to be a crucial enabler for online transaction decisions [START_REF] Kim | An empirical study of customers' perceptions of security and trust in e-payment systems[END_REF]. The impersonality of geospatial dataset selection decisions closely mirrors that of the e-Commerce transaction experience. Transactional risk is a vital precondition of e-Commerce trust [START_REF] Chopra | Trust in electronic environments[END_REF]: similarly, the risks involved in dataset selection and use (i.e., the importance of selecting the right dataset for a given purpose) establish a need for dataset users to trust in dataset providers to deliver a reliable, quality dataset to meet their needs. In B2C e-Commerce, trust indicators are used to engender consumer trust in e-Vendors; it can, therefore, be argued that it should be possible to establish and deploy similar trust indicators in the GIS domain to convey information about the trustworthiness of geospatial datasets and dataset providers. In essence, drawing on the parallels with B2C e-Commerce, it can be argued that representation and visualisation of key trust indicators associated with geospatial datasets and their producers has the potential to support more effective, informed, and trust-based selection of quality datasets. Surprisingly, research into mechanisms of representing trust in the GIS domain has not yet received the same level of attention as it has in the e-Commerce domain [START_REF] Skarlatidou | Guidelines for trust interface design for public engagement Web GIS[END_REF][START_REF] Moreri | Volunteered geographic information quality assessment using trust and reputation modelling in land administration systems in developing countries[END_REF].

To tackle the challenge of data quality and trustworthiness assessment and dataset selection decision making, we present a GEO label -a voluntary label designed to improve user assessment of the quality of geospatial datasets and promote trust in datasets that carry the label. This paper presents research conducted to define a GEO label that has the capacity to act as a trust indicator for geospatial data. We also introduce a Web service developed to support GEO label generation for datasets by combining producer metadata (from standard catalogues or other published locations) with structured user feedback.

The following section of this paper outlines a review of related work (much of it from the field of B2C e-Commerce which considers consumer trust in service provision, aligning well with the concept of consumer trust in GIS dataset provision) which illustrates some of the important concepts underpinning this research arena. Subsequent to this, we present the research we have conducted to define and develop a GEO label and to establish a Web-service for the generation of GEO labels. We conclude with a discussion and reflection on trust in the GIS domain based on our research experience.

Related Work

Prior to its recent introduction to the GIS domain, trust has been extensively researched and successfully adopted in B2C e-Commerce to enable online marketplace transactions [START_REF] Moreri | Volunteered geographic information quality assessment using trust and reputation modelling in land administration systems in developing countries[END_REF]. In this section, we review related work on trust to draw important parallels with the GIS domain and demonstrate the significance of trust in geospatial dataset selection and use.

Concept and Models of Trust

Trust is a fundamental part of our everyday life [START_REF] Hosmer | Trust: The connecting link between organizational theory and philosophical ethics[END_REF]. Without the presence of trust, society would experience a loss of effectiveness, task performance and dynamism leading to its inevitable destruction [START_REF] Marsh | Formalising Trust as a Computational Concept[END_REF]. There exist many types of trust -with trust being viewed as a multi-dimensional concept -and there are many disciplines and research fields (e.g., economics, social psychology, sociology, management, marketing, information systems, commerce, and e-Commerce) that study this phenomenon. The definition of trust largely depends on the nature of the relationships and contexts to which it applies [START_REF] Hosmer | Trust: The connecting link between organizational theory and philosophical ethics[END_REF]. In sociology, trust is described as a mechanism for coping with the freedom of others [START_REF] Luhmann | Trust and power[END_REF]. In psychology, trust is viewed as a personality characteristic (interpersonal trust) [START_REF] Rotter | Interpersonal trust, trustworthiness, and gullibility[END_REF] or a "psychological state" [19, p. 398]. In e-Commerce, many researchers adopt a common definition of trust as being a belief or positive expectation that a vendor will fulfil promised obligations and that the vendor will not take any actions that will negatively affect the trustee [START_REF] Geyskens | The effects of trust and interdependence on relationship commitment: A Trans-Atlantic study[END_REF].

A series of models of trust have been proposed [e.g., [START_REF] Ganesan | Dimensions and Levels of Trust: Implications for Commitment to a Relationship[END_REF][START_REF] Mcknight | Developing and validating trust measures for ecommerce: An integrative typology[END_REF][START_REF] Mccord | The impact of trust on the technology acceptance model in Business to Consumer E-Commerce[END_REF][START_REF] Tan | Online consumer trust: A multi-dimensional model[END_REF][START_REF] Lee | Towards a dynamic and composite model of trust[END_REF]. Ganesan and Hess [START_REF] Ganesan | Dimensions and Levels of Trust: Implications for Commitment to a Relationship[END_REF] present two dimensions of trust -credibility and benevolence. Business studies of trust have identified credibility (the belief that the vendor has the necessary capacity to complete a task effectively and reliably) and benevolence (the belief that the vendor has good intentions and will behave in a favourable manner even in the absence of existing commitment) as critical factors of trust [START_REF] Doney | An Examination of the nature of trust in buyer-seller relationships[END_REF].

Institutional trust comes from sociology and refers to trust in institutions, such as laws and regulation in society [START_REF] Pennanen | Consumers views on trust, risk, privacy and security in e-commerce: A qualitative analysis[END_REF] and the presence of essential structural conditions [START_REF] Mcknight | Developing and validating trust measures for ecommerce: An integrative typology[END_REF]. In e-Commerce, institutional trust denotes trust in the Internet as a whole and particularly trust in the technology that it offers [START_REF] Pennanen | Consumers views on trust, risk, privacy and security in e-commerce: A qualitative analysis[END_REF]. Interpersonal trust is an individual's trust in another specific party [START_REF] Tan | Online consumer trust: A multi-dimensional model[END_REF]; in e-Commerce, this type of trust can represent a customer's trust in an e-vendor, trust in third-party assurances of e-vendor trustwor-thiness and integrity, or a friend's recommendation of an online vendor [START_REF] Pennanen | Consumers views on trust, risk, privacy and security in e-commerce: A qualitative analysis[END_REF]. Dispositional trust was defined in the area of psychology, and refers to an individual's ability and willingness to trust in general. Dispositional trust is particularly important in the initial stages of a relationship and in novel situations where familiarity is absent [START_REF] Pennanen | Consumers views on trust, risk, privacy and security in e-commerce: A qualitative analysis[END_REF].

Further trust classifications include initial and experiential trust [START_REF] Marsh | Trust in design[END_REF], vertical and horizontal trust [START_REF] Lee | Towards a dynamic and composite model of trust[END_REF], and technological and relational trust [START_REF] Mccord | The impact of trust on the technology acceptance model in Business to Consumer E-Commerce[END_REF]. Initial or 'grabbing' trust refers to a first trusting judgement at the commencement of a novel situation or relationship and is highly influenced by an individual's disposition to trust. Experiential trust comes with experience and familiarity and is considered to be much more complex than initial trust. Lee and Yu [START_REF] Lee | Towards a dynamic and composite model of trust[END_REF] describe the notion of vertical and horizontal types of trust: "vertical trust captures the trust relationships that exist between individuals and institutions, while horizontal trust represents the trust that can be inferred from the observations and opinions of others" [25, p. 9]. McCord and Ratnasingam [START_REF] Mccord | The impact of trust on the technology acceptance model in Business to Consumer E-Commerce[END_REF] discuss technological and relational types of trust. They define technological trust as "the subjective probability by which an individual believes that the underlying technology infrastructure and control mechanisms are capable of facilitating inter-organizational transactions according to its confident expectations" [23, p. 921]; they refer to relational trust as "a consumer's willingness to accept vulnerability in an online transaction based upon positive expectations of future e-retailer behaviours" [23, p. 921]. In Section 7, we reflect on the parallels between established models of trust as outlined here and trust processes/mechanisms in the domain of GIS dataset provision and use based on our research outcomes.

e-Commerce Trust Indicators

Extensive research has been conducted in the e-Commerce domain to identify trust indicators that can be embedded within e-Commerce websites to engender user willingness to engage in online transactions [START_REF] Jarvenpaa | Consumer trust in an internet store[END_REF]. Many researchers [e.g., [START_REF] Jarvenpaa | Consumer trust in an internet store[END_REF][START_REF] Lumsden | How does personality affect trust in B2C e-Commerce?[END_REF][START_REF] Liu | Website attributes in urging online impulse purchase: An empirical investigation on consumer perceptions[END_REF] either directly or indirectly illustrate that these trust indicators (known as trust triggers) can be effective in engendering consumer trust in e-Commerce and, hence, in promoting online transactions. In essence, an online trust trigger is an element of a website that acts as an indicator of the trustworthiness of the website [START_REF] Lumsden | How does personality affect trust in B2C e-Commerce?[END_REF]. Lumsden and MacKay [START_REF] Lumsden | How does personality affect trust in B2C e-Commerce?[END_REF] identify nine of the most generally agreed-upon trust triggers that are commonly used in e-Commerce, namely: customer testimonials and feedback; professional website design; branding; third party security seals; up-to-date technology and security measures; alternative channels of communication between consumers and the vendor; clearly stated policies and vendor information; consistent (professional) graphic design; and ease of navigation. The parallels between trust triggers in e-Commerce and the facets of GIS datasets on which dataset users base selection decisions (according to the outcomes of our research) are discussed in Section 7.

Trust in the GIS Domain

There are a number of parallels between consumers' decisions to transact with a given e-Commerce vendor and dataset users' decisions to adopt one from n datasets for their given needs. While trust has to date received relatively little attention in the GIS domain, some GIS researchers and scholars have made attempts to highlight the importance and relevance of trust to geospatial data and systems. Harvey [START_REF] Harvey | Developing Geographic Information Infrastructures for Local Government: The Role of Trust[END_REF] evaluated effects of trust on development of the National Spatial Data Infrastructures (NSDI) in the United States. The results of telephone and face-to-face interviews, surveys and workshops with local government agencies' staff indicated that trust directly impacts willingness to share data. Bertino et al. [START_REF] Bertino | security and privacy for geospatial data: Concepts and research directions[END_REF] discuss the role of trust in terms of managing, accessing and sharing of geospatial data that is used for safety-critical applications.

They propose that, to engender user trust in geospatial data, geospatial data repositories should: maintain complete logs of data provenance including data source and the submission date; utilise mechanisms for dynamic verification of the data source; and introduce privacy policies for protecting sensitive information from privacy violations. With recent growth in production and availability of volunteered geographic information (VGI) which is commonly generated by non-experts, trustworthiness of VGI is increasingly attracting the attention of the GIS community. With a focus on filtering more reliable socially-generated geospatial content, Bishr and Janowicz [START_REF] Bishr | Can we trust information? -The case of Volunteered Geographic Information[END_REF] propose using trust as a proxy measure of VGI quality. They argue that quality is a subjective measure, but if trust-rated geospatial information is useful and relevant to many users then it is of satisfactory quality. Keßler and de Groot [START_REF] Keßler | Trust as a proxy measure for the quality of volunteered geographic information in the case of OpenStreetMap[END_REF] also support the idea of using trust as a proxy measure of VGI quality and identify five provenance-based trust parameters of VGI observations, namely: versions; users; confirmations; tag corrections; and rollbacks. Moreri et al. [START_REF] Moreri | Volunteered geographic information quality assessment using trust and reputation modelling in land administration systems in developing countries[END_REF] present a novel trust and reputation modelling methodology to establish the quality and credibility of VGI such that it can be considered in land administration systems on a fit for purpose basis. Their research is motivated by lack of official geospatial data in developing countries.

Despite recent research efforts to highlight the importance of trust in the GIS domain, thus far no practical work has been done to establish a standard visualisation of trust that users can utilise to compare the trustworthiness of datasets. There is also a lack of transition of trust knowledge from other domains such as psychology, sociology and e-Commerce where notions of trust and trust cues have been established and empirically confirmed.

GEO Label Initiative

The Global Earth Observation System of Systems (GEOSS) [START_REF]GEOSS[END_REF] is a distributed 'system of systems' which is being constructed by the Group on Earth Observation (GEO) [START_REF] Geo | [END_REF] to provide decision-support tools to a wide variety of users. Given that the GEOSS is estimated to contain more than 28 million dataset records and is constantly growing [START_REF] Zabala | Rubric-Q: Adding quality-related elements to the GEOSS clearinghouse datasets[END_REF], choices faced when selecting a dataset can (depending on usage domain) be quite daunting. With such a huge choice of datasets comes the problem of data quality assessment and dataset selection decision making. To tackle this challenge, the GEO Science and Technology Committee (STC) [40] proposed to establish a GEO label -a label "related to the scientific relevance, quality, acceptance and societal needs for activities in support of GEOSS as an attractive incentive for involvement of the S&T communities" [41, p. 2]. The STC suggested that the development of such a label could significantly improve user recognition of the quality of geospatial datasets and that its use could help promote trust in datasets that carry the established GEO label [START_REF]A GEO Label: Informing users about the quality, relevance and acceptance of services, data sets and products provided by GEOSS[END_REF]. Research presented in this paper was conducted to define and develop the proposed GEO label to act as a quality and trustworthiness indicator and support fitness for purpose dataset evaluation.

Method

The main focus of our research was to design the concept of a GEO label founded on knowledge elicited about how geospatial data users select datasets to use, the reasoning behind their selection decisions, and what mechanisms could improve their experience.

Our research adopted an iterative user-centered design (UCD) approach in order to generate solutions that are tailored to geospatial data users' needs and that are likely to garner user acceptance once deployed.

Utilising various tried-and-tested UCD methods, our research comprised a series of phases of research (exploration, development, evaluation and validation), with each phase building upon the knowledge gathered in the previous phase(s) [START_REF] Lush | Visualisation of quality information for geospatial and remote sensing data: providing the GIS community with the decision support tools for geospatial dataset quality evaluation[END_REF].

A preparatory phase [START_REF] Lush | Visualisation of quality information for geospatial and remote sensing data: providing the GIS community with the decision support tools for geospatial dataset quality evaluation[END_REF][START_REF] Lush | Developing a geo label: providing the GIS community with quality metadata visualisation tools[END_REF] was conducted using a series of semi-structured, face-toface and telephone interviews with geospatial data expert users and producers. The intention was to uncover initial information about dataset selection, including their use and production within representative application areas, in order to inform later research phases. A total of 18 interviewees were recruited, representing a variety of expert groups including end data users, researchers, data archivists, academics, and data producers.

Phase I [START_REF] Lush | Visualisation of quality information for geospatial and remote sensing data: providing the GIS community with the decision support tools for geospatial dataset quality evaluation[END_REF] was conducted via a comprehensive online questionnaire-based survey that comprised over 60 questions to solicit initial geospatial data producers' and users' views on the concept of a GEO label and the role it should serve. A total of 87 valid questionnaire responses were received: 57 from self-identified dataset users and 30 from self-identified dataset producers.

Phase II [START_REF] Lush | Visualisation of quality information for geospatial and remote sensing data: providing the GIS community with the decision support tools for geospatial dataset quality evaluation[END_REF] focused on the iterative design of the graphical representation of the GEO label. A comprehensive questionnaire-based study that also comprised over 60 questions was conducted to solicit geospatial data producers' and users' views on the proposed GEO label visualisations. A total of 26 valid questionnaire responses were received, 10 from 'primarily dataset users', 3 from 'primarily dataset producers', and 13 from 'equally data users and data producers'. Following this, GEO label designs were adapted and improved in line with geospatial experts' feedback and recommendations.

Findings and Outcomes

In this section, we highlight the key findings from the studies conducted to define and develop a GEO label.

Initial Investigation

Verbatim transcripts of the interview recordings were generated to support detailed data analysis. The transcripts were thematically analysed to identify the informational facets of importance to users when assessing dataset fitness for purpose and to derive detailed user requirements that relate specifically to quality and trustworthiness assessment of datasets for the purpose of making dataset selection decisions. The analysis of the interview transcripts identified that geospatial data users highly value good quality metadata records. The study participants stated that complete and well-documented metadata records are essential in the assessment of geospatial data quality and trustworthiness. Core metadata defined in ISO standards must be provided with geospatial datasets to enable comparative evaluation of dataset quality and trustworthiness. The study revealed the importance of dataset provenance and licensing information when assessing whether to trust that a dataset was fit for purpose. Data users confirmed that provenance information is usually incomplete and licensing information is normally missing from the metadata records of datasets. Dataset users are also interested in soft knowledge about data quality -i.e., data providers' comments on (a) the overall quality of a dataset, (b) known data errors, (c) potential data use, and (d) any other information that can help in the assessment of fitness for use of datasets. Also important when selecting a quality dataset are peer recommendations and reviews: dataset users are keen to be able to obtain feedback from their peers and are willing to accept peer recommendations when trying to select the most appropriate dataset for their given needs. The study results revealed the importance of citation information when assessing whether a dataset is fit for purpose, yet there was general consensus that citation information is, unfortunately, hard to acquire. It was discovered that, when selecting a dataset, users typically seek information about dataset providers and, in particular, value the availability of valid contact details for providers. Finally, study findings indicated that having side-by-side dataset and metadata comparison functionality would make the dataset selection process much easier for users.

The results suggested that a GEO label would best serve a drill-down function whereby, at the top level, it visually represents the availability of specific informational elements for its associated dataset and, thereafter, permits users to click the label to drill down into the detail for each informational element. Based on the interviewees' responses, GEO label-appropriate facets were identified as potential candidates for inclusion in the GEO label, namely: the reputation of the dataset producer; producer comments on the dataset quality; the dataset's compliance with international standards; community advice; dataset ratings; expert value judgments; links to dataset citations; quantitative quality information; and side-by-side metadata comparison. It was decided that provenance information could be effectively conveyed via the producer profile, producer comments and citations information facets. Licensing information was not included due to perceived lower importance and the fact that it is nearly always missing from dataset descriptions.

Phase I and Phase II Studies

To investigate geospatial data producers' and users' views on the concept of a GEO label and the role it should serve, an online questionnaire-based survey was conducted.

The questionnaire presented various examples of e-Commerce review/rating systems (some incorporated click-to-verify/drill-down functionality to access additional information) and certification programmes and seals in order to explore respondents' preferences for the role of a GEO label.

Overall, the results of this study showed that users and producers of geospatial datasets appeared to have generally very positive attitudes towards the development and introduction of a GEO label. The study illustrated that geospatial dataset users rely heavily on metadata records when assessing dataset fitness for use, and reiterated the problems associated with the lack of uniform availability of quality-associated information despite ongoing standardisation efforts. For these reasons, many respondents agreed that a GEO label could potentially fulfil a certification or assurance seal function and be used to impose higher standards on provision of metadata records. Respondents demonstrated positive attitudes towards the concept of a GEO label that provides some sort of rating and review facilities, seeing this as appropriate support for more subjective metadata recording and assessment for datasets. The majority of users and producers strongly supported the notion of a GEO label providing an all-in-one drill-down interrogation facility that would combine expert value judgements, community advice, links to citation information, side-by-side visualisation of metadata records, etc.

Based on the study findings, prototypic GEO label graphic representations (i.e., static images) were developed which could potentially be used to convey availability of spatial dataset quality information (see Fig. 1). These GEO label visualisations combined the 8 identified and confirmed informational aspects, namely: dataset producer information; producer comments on the dataset quality; the dataset's compliance with international standards; user feedback (community advice); user ratings of the dataset; expert reviews (expert value judgments); dataset citations; and quantitative dataset quality information. Side-by-side metadata visualisation would require at least two datasets and does not represent an informational facet of a single dataset alone, consequently it was decided not to include this function in the GEO label visualisation itself.

Each informational facet was designed to show whether the information it represents is 'available', 'not available' or 'only available at a higher level' (i.e., information is available for a parent dataset) for the dataset with which the GEO label is associated. A questionnaire-based study was designed to evaluate the effectiveness of, or potential issues with, the proposed GEO label designs and to arrive at a final, communitysupported GEO label representation. Overall, the study results indicated that, unfortunately, none of the proposed GEO label visualisations were as yet sufficiently effective (in the eyes of the intended community of use at least) to stand as the final GEO label design. Nevertheless, respondents' feedback provided rich information on which basis to identify essential GEO label design modifications and improvements and derive user-defined GEO label requirements. The GEO label facet icons and the overall label design underwent some modifications and improvements. Despite the attempt to convey provenance information via the producer profile, producer comments and citations information facets, data producers argued that producer-related quality information was underrepresented in the label and requested that an additional facet was established to solely represent lineage information: as such, a data provenance information facet was added to the final GEO label design. Conversely, feedback and ratings facets were combined into a single user feedback facet because the study results indicated relatively low attributed importance and perceived redundancy of the user ratings facet. The final GEO label therefore comprised the following facets (see Table 1): producer information; producer comments; the dataset's lineage/provenance information; the dataset's compliance with international standards; user feedback (community advice); expert reviews (expert value judgments); dataset citations; and quantitative dataset quality information. These were combined into a label as shown in Fig. 2.

Table 1. Graphical representations and descriptions of the GEO label informational aspects.

Facet Icon Facet Description

Producer profile conveys availability of information about the producer of the dataset, e.g., organisation or individual who produced the dataset.

Producer comments conveys availability of any informal comments about the dataset quality as provided by the dataset producer, e.g., any identified problems.

Lineage information conveys availability of lineage/provenance information, e.g., processing applied to data and number of process steps.

Standards Compliance conveys availability of information about dataset's compliance with international standards, e.g., compliance with ISO 19115.

Quality information conveys availability of formal quality measures of the dataset, e.g., uncertainty measures, errors, accuracy information, etc.

User feedback conveys availability of feedback, comments and ratings provided by the users of the dataset, e.g., identified problems, etc.

Expert reviews conveys availability of domain experts' comments on dataset quality, e.g., results of formal quality checks, expert suggestions, etc.

Citations information conveys availability of citations where the dataset was used and cited, e.g., formal reports on dataset quality checks, journal articles, etc.

Fig. 2.

Final GEO label design: left: information is available; middle: information is available at a higher level; right: information is not available.

To convey the availability of quality information, each informational facet can represent one of three data availability states: 'available'; 'not available'; and 'available only at a higher level' (to indicate that information is not immediately available for the dataset, but is available for a parent dataset). These three information availability states are expressed by varying the appearance of the facet icons as shown in Table 2. The final GEO label design was formally evaluated as part of a study of a decision-support system which was developed to utilise the label in selection of datasets: discussion on this is outside the scope of this paper but interested readers can read more here [START_REF] Lush | Visualisation of quality information for geospatial and remote sensing data: providing the GIS community with the decision support tools for geospatial dataset quality evaluation[END_REF].

Table 2. Graphical representations and descriptions of the GEO label availability states.

Facet Appearance Availability State Description

Fully filled-in background + white icon with black outline -information is available for this dataset.

White background + white icon with black outline -information is not available for this dataset (at any level).

Partially filled-in background + white icon with black outline -information is available only at a higher level for this dataset.

GEO Label Service Implementation

To support use of the graphical GEO label, we developed a GEO label service as a stand-alone Web-based server-side application, exposed via a publicly available RESTful API. Representational State Transfer (REST) is an abstract architectural style that constrains the implementing application to adopt a stateless client-server model with a uniform interface, meaning that "resources" made available by an application are represented by a Uniform Resource Identifier (URI) with a communication protocol that defines methods for accessing and modifying the state of these resources. A prime example of a system implementing this architecture with the Hypetext Transfer Protocol (HTTP) used for communication is the World Wide Web, where clients use HTTP method verbs to inform a server how to process their requests for a resource's URIe.g., GET for the retrieval of information and POST for accepting data (commonly used when creating new resources).

The GEO label service is designed to dynamically process producer metadata and feedback XML documents for a given dataset and, based on evaluated information availability, build a clickable SVG (Scalable Vector Graphic) GEO label representation for that dataset. The service accepts encoded URLs of publicly available metadata documents or metadata XML files as part of an HTTP GET request, or locally-available files uploaded through a POST request, and applies XPath and XSLT mappings to transform the supplied XML documents into SVG representations. The service is underpinned by two metadata XML-based quality models that were developed by the Ge-oViQua project [START_REF] Geoviqua | [END_REF]. The first is the Producer Quality Model (PQM) [START_REF] Bastin | Deliverable D6.1 Data Quality Encoding as a Best Practice Paper[END_REF] that extends ISO 19115:2003 [44], ISO 19115-2:2009 [45] and ISO 19157:2013 [46], adding means to report publications, discovered issues, reference datasets used for quality evaluation, traceability, and statistical summaries of quantified uncertainty. This model introduces elements to record qualitative and quantitative quality information, and to identify resources (i.e., geospatial datasets) in order to relate metadata in hierarchical or other ways. The second is the User Quality Model (UQM) [START_REF] Bastin | Deliverable D6.1 Data Quality Encoding as a Best Practice Paper[END_REF][START_REF] Broek | Deliverable D3.2 User Feedback Elicitation Tool[END_REF], developed to enable application of 'customer' reviews to datasets which span a variety of user expertise levels, thematic, temporal and spatial domains. This model re-uses a few ISO quality and metadata elements, and elements of the PQM, but is far less strictly bound to existing ISO schemas. These two models aim to fill significant perceived gaps identified by users and producers of geospatial data, such as the formalisation of soft knowledge quality parameters (e.g., discovered issues, publications, lineage), the standardisation of statistical quality metrics, and the ability to collect feedback from users to support the more 'user-centric' metadata. Although the services primarily rely on the GeoVi-Qua quality models, an external XPath configuration file which is used for determining whether information is available can be adapted to support any XML-based metadata models.

Generated SVG GEO labels offer dynamic hover-over functionality for obtaining quick summary information. Hovering over an individual facet in the GEO label displays a summary of the information related to the facet for the associated dataset -e.g., producer name, producer comments, the name of the standard to which the dataset complies, etc. (see Fig. 3).

Fig. 3. Examples of producer profile and producer comments hover-over functionalities.

A drilldown GEO label function is designed to provide detailed structured information extracted from the associated dataset's metadata record when a facet is clicked. The GEO label service API is used to transform producer metadata and feedback XML documents into styled, structured HTML pages. Figure 4 provides an example of a citations information summary page that was generated using the GEO label drilldown function. The practical implementation of the GEO label demonstrated that producer metadata documents can, in practice, be effectively combined with user feedback to generate an integrated visualisation of a user-focused summary of geospatial dataset quality and trustworthiness. It has, additionally, confirmed the feasibility of not only the drilldown GEO label function for obtaining detailed dataset information, but also the hover-over function for viewing a quick quality summary. The SVG format of the GEO label representation allows for integration of the essential dataset quality and trustworthiness information and ensures label interactivity. Regarding the technological side of the GEO label implementation, the key advantage of the service is in its interoperabilityit allows for the GEO label to be integrated within any GIS application that supports HTTP requests.

Discussion and Reflection on Trust in the GIS Domain

The findings from the studies presented here lend empirical evidence supporting observations of direct parallels between geospatial data quality and trust themes and welldefined trust models and trust triggers that are already used extensively in B2C e-Commerce to engender consumer trust and increase user willingness to engage in online transactions. Compliance of geospatial data and metadata with international standards directly relates to vertical and institutional trust and is comparable to use of B2C e-Commerce trust seals. Peer reviews and recommendations on quality of geospatial datasets together with ratings of datasets relate to horizontal trust and reflect consumer testimonials that are widely used by online vendors to engender consumer trust in the products and/or services that a vendor provides. Expert reviews also relate to horizontal trust but are different from consumer testimonials; hence this informational aspect does not have an exact counterpart in terms of an e-Commerce trust trigger. Producer reputation relates to credibility and benevolence dimensions of trust and reflects the e-Commerce branding trust trigger. The B2C e-Commerce trust trigger relating to alternative channels of communication is also relevant here since geospatial data users are highly interested in the availability of dataset producers' contact information. Informal producer comments (soft knowledge) also relate to producer credibility and benevolence, although this informational aspect does not have a corresponding e-Commerce trust trigger. Availability of citations information relates to both horizontal and vertical dimensions of trust -horizontal trust is supported by descriptions of the dataset quality and recommendations on the dataset's use supplied in scientific papers, while vertical trust is trust in a journal or a conference where the document was published. Availability of quantitative quality information and licensing information links to institutional trust since it indicates that the dataset adheres to some international standard and is supplied in a structured manner. Geospatial dataset provenance and the method(s) adopted for data collection and processing relate to vertical and technological dimensions of trust, respectively -i.e., trust in the organisation that produced the dataset and technologies that were used to collect and process the data to produce the dataset. Citations information, quantitative quality information, data provenance and licensing do not have corresponding B2C e-Commerce trust triggers.

Trust clearly has the potential to have a major impact on users' geospatial dataset selection and quality evaluation processes. When searching for a suitable geospatial dataset, users may come across new data repositories or unknown data providers, in which case they have to decide whether to engage in a trusting relationship (initial trust) with that provider. Users may reflect on their previous experiences with geospatial data producers and providers to decide whether or not to return to a dataset provider to acquire future data sets (experiential trust). Furthermore, in any decision to trust a dataset provider, a user is essentially making an assessment of the provider's credibility, the technological trustworthiness of the provider or producer, and the observance of standards set by higher orders (vertical trust). Users may also contact their peers, work colleagues or friends to get advice and recommendations on what data could be suitable for given tasks (horizontal and interpersonal trust), they may seek information on projects or companies who have previously used a given dataset (horizontal trust), or they may look to journal papers, expert reviews and technical reports where dataset quality checks have been reported when making a selection decision (horizontal trust). When selecting from several dataset options, some users may be more keen on datasets that adhere to international standards and are supported with standardised metadata documentation (institutional and vertical trust). In contrast, in situations where consequences of data misuse are very severe, users may choose not to select datasets themselves but to use a third-party organisation to select datasets for them (institutional trust and third-party credibility).

The mapping between trust concepts and GIS dataset selection and use suggests that trust plays a vital role in geospatial data selection and use. Every time geospatial data users select a dataset to use, they are likely to have to make a trusting decision, often without even realising they are doing so. We believe that the GEO label effectively combines various notions of trust discussed above and can act as an all-in-one geospatial data trust indicator. It represents a novel decision support mechanism that enables a more efficient and informed evaluation of geospatial dataset quality and trustworthiness, and facilitates more effective dataset intercomparison and selection.

Conclusion

This paper described, in brief, user-centred design research conducted to establish the concept of a GEO label and identify the role it should serve in the visualisation of geospatial data quality and trustworthiness. The paper presented three phases of research conducted to: (a) identify the informational aspects upon which users rely when assessing geospatial dataset quality and trustworthiness; (b) elicit initial user views on the role of a GEO label in supporting dataset comparison and selection; and (c) evaluate prototype label visualisations to arrive at a final GEO label representation. The findings indicated that, to engender user trust, geospatial data producers need to supply complete metadata records, supporting documentation, and contact information with the datasets that they produce. It was also discovered that peer recommendations are of great value to geospatial data users and that users would want to see e-Commerce review functionality available in geospatial data portals, catalogues and clearinghouses. Practical implementation of the GEO label demonstrated that it is possible to develop an effective voluntary quality label without having to establish a new standard, standardisation body or a certification programme. The developed solution not only fulfills the needs of the geospatial community, but also addresses the STC's initial vision of a GEO label that would comprise objective labelling (producer metadata) and subjective labelling (user-focused metadata).

The GEO label integration into geospatial data portals should raise community awareness of metadata incompleteness. It is much easier to conceal metadata incompleteness in complex XML files; even tabular views can give a false impression of information availability since some records provide long lists of keywords, responsible parties, and points of contact. For geospatial data producers, the GEO label can act as a graphical template of quality and trust information that should be provided with every geospatial dataset and should encourage producers to supply rich metadata.

Via literature review, discussion and reflection on our findings, this paper also demonstrated how research on trust in other domains can be applied to geospatial data and GIS applications. It was revealed that geospatial data quality and trust indicators largely mirror B2C e-Commerce trust triggers. Drawing on the research and knowledge in the e-Commerce domain, it is suggested that the GIS domain should employ similar trust promoting mechanisms to engender user trust in geospatial data and GIS applications.

Fig. 1 .

 1 Fig. 1. Prototypic graphic representations of the GEO label.

Fig. 4 .

 4 Fig. 4. Example of citations information drilldown page.


Dhiren A Audich
email: daudich@uoguelph.ca

Rozita Dara

Blair Nonnecke
email: nonnecke@uoguelph.ca

Privacy Policy Annotation for Semi-Automated Analysis: A Cost-Effective Approach

Privacy policies go largely unread as they are not standardized, often written in jargon, and frequently long. Several attempts have been made to simplify and improve readability with varying degrees of success. This paper looks at keyword extraction, comparing human extraction to natural language algorithms as a first step in building a taxonomy for creating an ontology (a key tool in improving access and usability of privacy policies). In this paper, we present two alternatives to using costly domain experts are used to perform keyword extraction: trained participants (nondomain experts) read and extracted keywords from online privacy policies; and second, supervised and unsupervised learning algorithms extracted keywords. Results show that supervised learning algorithm outperform unsupervised learning algorithms over a large corpus of 631 policies, and that trained participants outperform the algorithms, but at a much higher cost.

Introduction

A 2015 Pew Research Centre survey found that 91% of American adults either agree or strongly agree that they have lost control of how their private information is collected and used [START_REF] Madden | Americans Attitudes About Privacy, Security and Surveillance[END_REF]. The collection of personally identifiable information (PII) by online service providers is often justified with claims of creating a more user-centric web experience. However, PII is sold and shared frequently with third parties that use it to profile users and track them across domains. While users are increasingly concerned about their privacy online [START_REF] Jensen | Privacy Policies as Decision-Making Tools: An Evaluation of Online Privacy Notices[END_REF] they scarcely understand the implications of PII sharing [START_REF] Winkler | Privacy Policy Analysis of Popular Web Platforms[END_REF].

Privacy policies are the only means of informing users and mitigating their fears over privacy loss, and by law, companies have to disclose the gathering, processing, and sharing of PII in their privacy policies [START_REF]Privacy Online: Fair Information Practices in the Electronic Marketplace[END_REF][START_REF][END_REF][START_REF]Personal Information Protection and Electronic Documents Act[END_REF][START_REF]Digital Privacy Act[END_REF]. Unfortunately, most policies are often lengthy, difficult and time-consuming to read, and as a result are infrequently read [START_REF] Mcdonald | Cost of reading privacy policies, the[END_REF][START_REF] Mcdonald | A comparative study of online privacy policies and formats[END_REF][START_REF] Milne | Strategies for reducing online privacy risks: Why consumers read (or don't read) online privacy notices[END_REF][START_REF] Jensen | Privacy Policies as Decision-Making Tools: An Evaluation of Online Privacy Notices[END_REF]. The demotivating nature and the difficulty of reading privacy policies amounts to a lack of transparency. Failing to provide usable privacy policies prevents users from making informed decisions and can lead them to accept terms of use jeopardizing their privacy and PII.

Recently, Cranor et al. showed through their analysis of 75 policies that most policies do not provide enough transparency about data collection for the users to make informed privacy decisions [START_REF] Cranor | Are They Worth Reading-An In-Depth Analysis of Online Trackers' Privacy Policies[END_REF].

In addition to length and readability, privacy policies also differ from one another by their content of legal and technical jargon, and coverage [START_REF] Mcdonald | A comparative study of online privacy policies and formats[END_REF][START_REF] Mcdonald | Cost of reading privacy policies, the[END_REF][START_REF] Wilson | The creation and analysis of a website privacy policy corpus[END_REF]. While it is true that FIPPs (Fair Information Practice Principles) and OECD (Organization for Economic Co-operation and Development) offer general guidelines for writing privacy policies, they only provide a conceptual framework; a qualitative review of policies reveals that language and structure being used differs between policies and economic zones (E.U., U.S.A, Canada) [START_REF] Mcdonald | A comparative study of online privacy policies and formats[END_REF][START_REF] Sadeh | The usable privacy policy project: Combining crowdsourcing, machine learning and natural language processing to semi-automatically answer those privacy questions users care about[END_REF]. There is also an inconsistent amount of jargon used between policies, and policies of organizations in the E.U. tend to have supplementary information that tends to be absent from American and Canadian policies [START_REF] Wilson | The creation and analysis of a website privacy policy corpus[END_REF][START_REF] Sun | Automatic Evaluation of Privacy Policy[END_REF][START_REF] Cranor | Are They Worth Reading-An In-Depth Analysis of Online Trackers' Privacy Policies[END_REF]. Boilerplate language is mostly the norm for cookie policies.

To improve readability and semi-automate their evaluation, attempts to introduce a standard structure to privacy policies has met with limited success. For example, the preeminent Platform for Privacy Preferences (P3P) [START_REF] Cranor | A P3P preference exchange language 1.0 (APPEL1. 0)[END_REF][START_REF] Cranor | P3P: Making privacy policies more useful[END_REF] failed due to poor adoption and issues related to validating policies [START_REF] Lämmel | Understanding privacy policies[END_REF]. In response to user concerns, several prominent online service providers started using privacy enhancing technologies (PET), e.g., opt-out mechanisms, anonymisation of personal data, and layered policies [START_REF]Ten steps to develop a multilayered privacy notice[END_REF][START_REF] Munur | Best Practices in Drafting Plain-Language and Layered Privacy Policies[END_REF]. Without these becoming a common standard, opaque and verbose policies are still the norm.

Given the widespread deployment of privacy policies and their importance to users, we propose to semi-automate the evaluation of policies and support their reading through the use of intelligent reasoning. By combining intelligent reasoning with natural language processing (NLP) techniques we hope to reduce the time needed for users to find key information in policies by highlighting sections directly related to users' privacy concerns.

The first step to constructing an intelligent reasoning system would be to conduct contextual analysis of privacy policies, for comprehension, and capture it in a taxonomy, a hierarchical representation of privacy policy concepts. In other words, the taxonomy would capture the vocabulary of privacy policies. There are two main ways to capture vocabularies: manually and automatically. Manually capturing the vocabulary involves reading and knowing domain text which would involve hours of manual labour that can be costly. An easier approach would be to use NLP techniques to automatically extract keywords for taxonomy creation [START_REF] Cimiano | Text2Onto[END_REF][START_REF] Velardi | Using text processing techniques to automatically enrich a domain ontology[END_REF]. This requires fewer man hours compared to the manual methods; hence, is cheaper. Presently, no taxonomy exists for the online privacy policy domain. Hence, the ultimate objective of this research is to create and validate a taxonomy with the aid of NLP algorithms and then further by subject matter experts. This paper focuses only on the keyword and keyphrase extraction part of this larger process.

Keyword and keyphrase extraction forms the backbone of topic modelling and information retrieval systems. In this paper, NLP was used to extract keywords and keyphrases from hundreds of policies. In addition, we employed trained participants to extract keywords and phrases from a much smaller subset of policies, with the intent of comparing the manual extraction to that of the NLP. The overall goal of evaluating automatic keyword extraction algorithms was to examine which algorithm performs best against human annotators for the domain of online privacy policies. The best of these would then be chosen to become part of a larger process to enrich domain-expert curated taxonomy which would then be used to construct an ontology.

Our research confirms that whilst automatic keyword and keyphrase extraction remains a difficult task, supervised learning algorithms perform marginally better against unsupervised algorithms. Furthermore, trained annotators, collectively, can cheaply out-perform domain experts; their combined output being further used to improve the training set for the supervised learning algorithm.

The remainder of this paper is structured as follows: Sec. 2 discusses related work; Sec. 3 describes previous work on which the current research is built on; Sec. 4 presents motivation and describes methodology; Sec. 5 presents our investigation with training non-domain experts for the task of keyword and keyphrase extraction; Sec. 6 presents our investigation after working a supervised learning algorithm for the task of keyword and keyphrase extraction; Sec. 7 discusses the implication and avenues for future work; we conclude our work in Sec. 8.

Related Work

In a previous study, Wilson et al. (2016) created the OPP-115 corpus, a corpus of 115 manually annotated privacy policies with 23, 000 data practices. A data practice is roughly defined as a purpose or consequence of collecting, storing, or generating data about a user. In the study, the 10 domain experts (privacy experts, public policy experts, and legal scholars) used a custom designed webbased tool to annotate practices and assign various attributes to them for classification purposes. Each policy took an average of 72 minutes to annotate. Whilst this approach produced a high quality and nuanced data set, it is costly to expand and maintain such a knowledge base both in time and money. Automation and crowdsourcing could reduce the cost of creating and maintaining such a data set, and still maintain a reasonable amount of quality.

In the research conducted by Ramnath et al., the researchers proposed combining machine learning and crowdsourcing (for validation) to semi-automate the extraction of key privacy practices [START_REF] Ramanath | Identifying relevant text fragments to help crowdsource privacy policy annotations[END_REF]. Through their preliminary study they were able to show that non-domain experts were able to find an answer to their privacy concern relatively quickly (∼ 45s per question) when they were only shown relevant paragraphs that were mostly likely to contain an answer to the question. They also found that answers to privacy concerns were usually concentrated rather than scattered all over the policy. This is an important find because it means that if users were directed to relevant sections in the policy they would be able to address their privacy concerns relatively quickly instead of reading the entire policy. Additionally, Pan and Zinkhan have showed that when users are presented with a short and straightforward policy, they are more inclined to read it [START_REF] Pan | Exploring the impact of online privacy disclosures on consumer trust[END_REF].

In a more recent user study conducted by Wilson et al. (2016), the quality of crowdsourced answering of privacy concerns was tested against domain experts with particular emphasis on highlighted text. The researchers found that highlighting relevant text had no negative impact on accuracy of answers. They also found out that users tend not to be biased by the highlights and are still likely to read the surrounding text to gain context and answer privacy concerning questions. They also found an 80% agreement rate between the crowdsourced workers and the domain experts for the same questions [START_REF] Wilson | Crowdsourcing Annotations for Websites' Privacy Policies: Can It Really Work?[END_REF]. Similarly, Mysore Sathyendra et al. showed through their study that it was possible to highlight and extract opt-out practices from privacy policy using keywords and classification algorithms with reasonable accuracy; of the various models tested, best model used a logistic regression classification algorithm with a manually crafted feature set and achieved an F1 score of 59% [START_REF] Sathyendra | Automatic Extraction of Opt-Out Choices from Privacy Policies[END_REF].

The general drawback of crowdsourcing, especially with respect to privacy policies, is that it relies on non-expert users to read policies to provide data. Since most users are not motivated to reading policies to begin with, it would take a long time to crowdsource enough data to be useful. However, what is clear is that highlighting relevant text with appropriate keywords can still provide some feedback to the concerned users that are inclined to read shorter policies.

Background

Keyword/keyphrases extraction remains a difficult task; the state-of-the-art performance of keyword extraction algorithms hovers around 20-30% [START_REF] Kim | SemEval-2010 Task 5: Automatic Keyphrase Extraction from Scientific Articles[END_REF]. Keyword (or keyphrase) extraction has been historically used to recognize key topics and concepts in documents. This task involves identifying and ranking candidate keywords based on the relatedness to the document. Keyword extraction algorithms utilize various techniques to perform their task: statistical learning, part-of-speech (POS) tagging, lexical and syntactic feature extraction. Generally, they work in two steps:

1. Identifying candidate keywords/keyphrases from the document using heuristics. 2. Recognizing if the chosen candidate keywords/keyphrases are correct or not using supervised and unsupervised methods.

Supervised Learning vs. Unsupervised Learning

Machine learning can be divided into two broad types: supervised and unsupervised learning. The majority of the machine learning techniques involve supervised learning algorithms which rely on a tagged corpus for training a model to learn features (keywords) from the text. After sufficient training, the model is then applied on similar corpus to extract keywords. The keyword assignments made over the training data set forms the reference, also known as controlled vocabulary, and treated as classes used in a classification problem. Some examples of supervised learning algorithms include, K-nearest neighbour (k-NN) [START_REF] Yeung | A k-Nearest-Neighbour Method for Classifying Web Search Results with Data in Folksonomies[END_REF][START_REF] Tan | Neighbor-weighted K-nearest neighbor for unbalanced text corpus[END_REF], Naive Bayes (NB) [START_REF] Frank | Domain-specific keyphrase extraction[END_REF], GenEx [START_REF] Turney | Learning algorithms for keyphrase extraction[END_REF], and Support Vector Machines (SVM) [START_REF] Zhang | Keyword extraction using support vector machine[END_REF]. Since creating a tagged corpus is a very time consuming task, unsupervised learning algorithms are used which do not require any training set for the training of models. They instead rely on linguistic and statistical features of the text. The task is framed as a ranking or clustering problem.

Per Policy Keyword Extraction Fares Best

Our previous research [START_REF] Audich | Extracting keyword and keyphrase from online privacy policies[END_REF] used five unsupervised learning algorithms to extract keywords for the purpose of identifying key concepts with the goal of generating a taxonomy for the online privacy policy domain. The research was conducted in two experiments. In the first, the algorithms were evaluated over a smaller corpus where a set of manually extracted terms by the researcher was used as the baseline. Researcher's manually extracted terms are also used in Experiment I (Sec. 5) as the baseline. Second, the algorithms were evaluated over a larger corpus where the results from the best performing algorithm from the first experiment was held as the baseline. While the algorithm Term-Frequency Inverse Document Frequency (TF-IDF) achieved an F 1 -score of 27% over a small corpus (21 policies), over a large corpora (631 policies) algorithms evaluating single documents individually, such as AlchemyAPI1 and TextRank [START_REF] Mihalcea | TextRank: Bringing order into texts[END_REF], performed the best.

Results from both experiments suffered from four major types of errors. Overgeneration errors are a type of precision error where the algorithm incorrectly identifies a candidate term as a keyword because one of its sub-words appears frequently in a document or corpus. Redundancy errors are a type of precision error where the algorithm correctly identifies a keyword simultaneously identifying another keyword that is semantically similar, e.g. 'account use' and 'account usage'. Infrequency errors occur when a candidate term is not selected due to its low frequency of appearance in a document or a corpus. Evaluation errors are a type of recall error that occur when a candidate term is not identified as a keyword despite it being semantically similar to a baseline keyword. Since the unsupervised algorithms focus on the task of ranking and/or clustering based on semantic and lexical analysis, these errors are a result of language used in the privacy policies which tends to be inconsistent between policies.

The alternative to unsupervised learning algorithms are supervised learning algorithms in which a model is first trained on a set of manually extracted terms, and thus the task becomes one of classification, i.e., whether a candidate keyword should be classified as a document keyword or not. In case of privacy policies, we expect the results to improve because training a model would reduce various errors that occurred with unsupervised learning. For example, since the keywords are first being extracted manually; if the terms are infrequent, the trained model would learn this bias reducing infrequency errors. In this paper we employ a supervised learning algorithm to test our hypothesis.

Study Design

The primary objective of this experiment was to test whether a supervised learning algorithm could outperform the unsupervised learning algorithms used in our previous study. As such, choosing an effective supervised challenger was key. To compare unsupervised algorithms, a supervised learning algorithm, KEA, was investigated. KEA is an effective supervised learning algorithm utilizing a Naive Bayes algorithm for training learning models [START_REF] Witten | KEA: Practical automatic keyphrase extraction[END_REF]. It is a simple and well-known algorithm that is has been often used as a baseline throughout the literature [START_REF] Kim | Re-examining automatic keyphrase extraction approaches in scientific articles[END_REF][START_REF] Siddiqi | Keyword and Keyphrase Extraction Techniques: A Literature Review[END_REF][START_REF] Kim | Extracting keywords from multi-party live chats[END_REF][START_REF] Grineva | Extracting key terms from noisy and multitheme documents[END_REF][START_REF] Yih | Finding advertising keywords on web pages[END_REF][START_REF] Hasan | Automatic Keyphrase Extraction: A Survey of the State of the Art[END_REF].

KEA works in two phases: training and model creation, and extraction. In the training phase, candidate keyphrases are selected both from the training documents and the corpus, features (attributes) are calculated, and keyphrases determined. Candidate keyphrase selection works in three phases: text pre-processing, identifying the candidate keyphrases, and stemming and case-folding. Features that are calculated are, TF-IDF and first occurrence (the first time the term occurs in a document), which are then discretized for the machine learning scheme. Finally, the keyphrase are determined from the discretized values using the Naive Bayes technique [START_REF] Lewis | Naive (Bayes) at forty: The independence assumption in information retrieval[END_REF]. In the extraction phase, the candidate keyphrase selection is repeated on the documents to calculate feature values. Then the Naive Bayes algorithm is used along with the values calculated in the model to determine if a candidate keyphrase is a keyphrase or not.

The study was broadly broken up into two parts: Experiment I examines manual extraction by trained non-domain experts, while Experiment II compares supervised and unsupervised techniques.

Experiment I: Manual Keyword Extraction

Participants

Four participants were selected for this experiment. Since this was a preliminary study conducted to test if non-experts can be trained enough to extract important keywords, we thought that 4 participants was enough. All had a graduate level education in Computer Science with varying knowledge of online privacy and were male with a mean age of 33.65 (range 22 -60). None had a research background in privacy and they rarely read website privacy policies.

Procedure

The participants were briefed on intelligent reasoning systems and taxonomies to ensure they understood the basic concepts and how their keywords would be used. A set of criteria for manual extraction was then provided with examples to help participants select the appropriate keywords and keyphrases as illustrated in Table 1). In order to have some time limit, it was estimated that it took less than 2 hours to read and annotate 5 policies. Hence, the participants were given 2 hours to read 5 privacy policies (a subset of the 21 privacy policies used for the training model; see Table 2), and highlight terms (unigrams, bigrams, trigrams, etc.) they thought were important: concepts, themes, and terms; pertaining to the online privacy domain and as outlined in the criterion. The 5 policies selected were from different industry sectors; intended for a diverse audience; and conformed to the laws of multiple countries; they included: Google, Facebook, UEFA, Royal Bank of Canada, and Wal-Mart (including policy for California). Over a 2 hour period, each participant was presented with the privacy policies in a different order to reduce the possibility of an ordering bias.

Participants used an open source program called 'Skim'2 for annotation. A Python script was then used to extract all of the highlighted keywords and store the results in a comma separated values (CSV) file for further analysis. To ensure consistency of key term extraction across the data sets, the following post-processing steps were taken to normalize the text:

1. All of the terms were first converted to lowercase. 2. Non-printable characters (as defined by the string.printable set in Python3)

were removed; and the remaining special characters that were not caught by previous filters (*@#), as well as other ASCII based characters from the string.punctuation set in Python3 were removed. 3. Tokenized numbers were also removed as they do not tend to add value to the taxonomy e.g. '1945'. 4. The standard Porter Stemming Algorithm [START_REF] Porter | An algorithm for suffix stripping[END_REF] was used from the NLTK 3library to consolidate inflected word forms to their root. 5. Finally, duplicates were removed from the resulting sets.

Results

First, the data collected from all of the participants were compared to the data set generated by the primary researcher. The results are shown in Table 3.

It must be noted that participant 3 only completed 3 of the 5 policies because he found reading some policies quite challenging and hence taking longer to read. He also reported to initially having trouble understanding the task. Despite this, participant 3 was not dropped because we were mostly interested in the quality of the keywords rather than completion of task specifically. Furthermore, our analysis is mostly based on individual work, and despite the third participant failing to complete the task we wanted to highlight that they were still able to achieve results about half as good as the researcher.

In general, participants reported that policies were repetitive and often vaguely described their intent with regard to collecting personal information. When asked to state which privacy policy was most clear and readable, Facebook was described as the most transparent with UEFA being the least. The highest F 1 -score was 59% with a mean of 51.75%.

In order to test the collective efficacy of the annotations, the researcher's data set was compared with the combined data set of all of the participants. The results are reported in Table 4. Finally, all five data sets were compared to each other by holding one of the data set as the baseline and comparing it with the rest. The results are reported in Table 5. The mean of all of the values was 52.1%, which agreed with the previous analysis in Table 4. This was significantly higher than the 20 -30% performance of most state-of-the-art keyword extraction algorithms. For this experiment, a corpus of 631 privacy policies as developed by the Data Management and Privacy Governance Lab at the University of Guelph was used to evaluate the supervised learning algorithm-KEA. This is the same corpus that was used to evaluate unsupervised learning algorithms in the previous paper.

Part I

In the first part of the experiment, a set of 21 policies were used for manual extraction of keywords which in turn were used to train the learning model for KEA. The 21 policies were qualitatively determined to ensure diversity. They were selected based on the their: length, transparency, comprehension (level of difficulty), intended geographic audience (U.S., E.U, Canada), industry sectors (healthcare, e-commerce, etc.), and the most visited websites 4 . A breakdown of these policies is summarized in Table 2. An overview of Experiment II Part I is shown in Fig. 1. Results. Once a model was trained over the manually extracted set of keywords, KEA was then run over the entire corpus. Results from the algorithm were then compared with the results from unsupervised algorithms (see Table 6). Initial results showed that the supervised algorithm performed better than the unsupervised ones but not significantly.

Part II

To demonstrate that KEA performs well over smaller training sets, and better when results across multiple annotators are combined, in the second part of the experiment, the model was trained on the annotation for only the 5 policies chosen for Experiment I (Sec. 5). The trained model was then tasked with extracting keywords and keyphrases from the rest of the corpora (626 policies). This was done for all the participants and the results were compared against each other; reported in Table 7. Since the training set only contained 5 privacy policies with roughly half of the terms present amongst all of the participants, the training set did not contain enough variance to train distinctly different models. The obvious exception being participant 3 who only annotated 3 policies.

Once again, participants' data sets were combined and compared against the researcher's data set of generated keywords to see if the accuracy rose with more number of annotators; the results are presented in Table 8. The combined data set achieves a score of F 1 -score of 71%. One reason why the scores are different for the combined data sets (Table 4) is due to the smaller training set for the third participant. With less labelled data to train with, the training algorithm could overgeneralize what is not considered a keyword and hence ignore a large number of candidate keywords. The mean F 1 -score of 52% in Experiment I (Table 3) demonstrates an important aspect of keyword extraction with privacy policies-keyword extraction with privacy policies is a hard task. While 52% is a score better than the 20 -30% performance of most state-of-the-art keyword extraction algorithms, a qualitative analysis of the annotated keywords/keyphrases by the participants suggests that despite having a concrete set of rules, examples, and training, participant's understanding of the technical terms and the text can still result in a diverse and non-overlapping set of terms. What one participant considers an important concept is not shared by a peer. This was mostly true for the more ambiguous and less technical parts of the policies, while technical details were easily picked up by all participants. Since all of the participants had a background in Computer Science, technical details would be easy to comprehend and more transparent to them. However, this might not be the case if the participant had no technical background; then, everything would have been equally less transparent. When written ambiguously privacy policies are difficult to comprehend. In Part I of Experiment II it was found that the supervised learning algorithm improved the F 1 -score of keywords being extracted. This is important for two primary reasons: quality and cost. One of the most time consuming tasks involved in generating a taxonomy is capturing major important themes and concepts of the target domain. This is where keyword extraction plays an important role in reducing the time taken to capture all of the themes and concepts. In this case, instead of reading a large number of privacy policies to identify all of the important themes and concepts, supervised learning promises to be a viable alternative. The generated keywords act as candidate terms that can be used to enrich a taxonomy, thus reducing the cost and time of reading a large number of privacy policies as well as improving the quality of the taxonomy by including terms that might have been covered in text that might not have been read due to resource restrictions.

Furthermore, the diversity of keywords/keyphrases found, in Experiment I, between participants can be used to improve the training data for supervised learning. The training data in Experiment II Part I was generated by a single researcher, it might be useful for another researcher or domain expert to read a set of non-overlapping policies and generate another set of training data. This would not only validate the current training data but also create more labelled data to train the model on. It would also prove helpful to review the candidate terms generated by participants in the first experiment and enrich the present training data, i.e., carefully merge all of the data sets to create a more comprehensive data set that captures all relevant keywords/keyphrases including the ones that might have been missed by individuals within the group.

Part II of the second experiment showed that if non-domain experts are given sufficient training, a supervised learning algorithm trained with their labelled data set could result in a training model that is able to extract most of the keywords and keyphrases that are being extracted with the help of training model, and built with labelled data from domain experts. Hence, it is possible to reduce cost and train non-experts and extract keywords and keyphrases with reasonable success.

Currently, privacy policies are heterogeneous as there are no laws/guidelines that mandate a certain structure, concepts, or terminology. This makes finding, identifying and understanding relevant information a time consuming task. By utilizing an intelligent reasoning system and mapping important concepts, ideas, and themes our work helps to identify important sections in an unstructured privacy policy; thus, resulting in less time needed to find important information in policies improving transparency and making polices more usable. It could further be used to introduce structure in future policies.

Conclusion

In this paper, we extended our previous work on keyword and keyphrase extraction over the domain of online privacy policies. In Experiment I, the difficulty of extracting keywords from privacy policies was demonstrated and the challenges associated with this task was discussed. In Experiment II we applied a supervised algorithm for keyword extraction, and demonstrated its superior performance over unsupervised algorithms applied to the same corpus of 631 online privacy policies. Our results confirm that using natural language processing techniques for keyword and keyphrase retrieval from privacy policies remains a challenging task.

Our preliminary results will guide further research in the field of online privacy and machine learning, and making policies more transparent and usable. We intend to improve our training set by having other domain experts (researchers and lawyers) identify key concepts, ideas, and themes in a non-overlapping set of privacy policies. In addition, it will be useful to compare how trained supervised algorithms perform against domain experts. Our research forms the first step in creating a context aware system for real-time privacy policy evaluation.

Fig. 1 .

 1 Fig. 1. Experiment II Part I: Testing KEA against manual extraction and AlchemyAPI.

Table 1 .

 1 Criteria for manually extracting key terms.

	Concept	Examples
	Legal terms	Online Privacy Protection Act, non-
		disclosure agreement
	Legal organizations (government, regula-	federal trade commission
	tory, commercial, and computing organiza-
	tions)
	Acronyms of legal organizations and acts FTC, COPPA
	Legal entities that can be used to define an	personal information, address, account id,
	organization or an individual	internet protocol address
	Data sharing	3rd party cookies, aggregate information,
		google analytics
	Hosting	backup storage, servers
	Web & tech related terms	ad data, cookies, analytics, tracking cookies
	Legal actions and legal processes	tracking, surveillance
	Mobile privacy	geo-location, device identification

Table 2 .

 2 Breakdown of the 21 privacy policy corpus for Experiment I.

	Domain	No. of websites selected
	Healthcare	1
	Insurance	2
	Banking & Financial	5
	E-Commerce	3
	File Sharing	1
	Search Engines	2
	Social Networking	3
	EU Specific	3
	Cloud Hosting	1
	Total	21

Table 3 .

 3 Results from manual keyword extraction by participants.

		Researcher 1	2	3	4
	Terms	560	581 650 353 504
	Precision		49% 55% 59% 56%
	Recall		51% 64% 37% 51%
	F1-score		50% 59% 45% 53%
	JSC		0.67 0.58 0.71 0.64

Table 4 .

 4 Comparing performance of manual extraction primary researcher vs. combined data set generated by participants.

		Researcher Participants
	Terms	560	1038
	Precision		40%
	Recall		75%
	F1-score		52%
	JSC		0.65

Table 5 .

 5 Comparing F1-scores between participants' and researcher's data sets.

		Researcher P1 P2 P3 P4
	Researcher	-	50% 59% 45% 53%
	P1	50%	-56% 46% 52%
	P2	59%	56% -50% 58%
	P3	45%	46% 50% -55%
	P4	53%	52% 58% 55% -
	6 Experiment II: Supervised Learning

Table 6 .

 6 Comparing performance of unsupervised vs. supervised learning algorithms for keyword extraction

		AlchemyAPI KEA
	Terms	12635	10798
	Precision	3%	4%
	Recall	44%	47%
	F1-score	5%	7%
	JSC	0.97	0.97

Table 7 .

 7 Comparing F1-scores between participants' and researcher's generated data sets from KEA.

		Researcher P1 P2 P3 P4
	Researcher	-	62% 70% 64% 72%
	P1	62%	-71% 75% 72%
	P2	70%	71% -67% 79%
	P3	64%	75% 67% -67%
	P4	72%	72% 79% 67% -

Table 8 .

 8 Comparing performance of keyword extraction from KEA based on primary researcher vs. combined data set generated by participants.

		Researcher Participants
	Terms	30261	49377
	Precision		57%
	Recall		93%
	F1-score		71%
	JSC		0.45

http://www.alchemyapi.com/products/alchemylanguage/keyword-extraction

https://sourceforge.net/projects/skim-app/

http://www.nltk.org/

As listed under: https://en.wikipedia.org/wiki/List_of_most_popular_ websites