Yacine Gaci
email: yacine.gaci@univ-lyon1.fr

Boualem Benatallah
email: boualem.benatallah@gmail.com

Fabio Casati
email: fabio.casati@gmail.com

Khalid Benabdeslem
email: khalid.benabdeslem@univ-lyon1.fr

Conceptual Similarity for Subjective Tags

Tagging in the context of online resources is a fundamental addition to search systems. Tags assist with the indexing, management, and retrieval of online products and services to answer complex user queries. Traditional methods of matching user queries with tags either rely on cosine similarity, or employ semantic similarity models that fail to recognize conceptual connections between tags, e.g. ambiance and music. In this work, we focus on subjective tags which characterize subjective aspects of a product or service. We propose conceptual similarity to leverage conceptual awareness when assessing similarity between tags. We also provide a simple cost-effective pipeline to automatically generate data in order to train the conceptual similarity model. We show that our pipeline generates high-quality datasets, and evaluate the similarity model both systematically and on a downstream application. Experiments show that conceptual similarity outperforms existing work when using subjective tags.

Introduction

As products and services proliferated the Internet in recent years, tagging came into prominence to facilitate the consumption of online information [START_REF] Smith | Tagging: people-powered metadata for the social web[END_REF]. Tagging is the practice of assigning labels and keywords to online resources. It plays a pivotal role in the indexing, management and retrieval of factual information. On the other hand, recent years have witnessed a major shift in people's expectations when searching online [START_REF] Li | Subjective databases[END_REF]. Beside the factual data such as a restaurant's cuisine type or a camera's resolution, the search trend evolved to be more experiential [START_REF] Li | Subjective databases[END_REF]. Common search queries include attributes such as delicious food for restaurants or long-lasting battery for cameras. Previous work [START_REF] Li | Subjective databases[END_REF][START_REF] Gaci | Subjectivity aware conversational search services[END_REF] called this new set of attributes as subjective tags because they are short phrases that hint towards the subjective quality of products and services.

Subjective tags are particularly useful in enhancing online experiential search. In this context, users who are seeking subjective experiences, include sets of tags they care about in their queries, and it is the search system's responsibility to fetch products and/or services that have been previously described with matching tags. Deciding whether two given subjective tags match or not implies using a similarity measure, for which cosine similarity remains a convenient, yet arbitrary default [START_REF] Zhelezniak | Correlations between word vector sets[END_REF][START_REF] Li | Subjective databases[END_REF][START_REF] Chee | Searchlens: Composing and capturing complex user interests for exploratory search[END_REF]. Recent search systems such as OpineDB [START_REF] Li | Subjective databases[END_REF] or SearchLens [START_REF] Chee | Searchlens: Composing and capturing complex user interests for exploratory search[END_REF] rely mostly on cosine similarity when it comes to comparing tag-like short phrases, since it is easy to use and provides simple geometric interpretations [START_REF] Zhelezniak | Correlations between word vector sets[END_REF]. However, recent studies [START_REF] May | On measuring social biases in sentence encoders[END_REF][START_REF] Zhou | Problems with cosine as a measure of embedding similarity for high frequency words[END_REF] argue that this interpretability becomes fogged when dealing with sentences or phrases, and cosine similarity suffers from severe limitations when used to compare multi-word textual inputs.

A lot of research has been directed toward proposing supervised methods for textual similarity, spanning a diverse set of paradigms, e.g. Siamese networks [START_REF] Bromley | Signature verification using a "siamese" time delay neural network[END_REF][START_REF] Ranasinghe | Semantic textual similarity with siamese neural networks[END_REF], Aggregation-Matching models [START_REF] Wang | A compareaggregate model for matching text sequences[END_REF]Wang et al., 2016[START_REF] Wang | Bilateral multi-perspective matching for natural language sentences[END_REF], or the recent cross-sentence attention paradigm which was made possible by the advent of the transformer architecture [START_REF] Vaswani | Attention is all you need[END_REF]. Although these models work fairly well on syntactically-correct sentences (Bethard et al., 2017), they lack effectiveness when used with shorter-spanned phrases such as subjective tags. A reason behind this is that subjective tags do not share the same structure of full sentences and hence require different treatment. As will be discussed later in this paper, our experiments confirm this limitation. A second drawback is that current similarity models are not explicitly trained to recognize conceptual similarities between the compared textual entities (e.g., meal and pizza share the concept of food; or background music and lighting share the concept of ambiance). Therefore, all conceptual reasoning is disregarded. In this work, we compel our own similarity model to encode more conceptual relationships as provided by a human (whom we call the designer) and further expanded by popular knowledge bases such as WordNet [START_REF] Fellbaum | Wordnet. The encyclopedia of applied linguistics[END_REF] or ConceptNet [START_REF] Speer | Conceptnet 5.5: An open multilingual graph of general knowledge[END_REF].

To illustrate the importance of capturing conceptual similarities between subjective tags, suppose a user searches for a restaurant serving delicious meals. A search system should be able to suggest a restaurant which has been tagged with tasty chicken wings among its search results, because meal and chicken wings share the same concept (that of food) even though meal and chicken wings are not semantically similar. As a result, traditional semantic similarity models (Bethard et al., 2017;[START_REF] Li | Subjective databases[END_REF][START_REF] Ranasinghe | Semantic textual similarity with siamese neural networks[END_REF] usually fail to meet this expectation and provide low similarity scores for the tags in the example. The same reasoning applies to other subjective tags, like high-autonomy camera and long-lasting battery, or romantic ambiance and low-beat music bar.

Aiming to solve the aforementioned drawbacks, we propose a new similarity model that focuses on learning and then using conceptual relationships as reflected in the training data. Given the new nature of subjective tags [START_REF] Li | Subjective databases[END_REF][START_REF] Gaci | Subjectivity aware conversational search services[END_REF], we are not aware of the existence of datasets that suit our needs. Besides, manually annotating data is expensive, and extending to other application domains (e.g. from restaurants to electronics) usually necessitates re-annotating from scratch. Therefore, the main contribution of this paper is a pipeline to automatically generate large synthetic datasets for the conceptual similarity task. First, we prompt the dataset designer to provide seed words for the concepts she needs her conceptual similarity model to learn about. Second, we exploit the simple structure of subjective tags [START_REF] Gaci | Subjectivity aware conversational search services[END_REF] to expand the seeds with conceptually related terms using knowledge bases, or the implicit knowledge encoded in existing language models to automatically generate large training data.

Our second contribution is the similarity model itself. Capitalizing on the latest advances in se-mantic similarity research [START_REF] Ranasinghe | Semantic textual similarity with siamese neural networks[END_REF][START_REF] Wang | Bilateral multi-perspective matching for natural language sentences[END_REF][START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], we propose a new similarity model by combining insights from aggregation-matching and cross-sentence attention paradigms. We show that conceptual similarity is better than cosine similarity with a margin of 17.42% in terms of Pearson correlation, or BERTbased similarity models through systematic evaluations. We also plug different similarity models into a tag-based search system and show that conceptual similarity outperforms them all. Also, we evaluate the quality of the automatically generated dataset through various experiments. We release our code and data in GitHub1 .

Related Work

Synthetic Dataset Generation

Acquiring training data is increasingly the largest and most pressing bottleneck in deploying machine learning systems [START_REF] Ratner | Snorkel: Rapid training data creation with weak supervision[END_REF]. The traditional way of doing so calls a team of experts to manually create and then label the data, incurring tremendous costs. Crowdsourcing alleviates part of this burden by proposing to a group of individuals of varying knowledge and expertise, the undertaking of the labeling task [START_REF] Brabham | Crowdsourcing[END_REF][START_REF] Howe | The rise of crowdsourcing[END_REF]. However, crowdsourcing runs the risk of corrupting the precision of the gold labels, and may inflict noise in the labeling process, especially when uneducated, careless or malicious workers are involved. A recent trend for acquiring training data is devising methods to automatically create, generate and label these critical building blocks of supervised learning systems with little effort [START_REF] Ratner | Data programming: Creating large training sets, quickly[END_REF][START_REF] Ratner | Snorkel: Rapid training data creation with weak supervision[END_REF][START_REF] Varma | Snuba: automating weak supervision to label training data[END_REF]. When one speaks of generating data, two problems are implicitly addressed: (1) generation of features (i.e. unlabeled raw data), and/or (2) generation of gold labels (i.e. automatic labeling).

First, we discuss the generation of features, for which two techniques are mainly used: templatebased generation [START_REF] Dev | On measuring and mitigating biased inferences of word embeddings[END_REF][START_REF] Nadeem | Stereoset: Measuring stereotypical bias in pretrained language models[END_REF][START_REF] Tulio Ribeiro | Beyond accuracy: Behavioral testing of nlp models with checklist[END_REF] and data augmentation [START_REF] Zhao | Gender bias in coreference resolution: Evaluation and debiasing methods[END_REF][START_REF] Zmigrod | Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology[END_REF][START_REF] Taylor | Improving deep learning using generic data augmentation[END_REF][START_REF] Nie | Adversarial nli: A new benchmark for natural language understanding[END_REF][START_REF] Kaushik | Learning the difference that makes a difference with counterfactually-augmented data[END_REF]. In template-based generation, a set of tokens iteratively replaces the placeholders in templates, creating a separate example each time. [START_REF] Dev | On measuring and mitigating biased inferences of word embeddings[END_REF] provide templates such as "The [PLACE-HOLDER] is a doctor", and insert words like man, woman, muslim, christian to create different examples to study social biases and stereotypes. In the same spirit, [START_REF] Nadeem | Stereoset: Measuring stereotypical bias in pretrained language models[END_REF] construct an evaluation dataset of biases through the use of templates and crowdsourcing, whereas Ribeiro et al.

(2020) designed a framework to test NLP systems where users construct their own test benchmarks via the use of templates. On the other hand, data augmentation techniques expect an already available set of data, that they augment and expand to create larger sets. This is usually achieved by searching for similar inputs in the feature space, applying small perturbations to the existing data without changing the labels [START_REF] Kaushik | Learning the difference that makes a difference with counterfactually-augmented data[END_REF], or through seed expansion techniques [START_REF] Fast | Empath: Understanding topic signals in large-scale text[END_REF][START_REF] Li | Subjective databases[END_REF][START_REF] Huang | Guiding corpusbased set expansion by auxiliary sets generation and co-expansion[END_REF] via similarity in word embeddings or with knowledge bases.

Our own data generation is a mix of both techniques. While it is fundamentally a seed expansion method where aspect and opinion terms that we use to express subjective tags are expanded into conceptually related terms, it also derives from template-based generation since we use the template "<opinion> <aspect>" (as in delicious food or romantic ambiance) to construct subjective tags. The closest work to ours in terms of seed expansion is Empath [START_REF] Fast | Empath: Understanding topic signals in large-scale text[END_REF] for studying topic signals in text. In Empath, a topic is defined by a set of seeds that are later expanded by either using word embeddings or crowdsourcing, to enrich each topic category. In contrast, we use the expansions to build sufficiently large labeled datasets. Moreover, we propose five different expansion techniques to increase the diversity of generated subjective tags.

The second problem in automatic data generation is generating the ground truth labels. Data programming [START_REF] Ratner | Data programming: Creating large training sets, quickly[END_REF] is a recent paradigm that enables the programmatic creation of largescale training sets in which different weak supervision sources (e.g. heuristics, knowledge bases, crowdsourcing) are combined. In Snorkel [START_REF] Ratner | Snorkel: Rapid training data creation with weak supervision[END_REF] and Snuba [START_REF] Varma | Snuba: automating weak supervision to label training data[END_REF], combination is done with a generative model that takes into consideration several properties of the weak classifiers including accuracy, coverage, and inter-correlations. Our work is different in two main aspects. First, Snorkel and Snuba are general frameworks that present general guidelines aiming to build labeling functions, whereas our method is much more specific, and focuses on similarity for subjective tags. Second, in this work, we generate and label training sets at the same time, in contrast to Snorkel whose purpose is to assign labels to already existing unlabeled data.

Textual Similarity

Apart from cosine similarity, we identify several similarity paradigms in the literature: (1) Siamese networks [START_REF] Bromley | Signature verification using a "siamese" time delay neural network[END_REF][START_REF] Ranasinghe | Semantic textual similarity with siamese neural networks[END_REF] where the same encoder is used to project inputs into the same embedding space. Then, the similarity decision is made based on the vector representations alone. (2) Aggregation-matching paradigm [START_REF] Wang | A compareaggregate model for matching text sequences[END_REF]Wang et al., 2016[START_REF] Wang | Bilateral multi-perspective matching for natural language sentences[END_REF] which adds explicit matchings between the representations of inputs, before aggregating them and computing similarity. (3) Crosssentence attention paradigm which is enabled by finetuning transformer models such as BERT on a similarity task [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Peinelt | tbert: Topic models and bert joining forces for semantic similarity detection[END_REF]. ( 4) Combining several weak similarity models such as simple neural networks, tree-based and/or probabilistic models through an ensemble (Bethard et al., 2017;[START_REF] Tian | Ecnu at semeval-2017 task 1: Leverage kernel-based traditional nlp features and neural networks to build a universal model for multilingual and cross-lingual semantic textual similarity[END_REF][START_REF] Lair | User-in-the-loop adaptive intent detection for instructable digital assistant[END_REF]. However, all these works focused solely on semantic similarity between syntactically correct sentences, whereas we focus on conceptual similarity between tag-like short phrases, similar to [START_REF] Mohd Anuar | Semantic retrieval of trademarks based on conceptual similarity[END_REF]; [START_REF] Zhu | Computing semantic similarity of concepts in knowledge[END_REF]. In contrast, we use knowledge graphs to generate data and train a supervised model. More details about our similarity model are provided in Section 4.

Pipeline to Generate Training Datasets

Borrowing from the Aspect-Based Sentiment Analysis literature [START_REF] Liu | Sentiment analysis and opinion mining[END_REF][START_REF] Gaci | Subjectivity aware conversational search services[END_REF], we define a subjective tag as the concatenation of an aspect term with an opinion term. The aspect term designates the component or the feature being described and the opinion term characterizes this feature. For example, delicious food is a subjective tag wherein food is the aspect while delicious is the opinion. This definition is sufficiently expressive to allow a wide range of subjective tags such as romantic ambiance, clean hotel rooms, long-lasting battery, great camera or amiable dentist.

Specific to this work, we define a concept as a set of aspect terms conceptually related to each other. For example, the concept of food can be described with the following set of terms: {food, plates, dishes, pizza, chicken wings, meal, pasta} while the concept of ambiance can be defined with {ambiance, atmosphere, lighting, background music, dance floor}. The goal of conceptual similarity is to consider the aspects belonging to the same concept as similar when described with similar opinions.

We cast conceptual similarity as a binary classification problem, where the positive label denotes similarity. These specifications enable automatic generation of high-quality labeled datasets for conceptual similarity of subjective tags, with minimal costs. To do so, the dataset designer provides a list of concepts. We then leverage seed expansion techniques to generate the dataset, through the pipeline illustrated in Figure 1. In the following, we describe each step of the pipeline in detail.

Providing Concept Seed Words

The first step in the pipeline is to provide seed words for the concepts that the dataset designer wants to take into consideration. For each concept i, the designer provides a list of aspect seed words A i , and m i lists of opinion seed words O j i where j ∈ {1...m i }; m i depends on the concept and the level of granularity the dataset designer aims to reach. For the sake of illustration, say that the designer wants to include the concept of food with three classes of opinions (delicious, horrible, healthy).

She may provide the following:

A i = {"food", "dish", "lunch", "pizza", "snack"} O 1 i = {"good", "delicious", "excellent"} O 2 i = {"bad", "horrible", "not seasoned"} O 3 i = {"healthy", "organic", "high quality"} A i lists aspect terms related to the concept of food. Each of O j i lists some opinion terms of the same nature, but different from one set to another. In the example above, O 1 i describes tasty food, O 2 i characterizes bad food, and O 3 i deals with healthy food. In this particular scenario, conceptual similarity trained on a dataset to be generated from these seed words considers the tags "good food" and "healthy food" as dissimilar because the terms good and healthy belong to different opinion sets.

If the dataset designer needs a more granular similarity model (like spicy food described as its own class), she only has to add another set with seed words depicting spiciness. Following these guidelines, the designer can express a wide range of concepts such as price, service, hygiene, and in other domains too (hotels, electronics, books, etc.)

Seed Word Expansion

We propose five different techniques to expand the set of seed words given by the dataset designer. We illustrate these techniques in Figure 2 and describe them in the following:

WordNet Expansion. For every seed, we collect its corresponding synsets from WordNet [START_REF] Fellbaum | Wordnet. The encyclopedia of applied linguistics[END_REF]. Then, for every synset, we retrieve its hyponyms, hypernyms, meronyms and sister terms as illustrated in Figure 2(a). We control the number of expansions through the use of hyperparameters such as the maximum number of synsets to include, and different booleans each telling whether we take hyponyms, meronyms, etc. respectively.

ConceptNet Expansion. For every seed, we obtain its is-a (i.e. parent concepts) and type-of (child concepts) relations. For example, meat and food are parent concepts for the word of interest, i.e. chicken. We also retrieve other children of the parent concepts as is shown in Figure 2(b). We control ConceptNet expansion with three hyperparameters: capacity which is the maximum number of relations to consider; minimum weight which specifies the relevance of the relation (high weights in ConceptNet [START_REF] Speer | Conceptnet 5.5: An open multilingual graph of general knowledge[END_REF] correspond to a strong relation); and a boolean specifying whether to include children of parent concepts into the expansion.

Word Embedding Expansion. The goal is to find the top_k words in the vocabulary that minimize the total distance between them and seed terms. Taking the example in Figure 2(c), pasta is less distant from all the seeds than morning is, thus pasta constitutes a better expansion. The parameters of this technique is the number of expansions Language Generation Expansion. This method plugs seed words into a template such as "These concepts are related: <seed_1>, <seed_2>, ... <seed_n>, and ", then asks an autoregressive language model to generate a continuation for this sentence. We then take the top_k words having the highest probabilities to be correct continuations. The hyperparameters are: the language model (e.g. GPT2, T5), the number of generations, and the maximum length of each generated expansion.

Masked Language Modeling Expansion. Similar to the previous expansion technique, we use a masked language model [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], where the template takes the following form: "<seed_1>, <seed_2>, ... <seed_n> and [MASK] are all related concepts." The masked language model produces, for every word in the vocabulary, its likelihood to replace the mask. So terms having the same concept as the seeds have higher probabilities. The parameters of this method are the number of top_k terms to take, and the masked language model under use, e.g. BERT, Albert... For every expansion technique, we can have as many expanders as there are parameter configurations. For example, two word embedding expanders, one based on Word2vec while the other on GloVe, are two different expanders. Or one that uses an euclidean distance while the other uses cosine similarity are also different expanders. We give the full list of parameter configurations we used for every expansion method in our experiments in Section A.2. For a new word to be considered as a correct expansion, we require that at least a sufficient number of expanders suggest the word. We specify this with min_consensus_rate which defines how many expanders need to produce the word in order to include it in the final expansions.

Random Sampling

We randomly choose an aspect term from one of the expanded aspect sets, and an opinion term from one of its associated opinion sets. These two terms are concatenated to form a subjective tag. For example, we may sample the aspect term waiters and the opinion term nice to form the tag "nice waiters". We repeat this process to construct as many subjective tags as the dataset designer needs.

Filtering

Random sampling from automatically generated sets of terms may lead to arbitrary tags. For instance, it may construct tags such as "helpful duty". 2 We eliminate those tags by using a language model which assigns likelihoods to sen-tences so that semantically sound sentences are given high likelihoods and low quality sentences get low likelihoods. We use GPT2 language model [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] by feeding it with subjective tags formatted according to this template: "the aspect is opinion". GPT2 should assign low probabilities to sentences such as "the duty is helpful", and high probabilities to sentences such as "the service is helpful" or "the waitstaff is agreeable". We manually select the probability threshold above which sentences make sense, and keep the generated tags that score above that threshold.

Pairing and Labeling

We randomly sample two subjective tags t 1 and t 2 from the filtered list. If the aspect and opinion terms of t 1 and t 2 have been sampled from the same sets, the tags are considered similar (label is 1). In all other cases, the label is 0. To avoid class imbalance in the dataset, the dataset designer provides the minimal ratio of positive examples. We enforce this constraint by deliberately sampling similar tags from the same aspect and opinion sets.

Figure 1 summarizes our dataset generation pipeline with an example. This algorithm allows us to create high-quality training datasets with minimal effort. It can also be adapted to any domain. In Section 5.2, we evaluate the quality of datasets generated with this pipeline.

Conceptual Similarity Model

In this section, we present our approach to compute conceptual similarity for a pair of subjective tags. Following guidelines from the aggregationmatching paradigm [START_REF] Wang | A compareaggregate model for matching text sequences[END_REF], our model encodes explicit interactions between tags, e.g. whether the tags correspond to the same concept; whether they use the same opinions but with different aspects; whether the choice of words in the tags is similar but the tags themselves are not. To this end, we propose a novel bilateral matching model which automatically encodes such interactions and relationships before making a similarity decision. Given two subjective tags t 1 and t 2 , this model estimates their similarity by computing their probability of being perfectly similar P (sim = 1|t 1 , t 2 ). Figure 3 illustrates the different layers of this model.

We begin by feeding t 1 and t 2 into BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]. This serves two purposes: First, we get word embeddings for each word in the tags; second, we have a CLS vector that captures the relationship between t 1 and t 2 as a vector. Given BERT embeddings [u 1 , ..., u m ] and [v 1 , ..., v n ], we utilize mean pooling to obtain fixed-sized embeddings for each tag (u all and v all ). The next layer in the network matches each word embedding of one tag with all the word embeddings of the other tag. The matching is done in two directions (hence the bilateral aspect): (1) We match each u i with v all to compare each word u i in t 1 with all the words in t 2 , and encode their relationship. (2) We match each v i with u all to do the same in the reverse direction.

The matching function we use is the elementwise multiplication which has long been used in the NLP community as a proxy for similarity. Thus, we use it to match word embeddings of t 1 and t 2 . After the matching layer, we aggregate

[u ′ 1 , ..., u ′ m ] and [v ′ 1 , ..., v ′ n ]
to obtain fixed-length vectors for each tag via Bidirectional LSTM (BiLSTM) layers [START_REF] Hochreiter | Long short-term memory[END_REF], taking the last hidden states as tag embeddings u and v. At this step, we have encoded the relationship between t 1 and t 2 using two different paradigms: (1) aggregation-matching through the use of elementwise multiplication for matching and BiLSTM for aggregation (vectors u and v), and (2) the crosssentence attention paradigm through CLS vector, because BERT uses self-attention [START_REF] Vaswani | Attention is all you need[END_REF] to compute its vectors. We concatenate u, v and CLS and feed it to a classification head (FFNN layer) to estimate similarity.

Experiments

We use Restaurants as the test domain. We consider nine concepts that we use to automatically generate the training dataset: Food, Service, Price, Atmosphere, Location, Cleaning, Environment, Menu and Parking. Each concept consists of one set of aspect terms, and two to three sets of different opinion terms. The choice of concepts, and seed words for aspects and opinions was inspired by previous work [START_REF] Luiz | Choosing a restaurant: Important attributes and related features of a consumer's decision making process[END_REF] who conducted surveys and qualitative experiments on many restaurant-seeking participants, and identified the most important factors taken into account by these same participants in their decision-making process for choosing a restaurant. The full list of concepts and their seeds is in Section A.3, while the hyperparameter details for the similarity model are in Section A.1. In the following, we first compare conceptual similarity to various baselines. Next, we evaluate the quality of the automatically generated dataset. Finally, we assess the practical value of conceptual similarity by measuring its impact on a downstream search system proposed by [START_REF] Gaci | Subjectivity aware conversational search services[END_REF] that uses subjective tags.

Evaluating Conceptual Similarity

Existing similarity benchmarks provide similarity ground truth for syntactically correct sentences (Bethard et al., 2017). Hence, we cannot use them given that subjective tags are short phrases which do not draw from the same syntactically-complete sentence distribution. To the best of our knowledge, no benchmark for subjective tags exists. For this reason, we create our own test set by automatically extracting tags from Yelp's restaurant online reviews 3 using the tag extractor of SACCS [START_REF] Gaci | Subjectivity aware conversational search services[END_REF]. Given a snippet of text, SACCS extracts subjective tags as concatenations of aspects and opinions. We then map these extracted tags randomly into pairs. We select 500 such pairs and ask three participants to assign a similarity score between 0 and 5 for each pair of subjective tags. We then normalize the similarity scores to squash them into the unit range before taking the mean across the participants. As in standard similarity evaluations, we use three metrics: Pearson and Spearman correlation, and Mean Absolute Error (MAE). We compare our conceptual similarity model to several baselines: A Siamese network [START_REF] Ranasinghe | Semantic textual similarity with siamese neural networks[END_REF] and a random forest classifier with hand-crafted features [START_REF] Tian | Ecnu at semeval-2017 task 1: Leverage kernel-based traditional nlp features and neural networks to build a universal model for multilingual and cross-lingual semantic textual similarity[END_REF], both trained on the same dataset we use to train our own model. Also, owing to the universality of cosine similarity, we compare against it both with Paragram embeddings [START_REF] Wieting | Towards universal paraphrastic sentence embeddings[END_REF] and on BERT embeddings with different pooling methods, MEAN and CLS as in [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]; [START_REF] Li | Subjective databases[END_REF]. Finally, we train a BERT-based model that we augment with a classification head (BERT Classif) and finetune on the same training data we used to train our conceptual similarity to make it more competitive. Table 1 summarises the results.

We can see that conceptual similarity outperforms cosine similarity by a large margin (0.1742 points in Pearson correlation). This demonstrates that cosine should no longer be perceived as the default when it comes to measuring similarity for subjective tags. We also show that BERT alone cannot cater for a task as ambiguous as similarity for subjective tags, even when finetuned on the same training set that we use. This sheds light on the necessity to design custom models especially tailored for tag similarity. We argue that the effectiveness of our method stems from its ability to match different words of subjective tags using both attention and element-wise multiplication.

Existing information retrieval and tag-based search systems like [START_REF] Li | Subjective databases[END_REF][START_REF] Chee | Searchlens: Composing and capturing complex user interests for exploratory search[END_REF] blindly trust cosine similarity or a finetuned BERT without investigating their implications on the overall system performance. Our work highlights the limitations regarding main stream text similarity techniques for subjective tags and short phrases, as it gives guidelines as to how to design robust similarity models.

Evaluating the Quality of Training Data

We measure the quality of the automatically generated training dataset by injecting artificial noise in the data and checking whether it degrades in quality [START_REF] Jassar | Impact of data quality on predictive accuracy of anfis based soft sensor models[END_REF]. We define noise in this context as swapping the labels in the training set. For example, if the original line in the dataset was {t 1 , t 2 , 1}, the new noisy line would be {t 1 , t 2 , 0} and vice versa. We perturb fixed percentages of the training data ( 5%, 10%, 25% and 50%) and retrain the similarity model each time. The rationale of this experiment is that the introduction of noise should degrade the quality of training. In this spirit, if the similarity model trained on noisy data is of comparable accuracy to the one trained on the original unperturbed data, we argue that the original data was merely noise. On the other hand, if introducing noise degrades the performance of the similarity model, one can assume that the original data was of good quality. Table 2 shows the similarity correlations with human-defined scores as described in Section 5.1. We observe that instilling noise drops the accuracy of conceptual similarity. This reflects that the original unperturbed dataset is of high quality.

Experiments on a downstream System

In the following, we demonstrate the effectiveness of conceptual similarity when plugged into a downstream search application [START_REF] Gaci | Subjectivity aware conversational search services[END_REF]. We give a brief overview of the application, describe the baselines, benchmarks and evaluation metrics. System overview. SACCS [START_REF] Gaci | Subjectivity aware conversational search services[END_REF]) is a subjectivity-aware system to search for restaurants online. From their reviews, SACCS automatically extracts subjective attributes of restaurants offline in the form of subjective tags. Then, when users provide their search queries, they can include subjective tags as search filters. SACCS uses an underlying similarity model to compare between user-provided tags and those describing each restau- Baselines. We replace the similarity model used in SACCS with our conceptual similarity and the baselines we used in Section 5.1, to create as many baselines for this experiment.

Evaluation benchmark. We follow the same experiment used in [START_REF] Gaci | Subjectivity aware conversational search services[END_REF] to assess the overall quality of the search system, and hence evaluate the practical value of conceptual similarity. Mainly, we use the same crowdsourced evaluation benchmark as in [START_REF] Gaci | Subjectivity aware conversational search services[END_REF], consisting of subjective search queries with three levels of difficulty: Short queries have only one subjective tag; Medium queries have two; Long queries with three. Each difficulty level contains 100 different search queries, and each query is associated with a ranked list of relevant restaurants that best answer it.

Evaluation metric. We evaluate the final search quality using the popular Normalized Discounted Cumulative Gain (NDCG) [START_REF] Manning | Introduction to information retrieval[END_REF]. The closer the score is to 1 using this metric, the better are the search results overall. Given that we use the same system in all the baselines of this experiment, and that these differ only in the underlying similarity model in use, we infer that the NDCG scores directly reflect the quality of the similarity models. Table 3 shows the results.

Results. Table 3 demonstrates the effectiveness of conceptual similarity, outperforming all other similarity models on all levels of difficulty, especially the universal cosine similarity which performs worse by a margin of 2.76%. This experiment proves that conceptual similarity is efficient when plugged in tag-based search applications.

Conclusion

In this work, we propose conceptual similarity for subjective tags. We also propose a methodology to automatically generate training datasets for conceptual similarity with minimal effort given a domain and a set of concepts. Unlike traditional semantic similarity, our model is trained with conceptual signals as reflected in the generated dataset. Intrinsic and extrinsic experiments demonstrate the superiority of our approach on subjective tags.

On the other hand, we acknowledge the following limitations. Although the method is independent from the application domain, we constrained our evaluations to the Restaurants domain for reasons related to unavailability of test data. So we were forced to create our own test benchmark by asking three participants to give ground truth labels for 500 pairs of subjective tags. This may seem small-scale, which risks putting into question the conclusions regarding the superiority of our similarity approach. However, the extrinsic experiment that we conduct by using relatively larger crowdsourced data shows that our approach is efficient and outperforms other similarity models, which assuages our concern. As future work, we plan to apply our methods on other domains, e.g. hotels, or electronics.

In this paper, we build the whole argument of our contributions against the blind use of cosine similarity in tag-based search systems, and to replace it with our newly proposed conceptual similarity. However, we employ BERT and LSTMs in our model which incur a much higher computational cost than cosine similarity. The adoption of our model in practice depends on whether efficiency is a major concern in the downstream search application, i.e. whether a poor search inflicts major negative consequences in critical domains such as finances or regulations. It also depends on the underlying infrastructure into which conceptual similarity will be deployed, e.g. are there any GPUs in use? Is memory space enough to hold BERT and LSTMs? So whether to adopt our contributions in practice is a compromise between cost and efficiency.

A.2 Parameter Configurations of Expanders

To generate the dataset used in the experiments of this paper, we use all the expansion techniques described in Section 3.2. For each technique, we use different parameter configurations to increase the diversity of the generated expansions. For example, GloVe and Paragram embeddings do not generate the same words given that each embedding model has been trained differently, and thus encode the representation of words in a unique way. Also, in Language Generation Expansion, we use different language models with different allowed lengths. This is to enable the generation of n-grams, in addition to words. We give the list of the expanders we use, and their parameters in Table 4.

We have a total of 28 different expanders. We set the parameter min_consensus_rate to 0.3. Consequently, for a new token to be included in the final set of expansions and passed down to the subsequent steps of the dataset generation pipeline (see Section 3 and Figure 1), the token has to be suggested by at least 30% of expanders (9 different expanders in this case). We selected this value by doing a manual hyperparameter search over the following values of min_consensus_rate: {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}. We took the value (i.e. 0.3) that maximized the quality of the final generated dataset, as evaluated in Section 5.2. However, we chose the parameters of the respective expansion techniques manually without conducting a hyperparamter search for the following reasons: (1) There are too many parameters to test, which would make the search space exponentially larger, and thus expensive to explore. (2) The parameter selection of expansion techniques is subjective by nature. We manually chose the parameters such that they make sense (e.g. a negative capacity in ConceptNet Expansion or a very large top_k in Masked Language Modeling Expansion would not be useful), and such that the final expanders would generate a diverse set of expansions from a limited lexicon of seeds.

A.3 Concepts Used in this Work and their Seeds

We select 9 different concepts to include in the conceptual similarity model described in the experiments. We base our choice of concepts on substantial research in behavioral psychology [START_REF] Luiz | Choosing a restaurant: Important attributes and related features of a consumer's decision making process[END_REF] whose authors surveyed restaurant seekers and asked them about which factors influence their

Figure 1 :

 1 Figure 1: Labeled dataset generation pipeline

Figure 2 :

 2 Figure 2: Different expansion techniques: (a) WordNet, (b) ConceptNet, (c) Embedding, (d) Language generation, (e) Masked Language Modeling

Figure 3 :

 3 Figure 3: Similarity model architecture

Table 1 :

 1 Evaluation of similarity models

	Similarity Model	Pearson Spearman MAE
	Cosine (Word2vec)	0.6770	0.6190	0.2083
	Cosine (BERT MEAN) 0.3449	0.3312	0.5313
	Cosine (BERT CLS)	0.0497	0.0848	0.6920
	BERT Classif	0.5946	0.5404	0.1703
	Random Forest	0.6271	0.6324	0.2614
	Siamese	0.7058	0.6141	0.1903
	Conceptual Sim	0.8512	0.7388	0.1134

3 https://www.yelp.com/dataset

Table 2 :

 2 Evaluating similarity on noisy training data

	Noise level Pearson Spearman MAE
	Original	0.8512	0.7388	0.1134
	5% noise	0.7341	0.6641	0.1958
	10% noise	0.7788	0.7101	0.1898
	25% noise	0.7418	0.7055	0.2879
	50% noise	-0.1209	-0.0943	0.4078

Table 3 :

 3 Evaluating the ranking quality of a tag-based search system with different similarity models rant. The final output of SACCS is a ranked list of restaurants ordered by relevance to the user query.

	Similarity Model	Short Medium Long
	Cosine (word2vec)	0.7956	0.8579	0.8750
	Cosine (Paragram)	0.8072	0.8602	0.8741
	Cosine (BERT MEAN) 0.7807	0.8512	0.8740
	Cosine (BERT CLS)	0.7807	0.8498	0.8738
	BERT Classif	0.7968	0.8543	0.8744
	Random Forest	0.8048	0.8623	0.8790
	Siamese	0.7961	0.8618	0.8823
	Conceptual Sim	0.8232	0.8717	0.8839

Table 4 :

 4 The full list of expansion techniques and their parameter configurations that we used to expand the seed words in our experiments decision-making process when they chose between restaurants. In Table5, we describe the concepts that we use, and give their corresponding seeds for aspects and opinions. Food aspects food, menu, plate, cuisine, meal, lunch, dinner, breakfast, cooking, snack, beverage, drink, pizza, pasta, chicken, meat, steak, rice, soup, dessert, dish, fish, salad opinions 1 (good) tasty, good, excellent, succulent, okay, delicious, well seasoned, perfectly cooked opinions 2 (bad) bad, flavorless, bland, not seasoned, cold, disgusting, unappetizing, flat, gross, boring, awful, terrible, dry opinions 3 (healthy) healthy, organic, high quality, fresh opinions 2 (creative) novel, interesting, creative Service aspects staff, waiter, waitress, cashier, service opinions 1 (warm) friendly, smiling, good, helpful, likable opinions 2 (competent) knowledgable, quick, fast, efficient, high quality, professional opinions 3 (bad) grumpy, horrible, slow, irritating, bad Cleaning aspects place, hygiene, kitchen, bathroom, utensils, plates, cutlery, silverware, trays, dishes, table, chair, furniture opinions 1 (clean) clean, impeccable, bright, lavish, luxurious, washed, shining opinions 2 (dirty) dirty, bad, in bad shape, stained, greasy, not washed, poor, disgusting good, excellent, great, cozy, comfortable, sophisticated, good taste, pleasant, memorable, adequate, beautiful, soothing, calming, fancy, attractive, happy, relaxing, nice, charming opinions 2 (bad) bad, horrible, bad taste, uncomfortable, dark, noisy, terrible, crowded, sad, depressing, boring Location aspects location, area, place, address opinions 1 (good) near, good, downtown, lively, touristy, popular, secure, safe, good, trustable opinions 2 (bad) far, bad, polluted, remote, dark, unsafe, unsecure, dangerous Ambiance aspects ambiance, atmosphere, air, experience, environment, setting, decor, lighting, music, ventilation, furniture opinions 1 (good) cozy, good, excellent, romantic, nice, upscale, trendy, loved, enjoyed, fun opinions 2 (bad) horrible, terrible, disgusting, bad, not good, disappointing, noisy, dark, depressing, boring

		WordNet Expansion
	num_synsets hyponym meronym hypernym sisters
	3	true	true	true	true
	10	true	true	true	false
	5	true	false	true	true
		ConceptNet Expansion
	capacity	minimum_weight	second_level_expansion
	3	2.0		true
	5	3.0		true
	10	1.0		false
	Word Emebedding Expansion
	embedding_model num_words	distance_metric
	Word2vec	20		euclidean distance
	Word2vec	20		cosine similarity
	GloVe	20		euclidean distance
	GloVe	20		cosine similarity
	Fasttext	20		euclidean distance
	Fasttext	20		cosine similarity
	Paragram	20		euclidean distance
	Paragram	20		cosine similarity
	ConceptNet	20		euclidean distance
	ConceptNet	20		cosine similarity
	Language Generation Expansion
	model	top_k	max_length	num_beams
	GPT2	20	1		200
	GPT2	20	2		200
	T5 base	20	3		200
	T5 base	10	3		50
	Masked Language Modeling Expansion
	model			top_k
	BERT base			10
	BERT base			20
	BERT large			10
	BERT large			20
	RoBERTa large		10
	RoBERTa large		20
	ALBERT large		10
	ALBERT large		20

Table 5 :

 5 The full list of seeds (aspects and opinions) per concept used in our experiments

https://github.com/YacineGACI/conceptual-similarityfor-subjective-tags

This may be the result of expanding service to duty through WordNet, even though service in this case refers to the waiters in a restaurant

https://github.com/pytorch/pytorch

https://github.com/huggingface/transformers

A Appendix

A.1 Similarity Model Details & Hyperparameters

We use a hidden dimension of 128 for the LSTM layer, and 512 for the 2-layer classification FFNN.

We apply dropout with a ratio of 0.3. To train the model, we minimize cross entropy of the training set, and use Adam optimizer (Kingma and Ba, 2014) to update the parameters with 5e -6 as learning rate. For hyperparameter search, we pick the hyperparameters which work best on a development set that has been generated in the same way as the training set. We implemented conceptual similarity in Python using standard packages such as PyTorch 4 for neural networks, HuggingFace transformers library 5 for BERT and GPT2.