Such public data sharing puts at risk the users' privacy and expose them to a surveillance

J'adresse mes sincères remerciements au professeur Harald KOSCH et au professeur David GROSS-AMBLARD pour avoir consacré leur temps pour rapporter mon travail de thèse. Je remercie également professeur Ernesto DAMIANI, professeur Kokou YETONGNON et professeur Philippe ANIORTÉ d'avoir bien voulu faire partie des membres de mon jury. Merci à Philippe pour sa bienveillance, ses conseils et aussi son amabilité.

La thèse est un voyage avec un problème à résoudre certes, mais aussi une aventure que l'on vit avec des personnes pendant des années... des années de ce qui fut la période la plus intense de ma vie.

Merci au professeur Richard CHBEIR pour cette belle aventure dont il était le guide. Il a été celui qui n'a jamais cessé d'accroître ma capacité de produire et en même temps celui qui fut capable de me tirer vers le haut quand j'en avais besoin. Que ce travail soit le gage de mon infinie gratitude.

Merci au docteur Bechara AL BOUNA pour son aide depuis le début de mon travail de recherche et son soutien tant sur le plan humain que professionnel.

Merci au docteur Joe TEKLI pour ses conseils qui ont été d'une aide

Furthermore, new techniques are used today to extract publicly shared data and combine it with other data in ways never before thought possible. However, social networks users do not realize the wealth of information gathered from image data and which could be used to track all their activities at every moment (e.g., the case of cyberstalking). Therefore, in many situations (such as politics, fraud fighting and cultural critics, etc.), it becomes extremely hard to maintain individuals' anonymity when the authors of the published data need to remain anonymous.

Thus, the aim of this work is to provide a privacy-preserving constraint (de-linkability) to bound the amount of information that can be used to re-identify individuals using online profile information. Firstly, we provide a framework able to quantify the re-identification threat and sanitize multimedia documents to be published and shared. Secondly, we propose a new approach to enrich the profile information of the individuals to protect. Therefore, we exploit personal events in the individuals' own posts as well as those shared by their friends/contacts. Specifically, our approach is able to detect and link users' elementary events using photos (and related metadata) shared within their online social networks. A prototype has been implemented and several experiments have been conducted in this work to validate our different contributions.

Résumé

De nos jours, les réseaux sociaux ont considérablement changé la façon dont les personnes prennent des photos qu'importe le lieu, le moment, le contexte. Plus que 500 millions de photos sont partagées chaque jour sur les réseaux sociaux, auxquelles on peut ajouter les 200 millions de vidéos échangées en ligne chaque minute. Plus particulièrement, avec la démocratisation des smartphones, les utilisateurs de réseaux sociaux partagent instantanément les photos qu'ils prennent lors des divers événements de leur vie, leurs voyages, leurs aventures, etc. Partager ce type de données présente un danger pour la vie privée des utilisateurs et les expose ensuite à une surveillance grandissante. Ajouté à cela, aujourd'hui de nouvelles techniques permettent de combiner les données provenant de plusieurs sources entre elles de façon jamais possible auparavant. Cependant, la plupart des utilisateurs des réseaux sociaux ne se rendent même pas compte de la quantité incroyable de données très personnelles que les photos peuvent renfermer sur eux et sur leurs activités (par exemple, le cas du cyberharcèlement). Cela peut encore rendre plus difficile la possibilité de garder l'anonymat sur Internet dans de nombreuses situations où une certaine discrétion est essentielle (politique, lutte contre la fraude, critiques diverses, etc.).

Ainsi, le but de ce travail est de fournir une mesure de protection de la vie privée, visant à identifier la quantité d'information qui permettrait de ré-identifier une personne en utilisant ses informations personnelles accessibles en ligne. Premièrement, nous fournissons un framework capable de mesurer le risque éventuel de ré-identification des personnes et d'assainir les documents multimédias destinés à être publiés et partagés. Deuxièmement, nous proposons une nouvelle approche pour enrichir le profil de l'utilisateur dont on souhaite préserver l'anonymat. Pour cela, nous exploitons les évènements personnels à partir des publications des utilisateurs et celles partagées par leurs contacts sur leur réseau social. Plus précisément, notre approche permet de détecter et lier les évènements élémentaires des personnes en utilisant les photos (et leurs métadonnées) partagées au sein de leur réseau social. Nous décrivons les expérimentations que nous avons menées sur des jeux de données réelles et synthétiques. Les résultats montrent l'efficacité de nos différentes contributions. Figure 11: The framework for an inference detection system proposed in [START_REF] Chen | Database Security Protection Via Inference Detection[END_REF].

List of Figures

Figure 12: Online Social Footprint defined in [START_REF] Irani | Large Online Social Footprints--An Emerging Threat[END_REF].

Figure 13: Attribute list of personal information explored in [START_REF] Chen | Is more always merrier?: a deep dive into online social footprints[END_REF].

Figure 14: System architecture presented in [START_REF] Malhotra | Studying User Footprints in Different Online Social Networks[END_REF].

Figure 15: Overview of the approach proposed in [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF] to disambiguate identity web references using Web 2.0 data. Figure 16: Approach used in [START_REF] Minder | Social Network Aggregation Using Face-Recognition[END_REF] for data aggregation.

Figure 17: Overview of the distributed di.me system architecture in [START_REF] Scerri | digital. me towards an integrated Personal Information Sphere[END_REF].

Figure 18: The approach process proposed in [START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF].

Figure 19: Distinction between personal events and social events, as defined in [START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF].

Figure 20: Basic aspects of event description from [START_REF] Westermann | Toward a Common Event Model for Multimedia Applications[END_REF].

Figure 21: Event detection framework presented in [START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF][START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF].

Figure 22: Architecture diagram of event detection approach proposed in [START_REF] Sarma | Dynamic relationship and event discovery[END_REF]. -dimensional space: locations, times, and people names. Table 1: Comparison of inference control approaches.

List of Tables

I. Introduction

Large scale web applications are gaining increasing interest in recent times across a range of sectors, in both large and small firms. Companies are now constantly looking at what kind of data they have and what data they need in order to maximize their market position. In the era of big data, there are concerns about data privacy, and even the potential future value of data, as expressed in the White House Counsel John Podesta's 2014 report to the President on the challenges of Big Data [START_REF] Podesta | Big Data: Seizing Opportunities Preserving Values[END_REF]. The main added privacy risk is that this datafrom voice calls, emails and texts to uploaded pictures, video, and musicis being reused and combined with other data in ways never before thought possible. Further, the report states: "This is driving data collection to become functionally ubiquitous and permanent, allowing the digital traces we leave behind to be collected, analyzed, and assembled to reveal a surprising number of things about ourselves and our lives". With all these developments, the ever-increasing amount of information flowing through social media and blogging sites has reflected the need for heightened privacy controls. People are sharing and uploading upwards of 1.8 billion photos 1 a day 2 . In many situations, motivated by several campaigns such as politics, fraud fighting, cultural critics, and others, authors of some of these social medias need to remain anonymous. Consequently, when a data provider outsources or publishes multimedia documents, it becomes extremely hard sometimes to maintain individuals' anonymity mainly due, but not limited to: 1) the number of active social networks to which they actually participate, and 2) the trails of seemingly information they leave behind [START_REF] Malin | Trail re-identification and unlinkability in distributed databases[END_REF]. These trails of information make individuals victims of what is known by the Internet community as cyberstalking where an adversary clandestinely tracks the movements of an individual. 1 Both terms photo and image will be used interchangeably in the remainder of this study. 2 http://www.kpcb.com/internet-trends

I.1 The Mass Surveillance on the Internet

Technology has brought many advances and conveniences in various fields of study, including public health, education and transportation. However, it also comes with the cost of privacy. Many examples have been seen in the news (e.g., the US National Security Agency -NSA 3 -spying on German chancellor Angela Merkel 4 , NSA spying on 98 percent of South American communications 5 , etc.). "Everyone is under surveillance now" 6 , says Edward Snowden 7 , who was responsible for leaking PRISM, one of the surveillance programs of the United States government. PRISM is a tool used by the NSA to collect private electronic data belonging to users of major Internet services like Gmail 8 , Facebook 9 , Outlook 10 , and others 11 .

NSA programs collect two kinds of data: metadata and content. In essence, Snowden's disclosure was the entry of the term 'metadata' into common usage. This is information about the time and location of a phone call or email. In addition to written and oral communications, the NSA has collected millions of faces from web images to develop the large untapped potential of using facial images, fingerprints, and other identifiers to track suspected terrorists and other intelligence targets 12 .

3 https://www.nsa.gov/ 4 http://www.theguardian.com/us-news/2015/jul/08/nsa-tapped-german-chancellery-decades-wikileaks-claimsmerkel 5 http://techcrunch.com/2015/07/08/assange-nsa-intercepts-98-of-south-american-communications/ 6 http://www.theguardian.com/world/2014/may/03/everyone-is-under-surveillance-now-says-whistlebloweredward-snowden?CMP=twt_gu 7 A former employee of the CIA and of a private contractor working for the US National Security Agency (NSA) 8 https://mail.google.com/ 9 https://www.facebook.com/ 10 www.microsoft.com/en-us/outlook-com/ 11 http://www.theverge.com/2013/7/17/4517480/nsa-spying-prism-surveillance-cheat-sheet 12 See James Risen and Laura Poitras, , "NSA Collecting Millions of Faces From Web Images", in: The New York Timesof 31 May 2014 On the one hand, the NSA claims that it absolutely needs all this data to help prevent terrorist attacks. On the other hand, privacy activists critical of the NSA surveillance program disagree, arguing not only that the collection is based on a legal interpretation that goes way beyond what Congress allowed, but also that metadata includes personal information, which can build a more detailed profile even than listening into content 13 .

I.2 The Issue of Privacy Protection

The problem particularly occurs when the so-called "metadata" are shared with others and combined with publicly available information. For example, in a study released in January 2015, titled "Unique in the Shopping Mall: On the Reidentifiability of Credit Card Metadata", a group of data scientists analyzed credit card transactions made by 1.1 million people in 10,000 stores over a three-month period 14 . The database did not have any names, account numbers, or other obvious identifying features. Nevertheless, even with this anonymous data, data scientists could re-identify 90% of the shoppers as unique individuals and could uncover their records, just by knowing a few bits of information about them. And that uniqueness of behavior combined with publicly available information in multimedia documents, such as Instagram or Twitter posts, made it possible to re-identify people's records by name.

Personal data means any information relating to an identified or identifiable individual. An identifiable person is one who can be identified, directly or indirectly, in particular by reference to an identification number (e.g., social security number) or one or more factors specific to his physical, physiological, mental, economic, cultural or social identity (e.g., name and first name, date of birth, biometrics data, fingerprints, DNA, etc.) 15 . 13 http://www.theguardian.com/world/interactive/2013/nov/01/snowden-nsa-files-surveillance-revelationsdecoded#section/2 14 http://dujs.dartmouth.edu/computer-science/making-data-anonymous-not-enough-to-protect-consumerprivacy#.VgqxDMuqqko 15 http://www.cnil.fr/english/data-protection/personal-data-definition/ Therefore, even when all personally identifying information, such as name, birthdate, address, is completely removed from a database, that data is not as entirely anonymous as many people think. All of this bears out the need of better means to anonymize all multimedia documents that can be attributed to anonymous users. Unfortunately, removing all "personally identifying information" is not a trivial operation with all the digital trails users leave today in their regular activities on the Internet (e.g., blogging, social networking, and file sharing, etc.). Meanwhile, requiring an absolute guarantee that a data set cannot be linked back to individuals probably might render it useless in some cases or discourage any sharing.

"Finding the right balance between privacy and utility is absolutely crucial to realizing the great potential of metadata", say the researchers of the Credit Card Metadata study.

Anonymization is the modification of data so that sensitive information remains private. Deanonymization is the converse: re-identifying somebody in an anonymized networkor even simply learning something about them that was not meant to be attributable to them.

II. Thesis Context

In this work, we are interested in personal data that is left in the raw data of multimedia publications and that can be attached to an individual. In this section, we outline the main motivations behind our interest in exploiting metadata of photos shared within online social networks (OSN).

II.1 Popularity and Usage of Online Social Networks

Nowadays, the popularity of social networks is constantly growing. According to a survey conducted by the Pew Research Center 16 , the number of social networks' users has shown an explosive growth between 2005 and 2013, as illustrated in Figure 1. Seventy four percent (74%) of American Internet adults use social networking sites, and even more than their half (52%) now use two or more social media sites compared with 42% who did so in 2013.

II.2 Emergence of Online Sharing of Multimedia Content

With the advent of new types of media and the diversity of technologies supporting multimedia, it became easy to take a video, capture an audio or snap a picture. Furthermore, user-friendly social networks applications made multimedia sharing easy and therefore a popular activity. According to a report on Ipsos 17 , a survey indicated that pictures are the most popular shared item on social media sites (43% as seen in Figure 3). Indeed, photos have become the primary content form of the online sharing, expressing easily a variety of information such as interests, activities and life experiences. Figure 4 illustrates the daily number of photos uploaded and shared on social media (Facebook, Flickr, Instagram, Snapchat, and WhatsApp). However, social network users do not seem to fully realize the wealth of information (metadata) that can be obtained from those photos and which could be used to track their activities. For instance, the project "I Know Where Your Cat Lives" 18 raises concerns over online privacy by using public images of cats uploaded to photo sharing websites (e.g., Flickr 19 , Twitpic 20 and Instagram). Location coordinates embedded in the images' metadata are used to show where each cat lives and, more importantly, to track their owner's homes.

II.3 Tendency of Sharing Photos of Personal Events

People are increasingly sharing their daily lives online, exposing their everyday movements and putting more and more from their personal information online. From the coffee mug for #Breakfast to the #Sleeping#Cat, people are snapping photos of everything. Sharing real-life experiences with friends at any time and any place has become common on social networks, from social events (festive celebrations, concerts, sporting events, etc.) and personal events (family gathering, wedding and other parties, etc.) to the most intimate private moments, conducting sometimes to breach users' privacy and scandals.

Furthermore, people are constantly using their cell phones to capture events happening around them, whether they participated in or just observed. The evolution of smartphones has therefore changed the way people share their events and the way they interact with them as well. It is no more surprising that cameras and phones are these days present all the time in our life, even during others' precious moments such as weddings and births. People have all become so used to sharing instantly on social media applications what they are doing that they spend much time recording, for the purpose of sharing their experiences and activities online.

Because photos posted on social networks represent real stories from users' life, we focus in this work on detecting events from photos uploaded online on photo sharing websites.

III. Anonymity in Online Social Networks

Online privacy does not only concern terrorists, hackers or child pornographers. Some people are also worried about staying safe from their own governments or employers. Others might not want to let their online audience know their sex or sexuality, or just do not want their names attached to a comment or a question online. To be anonymous, means one is not identified by name or has an unknown name. A number of factors are likely to drive people to stay anonymous on the Internet. One of these factors is the fact that the Internet is forever.

Whatever question people ask or opinion they share, there it is, more or less forever. In "I Know Who You Are and I Saw What You Did: Social Networks and the Death of Privacy" [START_REF] Andrews | you are and I saw what you did: Social networks and the death of privacy[END_REF],

Lori Andrews shows, on the one hand, the importance of anonymous postings in line with the right to privacy of citizens. And, on the other, Andrews describes the potential of harmful anonymous postings in some extreme cases (e.g., defamation, discrimination, etc.). Indeed, online anonymity does not only help protect fundamental rights, but also ensures the personal safety of individuals expressing their opinions (e.g., postings about politics, political figures, social institutions and services, etc.). Given its ability to facilitate the rich, diverse and far ranging exchange of ideas, anonymous use of the Internet allows protesters to deliver their message to a larger audience without the message being prejudged (e.g., revolutions of the "Arab Spring"21 in 2011).

However, anonymity on the Internet may magnify particular harms and facilitate wrongdoing (e.g., terrorism, violation copyright laws, identity theft, cyber predators, cyberstalking, etc.). This becomes part of the problem of anonymous cybercrime. Cybercriminals have found ways to monetize personal information found on the user profile pages of social networks. For instance, this type of personal identifiable information (PII) harvesting allows cybercriminals to obtain answers to security questions used to verify the user's identity when attempting to log in to sensitive services such as online banking sites 22 .

III.1 The case of Cyberstalking

In the context of criminal activities involving human beings, a stalking crime is generally considered to be one in which an individual ("the stalker") clandestinely tracks the movements of another individual or individuals ("the stalkee"). Cyberstalking can be understood as a form of behavior in which certain types of stalking-related activities -which in the past have occurred in physical space -are extended to the online world [START_REF] Tavani | Cyberstalking, personal privacy, and moral responsibility[END_REF]. Internet stalkers can operate anonymously or pseudonymously while online.

In the next section, we focus our attention on a specific case of Internet stalking involving the previous French Prime Minister François Fillon. Actually, Fillon was using a fake account name "@fdbeauce" to remain anonymous on Twitter 23 . He was able to operate online pseudonymously to track messages of his ministers and journalists publishing information about him. In spite of this, we will see how it was possible for the adversary to recognize his target, and thus break his anonymity.

An adversary is somebody who attempts to reveal sensitive, private information about a target. A target is the particular social network member against whom an adversary is trying to breach privacy.

III.2 Motivating Scenario

François Fillon 24 , the main persona of our scenario, is a French politician present online with the Twitter account @fdebeauce 25 (shown in Figure 5).

Figure 5:

The Twitter account of François Fillon with the fake nickname "@fdbeauce". 23 https://twitter.com/ 24 François Fillon is a French lawyer and politician who served as Prime Minister of France from 2007 to 2012. https://en.wikipedia.org/wiki/Fran%C3%A7ois_Fillon 25 https://twitter.com/FrancoisFillon François Fillon 26 thought that using a fake account name on Twitter will save his privacy and keep him anonymous under the protection of the Internet security. In reality, it didn't take more than few minutes to reveal the hidden account of the French politician. In essence, the adversary used the available information on Twitter (account name, profile image and a previously published tweet) with some background knowledge to recognize his target.

Background knowledge is defined as any piece of information that is not directly revealed to the users but is available in the outside world or in a system and can be used to infer the attribute in question. It can also be understood as a mental model or world model that provides rules of how to link information together. Such background knowledge is often acquired from real world through experience and observation. Not only may inference require the use of background knowledge, but also the information being inferred (e.g., users' identity at physical appearance granularity) may not be stored in the application database [START_REF] Motahari | How did you know that about me? Protecting users against unwanted inferences[END_REF].

In the following, we describe the personal data provided by François Fillon that contained enough clues resulting in identity disclosure.

Identity disclosure is a privacy breach in which a presumably anonymous person is in fact identifiable.

III.2.1 Profile Information

The first clue that made this attack successful is the username or the Twitter alias "@fdbeauce". This alias is based on his real information: "F" as the initial of his name. Moreover, "debeauce" is taken from his village name "Beaucé". A fragment of the Wikipedia's page of François Fillon is shown in Figure 6.

III.2.2 Metadata

The second clue is the profile image published on this account (Figure 7), and more precisely the GPS coordinates embedded in its metadata indicating that it was taken in Beaucé.

Figure 7:

The profile image on "@fdbeauce" Twitter account.

His profile image shows a stream landscape without any description. François Fillon thought that it was therefore impossible to get any information about him from this image. However, some people were interested in exploring the metadata embedded in the image data. Metadata (see Figure 8) indicates that it was taken on October 31, 2011 at Beaucé, in Sarthe. Fillon also shared his complete address (postal code, city, and country), his phone number and email, which were stored in the memory of the camera used to take the photo. As shown before, François Fillon lives in a manor in Beaucé, in the department of Sarthe in western France. The Wikipedia page about François Fillon contains this information, including a picture of the manor where he lives, as illustrated in Figure 9. Hence, the username and profile image of François Fillon, combined with existing public knowledge from Wikipedia, allowed to re-identify him even when using a pseudonym.

III.2.3 Connections

People follow other users on Twitter to make connections. So the third clue to this attack is the connections of the previous Prime Minister. His social network on Twitter is made of French politicians since he followed some accounts of his ministers in the government who actively tweet online. One of the followers of François Fillon is Alain Lambert, a french politician. François Fillon appears in many images published by Lambert. Some of these images were taken in Beaucé (as specified in their metadata). This location information is not available from François Fillon's profile. However, it is obtained by exploiting other users' images in the same social network.

III.2.4 Events

The last clue that made the identification of François Fillon also possible is his first tweet "@alainlambert and I with Japanese prime minister" on October 23, 2011. This tweet announced a meeting with the Japanese prime minister in Japan and Alain Lambert. At the same time, the French News reported François Fillon's visit to Japan for two days. We illustrate in Figure 10 the tweet of "@fdebeauce" and the French News report about François Fillon. A simple comparison between where (Japan), when (October [START_REF] Scerri | Knowledge Discovery in distributed Social Web sharing activities[END_REF]2011) this meeting happened and who (Japanese Prime Minister) is involved in this event allowed to re-identify François Fillon even when using his pseudonym. Furthermore, it is interesting to note that the tweet of "@fdebeauce" and the French News report shows an event of two days. Such temporal information can be used by the adversary to learn sensitive values about his target.

III.3 Challenges

This scenario led us to consider three practices involving the exposure of personal information online.

First, Internet users readily provide personal data to social network sites. However, they do not realize that such data can provide further information through search engines or through combination with other users' shares.

Furthermore, users upload their photos online at any time and from anywhere. Nevertheless, they are not aware of what images can reveal about their identities. Different techniques can be applied to uploaded images. These techniques can provide extra information the users did not share, voluntarily or involuntarily, on their profiles.

Finally, users publish information about their events, travels, social activities, schedule, and so on. Initially, such data does not comprise information about their identities (e.g., the visit to Japan in our scenario). However, the Web is continuously changing, allowing adversaries to find information about any user, and combine them with public data to infer hidden personal user information (the real name in our scenario). Moreover, adversaries are able to link information together to infer more personal user information, such as social relations, regular locations, etc.

Following this scenario, we believe that the challenges faced to prevent the identification of an anonymous user from multimedia documents are the following:

III.3.1 Remove sensitive information of both textual and multimedia content

The first challenge is to identify data (either provided by the user or propagated from the user's connections) in the OSN, which may contain clues about the user's profile and/or events. This data must be removed since it can be used to infer the identity of an anonymous user.

III.3.2 Consider the complex nature of multimedia objects

The second challenge is to protect the privacy of users publishing any multimedia document.

More specifically, even if multimedia documents do not contain images of people, anonymous users can be readily identified from the combination of some metadata with other public multimedia document.

III.3.3 Preserve the utility of multimedia documents

The third challenge is to anonymize all sensitive data to ensure that multimedia documents shared online cannot be used to disclose the identity of the individual while keeping the utility of those shares. Let us go back to the tweet example of François Fillon: "@alainlambert and I with Japanese prime minister". Removing all clues from the tweet of "@fdebeauce" might achieve perfect privacy, but it will be of total uselessness if the tweet results as the following: "and I with".

III.4 Privacy Preservation Solutions in Multimedia Publication

Currently, there are many ways users can protect their privacy when publishing multimedia documents. In the following, we suggest two alternatives that users can use to stay anonymous online:

III.4.1 Burying personal data with outdated or false information

One alternative to protect online privacy is to alter personal data of the user who wants to remain hidden. This can be done by using tools that corrupt personal data existing about users online. The tool Undefined 27 is able to post content on behalf of user's social network account and to alter his navigation in order to confuse the components of the digital identity. Different strategies can be used to send corrupted data online, ranging from sending similar data (supporting his identity), to incomplete data (making it more confusing). Here we describe some possible ways to alter personal data online. The following example is a real case of burying personal data with false information (i.e., images) in order to stay (visually) anonymous online. Jonathan Hirshon 28 asked his friends to tag him in random photos. These photos ranged from a pig pushing a shopping cart to a Buddha chill-out poster to Bill Nye the Science Guy to Don Draper to a Masonic logo, Brad Pitt, a cat, Abraham Lincoln, a seagull, and so on... Using this technique, the collection of false positives associated with the tag "Jonathan Hirshon" could confuse Facebook and Google's algorithms. This "algorithm hacking" permitted to bury any real photo of Hirshon. This works well since a Google image search for his name does not return a single picture that is actually portraying him.

III.4.2 Removing or untagging content

Another alternative to preserve anonymity is to remove any published data that may be tied to a particular user. However, this is not always possible when the user is not the originator of the publication. The National Commission on Informatics and Liberty 29 (which is an independent administrative authority that operates in accordance with the data protection legislation, and also known as the CNIL) promoted the "right to be forgotten" and the "right to delisting". We will review briefly these data protection rights in the following:

The European Court of Justice enshrined the right to be forgotten in a 2014 judgment 30 .

Specifically, anyone who wishes to delete one or more results appearing from a search of his name can make a request to search engines, in particular Google. Google then reviews the application and grants the right if the legal conditions are met. The right to be forgotten is practically applied through the right to obtain the deletion of personal information, or the right to delisting on all of the search engine's domain names.

In practical terms, any individual who wants to see removed one or more results displayed following a search made on the basis of his/her name can make a request to a search engine, under certain conditions which take into account the public's right to information. Delisting does not lead to deletion of the information on the source website. The original information remains unchanged and will still be found on search engines using other search terms, or by direct access to the publisher's original source 31 . 29 http://www.cnil.fr/english/ 30 https://www.rudebaguette.com/2015/09/23/right-to-be-forgotten-cnil-and-googles-arm-wrestling-match-isonly-the-symptom-of-a-deeper-disease/ 31 http://www.cnil.fr/english/news-and-events/news/article/questions-on-the-right-to-delisting/

III.4.3 Privacy Preservation prior to Publication

Complete burying or removal of all (directly or indirectly) identifying personal data is impossible on the Internet today. Currently, new techniques are used to extract publicly shared data and combine it with other data in ways never before thought possible. No matter how much users cover their publications or posts, there could be always some information that can potentially be used to potentially identify them. Hence, controlling multimedia content prior to its publication is necessary to better prevent violation of anonymity. For this reason, we believe privacy protection (i.e., data anonymization) must be addressed prior to data sharing in online social networks.

IV. Contributions

In a viewpoint entitled "Managing Your Digital Life" [START_REF] Abiteboul | Managing your digital life[END_REF], Serge Abiteboul et al. stress the need for Internet users to manage themselves their personal data. Giving users more control over how others gather and use their personal data is now more than essential. To tackle this privacy problem, we suggest in our research to let an individual control the sharing of photos made (by himself or by other users) on social media sites such as Google or Facebook.

According to the authors of [START_REF] Abiteboul | Managing your digital life[END_REF], we live in a world where it is easy to publish information but difficult to remove it or sometimes to simply access it. Therefore, users must be warned of potential violation of their privacy, before it is too late. The main contributions of this work can be summarized as follows:

IV.1 A secured publication process of multimedia documents

We provide a privacy-preserving constraint (de-linkability) to bound the amount of information that can be used to re-identify individuals using online profile information. Delinkability ensures that individuals' identifiable information composed of both textual and multimedia content cannot be used to infer his/her identity. We formally define the identity anonymization problem in multimedia documents. Then, we provide a framework able to quantify the re-identification threat to sanitize multimedia documents to be published (MD *algorithm) and preserve at the same time their utility. Therefore, we provide a utility measure to determine to which extent a multimedia document remains consistent after the sanitizing process. Finally, we show a set of experiments elaborated to demonstrate the efficiency of our technique on real-world datasets.

IV.2 Event detection from images shared on social networks

We suggest a new approach to enrich the profile information of the individuals to protect.

Following our motivating scenario, we exploit personal events in the individuals' own posts as well as those shared by their friends/contacts. Specifically, our approach is able to detect users' elementary events using photos (and related metadata) shared within their online social networks. We also developed a prototype tool called Foto2Event to validate the relevance of our clustering algorithm on large datasets.

IV.3 Computing links between events on social networks

We present a meta-model that allows representing, combining and inferring inter-event relations in an expressive and flexible way. Our meta-model allows the discovery of relations that can exist between events detected in photos shared online. Using our model, we can automatically generate relations that correspond to different aspects of events based on a homogeneous representation (spatial, temporal and social). In addition, we present a methodology that can combine spatial, temporal and social relations of the event for modeling complex relations (e.g., ontological relations such as subevents, isA, etc. or application-based relations such as atHome, atAirport, etc.).

V. Report Organization

The chapters of this report are organized as follow:

Chapter 2 gives a review related to i) inference detection, ii) construction of online identity, and iii) event detection and linking approaches.

Chapter 3 presents our privacy preservation framework. We propose de-linkability, a privacy-preserving constraint to bound the amount of information shared that can be used to re-identify individuals. We describe a generic data model to deal with any type of multimedia data. Then, we introduce operators necessary to achieve de-linkability. We also provide a sanitizing MD * -algorithm to enforce de-linkability along with a utility function to evaluate the utility of multimedia documents that is preserved after the sanitizing process. Finally, we show the set of experiments we elaborated to demonstrate the efficiency of our technique.

Chapter 4 describes our model for detecting personal events using photos' metadata shared within online social networks. We consider a specific scenario in order to describe key challenging issues regarding event detection. Then, we present a clustering algorithm able to detect and link users' elementary events using photos (and related metadata) shared within their online social networks. We present our prototype Foto2Event and experimental tests conducted on the MediaEval dataset.

Chapter 5 introduces our methodology for describing links between events. First, we describe our meta-model based on the 4-Intersection Model. Then, we present our methodology to identify topological relations based on three main aspects of events: spatial, temporal and social. We explain how we can combine those basic relations to infer more complex relations such as ontological relations and application-based relations.

Chapter 6 concludes our work and presents some of our future research directions.

Chapter 2: Related Works

Abstract

The focus of this work is on understanding privacy preserving techniques to prevent privacy violations via data inferences that occur when background knowledge is combined with nonprivate data. We first address privacy threats due to social inferences. Social inferences are the subset of inferences that results from using social applications and can pose serious threats to the privacy of their users. Such inferences are often enabled by the public sharing of personal information through social media. For this reason, we present in a second phase approaches related to the construction of online identity from multimedia data shared on online social networks. Then, we provide a review on different approaches related to events detected on online social networks. We are mainly interested in two tasks: (1) Detecting events using shared photos and (2) Linking events. In this chapter, we summarize and discuss the limitations and features of the outlined approaches.

I. Introduction

The rapid growth in the electronic exchange of personal information through social media and blogging sites today has reflected the need for privacy protection. One of the biggest issues related to online privacy is online anonymity. In many cases, users may wish to stay anonymous to protect themselves from identification. However, with the increasing volume of published data, their personally identifiable information can be readily inferred from the publicly available information and further can disclose their identities. This type of privacy breach is known as identity disclosure in the literature.

In the last decade, more and more researches have focused on privacy-preserving data mining [START_REF] Agrawal | On the design and quantification of privacy preserving data mining algorithms[END_REF][START_REF] Clifton | Tools for privacy preserving distributed data mining[END_REF][START_REF] Evfimievski | Limiting privacy breaches in privacy preserving data mining[END_REF][START_REF] Evfimievski | Privacy preserving mining of association rules[END_REF][START_REF] Pinkas | Cryptographic techniques for privacy-preserving data mining[END_REF] . The aim of privacy-preserving data mining (PPDM) algorithms is to extract relevant knowledge from large amounts of data while protecting at the same time sensitive information [START_REF] Bertino | A Survey of Quantification of Privacy Preserving Data Mining Algorithms[END_REF]. Different techniques have been proposed so far, which include data modification, data encryption, and data anonymization [START_REF] Verykios | State-of-the-art in privacy preserving data mining[END_REF]. While data encryption requires a higher computing cost, data modification is widely used and includes a number of techniques such as data swapping, data shuffling, additive noise, etc. [START_REF] Heurix | Privacy-preserving storage and access of medical data through pseudonymization and encryption[END_REF]. Anonymization is the modification of data so that sensitive information remains private. Data anonymization techniques such as kanonymity has proven to be an effective means to reduce the risk of privacy disclosure via data inferences [START_REF] Farkas | The inference problem: a survey[END_REF].

In this chapter, we present several approaches developed for data anonymization for disclosure and inference control. More specifically, we focus on privacy violations via data inferences that occur when background knowledge is combined with non-private data. This knowledge could be stored in databases, ontologies, or might be available on the Web. Most of the works done in the literature to preserve anonymity on the Web assume that identifiable information and adversaries' background knowledge are needed to make inferences. Therefore, it is of importance to select relevant information about social network users that an adversary can attribute to them. Here comes the importance of online identity.

Online identity is an important issue in today's digital world as people are becoming more open to information exposure. We review existing approaches for public online identity construction. These approaches can be classified in three categories: Statistical approaches [START_REF] Irani | Large Online Social Footprints--An Emerging Threat[END_REF][START_REF] Chen | Is more always merrier?: a deep dive into online social footprints[END_REF][START_REF] Abel | Interweaving public user profiles on the web[END_REF], learning-based approaches [START_REF] Minder | Social Network Aggregation Using Face-Recognition[END_REF][START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF][START_REF] Malhotra | Studying User Footprints in Different Online Social Networks[END_REF] and ontological approaches [START_REF] Bojars | Towards Semantically-Interlinked Online Communities. Call for Take up Actions[END_REF][START_REF] Scerri | Knowledge Discovery in distributed Social Web sharing activities[END_REF][START_REF] Sauermann | Pimo-a framework for representing personal information models[END_REF][START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF][START_REF] Rowe | Mapping between digital identity ontologies through sism[END_REF]. Then we compare them and identify their strength and weakness. More specifically, we point out one major drawback related to the absence of the users' personal events in their approaches.

To overcome this drawback, we also explore event detection approaches in this study to build richer profile of social network users. First, we present existing event definitions and representations. Then, we classify existing algorithms for event detection from online social networks into two classes: event clustering and event hybrid approaches. Finally, we analyze they ability to consider links between events.

In this chapter, we first present related works in the area of inference detection and privacy preservation techniques. Then, we cover online identity construction approaches. Finally, we provide a review on different approaches related to event detection and linking from multimedia data (i.e., photos) in online social networks. In each subsection, we summarize, compare and discuss the limitations and features of the outlined approaches.

II. Inference Control

The need for privacy preservation in publicly released data is not new. In essence, anonymization on relational data has been studied for decades and has inspired many approaches and methods [START_REF] Ninghui | t-Closeness: Privacy Beyond k-Anonymity and l-Diversity[END_REF][START_REF] Machanavajjhala | /i>-diversity: Privacy beyond k-anonymity[END_REF][START_REF] Samarati | Protecting Respondents' Identities in Microdata Release[END_REF][START_REF] Sweeney | k-anonymity: a model for protecting privacy[END_REF] for social network anonymization. For this purpose, we start by presenting different approaches that have been proposed in the literature to handle identity anonymization in multimedia documents. These approaches can be categorized into three groups:

 Database inference approaches,  Ontology-based approaches, and  Web-based approaches.

II.1 Database Inference

Access control mechanisms are commonly used to provide control over who may access sensitive information. But in most cases, the problem is not access control, but inference control [START_REF] Farkas | The inference problem: a survey[END_REF][START_REF] Staddon | Web-based inference detection[END_REF]. In essence, malicious users may access innocuous data, and infer sensitive information from the received answers. To address this problem, a semantic inference model (SIM) is constructed in [START_REF] Chen | Database Security Protection Via Inference Detection[END_REF][START_REF] Li | Extracting the significant-rare keywords for patent analysis[END_REF] based on a domain semantic knowledge. SIM represents the dependent and semantic relationships among attributes of all the entities. The inference detection system proposed in [START_REF] Chen | Database Security Protection Via Inference Detection[END_REF] consists of three modules: knowledge acquisition, semantic inference model (SIM), and security violation detection including user social relation analysis, as shown in Figure 11. It utilizes both the user's current query and past query log to determine if the current query answer can infer sensitive information. An extension of the system can be achieved by considering the cases of multiple collaborative users based on the query history of all the users and their collaborative levels for specific sensitive information.

Figure 11:

The framework for an inference detection system proposed in [START_REF] Chen | Database Security Protection Via Inference Detection[END_REF].

Furthermore, most of the work addressing inference detection for database systems are based on functional dependencies in the database schema. The authors of [START_REF] Yip | Data level inference detection in database systems[END_REF] show that in order to detect inferences more adequately, the data itself must also be considered. To make this happen, five inference rules were identified: subsume, unique characteristic, overlapping, complementary, and functional dependency inference rules. A prototype has been developed to detect if a user can indirectly access data using two or more queries. Improvement in the system can be made by using distributed computing techniques to apply them on the queries in parallel or by employing any data access detection as an anomaly detection system.

In multilevel database, polyinstantiation occurs when different tuples with the same key, each at a different classification level, are allowed. However, there are problems limiting the use of polyinstantiation. To avoid these problems, a cover story is used for hiding the existence of a sensitive data. Therefore, a new technique is defined in [START_REF] Cuppens | Cover story management[END_REF] for managing cover stories free of semantic ambiguities, and offering a powerful security policy. The consistency and the security property of the multilevel database containing cover stories are also provided for the purpose of controlling database updates and deletions.

II.2 Ontology-based Inference

To formalize their privacy models, anonymization techniques [START_REF] Evfimievski | Privacy preserving mining of association rules[END_REF][START_REF] Verykios | State-of-the-art in privacy preserving data mining[END_REF][START_REF] Heurix | Privacy-preserving storage and access of medical data through pseudonymization and encryption[END_REF][START_REF] Sweeney | Achieving k-anonymity privacy protection using generalization and suppression[END_REF][START_REF] Sweeney | k-anonymity: A model for protecting privacy[END_REF] usually introduce adversary models which can conduct inference attacks in their systems. For instance, the works described in [START_REF] Geng | Privacy Measures for Free Text Documents: Bridging the Gap between Theory and Practice, in Trust, Privacy and Security in Digital Business[END_REF] and [START_REF] Ma | Preserving privacy on the searchable internet[END_REF] use ontologies to preserve the individual's privacy in free text documents where data structure is missing.

In [START_REF] Geng | Privacy Measures for Free Text Documents: Bridging the Gap between Theory and Practice, in Trust, Privacy and Security in Digital Business[END_REF], the authors measure sensitivity of identifiable information through a top-down propagation technique using prefixed sensitivity levels mapped to a reference ontology.

According to these computed sensitivity levels, words are disseminated. However, their approach considers only directly associated attributes (quasi-identifying entities, QIE) as keywords to a search engine. In essence, a common assumption in the privacy literature [START_REF] Zheleva | Privacy in social networks: A survey[END_REF] is that there are three types of possibly overlapping sets of personal attributes:  Identifying attributes: attributes, such as social security number (SSN), which identify a person uniquely,  Quasi-identifying attributes: a combination of attributes which can identify a person uniquely, such as name and address,  Sensitive attributes: attributes that users may like to keep hidden from the public, such as politic affiliation and sexual orientation.

In [START_REF] Ma | Preserving privacy on the searchable internet[END_REF], the authors use a probabilistic algorithm to mine all searchable information concerning individuals. They use domain-specific ontologies to capture inferable information and eventually provide more accurate results. They focus on all searchable information about users on the Web. Therefore, they define three categories of personal information with different privacy-sensitive degree: Identity information, privacy-Sensitive information, and other indirect information.

Unfortunately, the ability of these techniques to deal with adversaries enforced with plausible background knowledge is limited when using domain-specific ontologies to compute sensitivity levels. These so-called levels of sensitivity should depend mainly on the knowledge that the adversary already has acquired which could be out of scope of a specific ontology.

II.3 Web-based Inference

As we mentioned earlier, the problem of inference control comes from the fact that the knowledge that can be extracted from the union of the private and reference collections is greater than the union of what can be extracted separately from the two collections. To cope with this, it becomes important to determine what information can be released publicly without compromising certain secrets, and what subset of the information cannot be released.

With the explosion of user-contributed content on the Web, using online (multimedia) content can be helpful to detect inferences. However, it is exposed to uncertainty since it is originated from many sources. To overcome this imperfection, Schoenmackers et al. present HOLMES: a knowledge-based model construction [START_REF] Schoenmackers | Scaling textual inference to the web[END_REF]. To construct HOLMES, they combine information from multiple web pages to infer assertions not explicitly seen in the textual corpus. In order to produce the probabilistic network, they write inference rules as Markov logic Horn clauses [START_REF] Richardson | Markov logic networks[END_REF]. In [START_REF] Schoenmackers | Learning first-order Horn clauses from web text[END_REF], they proposed the SHERLOCK system to avoid labor and expertise in hand-crafting the appropriate set of Horn clauses for each new domain.

Word co-occurrence information is also important to extract personal information from the Web in different contexts. In [START_REF] Mori | Finding user semantics on the web using word co-occurrence information[END_REF], the authors propose a novel technique based on relevant occurrences to find user semantics (i.e., personal information) on the Web. Nevertheless, the amount of information is not always a relevant measure of dependency. For instance, two "tweets" with minimum co-occurrence might be issued by the same individual. Therefore, the authors in [START_REF] Gessiou | IRILD: An Information Retrieval Based Method for Information Leak Detection[END_REF] suggest that what may be of critical importance are phrases that return only a few results from search engines. Their method is based on fingerprints of information: cyclical hashing is used to split the document to multiple parts. They use search engines [START_REF] Podesta | Big Data: Seizing Opportunities Preserving Values[END_REF] to check the popularity of phrases and (2) to detect inferences between keywords. Their hypothesis is that if the search engine returns only a few results for the search, the phrase is then considered sensitive. The authors in [START_REF] Li | Extracting the significant-rare keywords for patent analysis[END_REF] consider both significant-frequent and significant-rare keywords.

Staddon et al. [START_REF] Staddon | Web-based inference detection[END_REF] initially based their work on an outside knowledge to detect clues of inference. They suggest a Web-based solution to control undesired inferences. They derive new knowledge by combining existing pieces of knowledge. They extract relevant keywords from the document to be published, and query the web in order to capture additional knowledge contributing to a privacy breach. Important improvements could include Web caching, besides an inference detection tool capable of functioning in real-time to significantly improve performance. In [START_REF] Chow | Detecting privacy leaks using corpus-based association rules[END_REF], the same authors refined their model of Webbased inference detection. They assume the existence of rules which specify how to derive new knowledge from the combination of existing pieces of knowledge. Their model finds all associations of a sensitive nature. They first search for the most frequent association of topicbased terms using search engines. Their findings are not limited to rules with large support. In essence, they also consider inference from low support patterns, since sensitive information has more likely a low support. In [START_REF] Staddon | A content-driven access control system[END_REF], they present an attribute-based encryption scheme for document redaction. They consider sensitive personal information, and also group of personal attributes that may indirectly allow inferring a person's identity.

In [START_REF] Chakaravarthy | Efficient techniques for document sanitization[END_REF], the authors present the notion of k-safety in which the identifying terms should be associated with at least k individuals. The authors in [START_REF] Nettleton | Document Sanitization: Measuring Search Engine Information Loss and Risk of Disclosure for the Wikileaks cables[END_REF] sanitize sensitive parts of the document to measure information loss and risk disclosure. They assume that a relevant sanitizing process could be applied to maintain the utility of information in the document.

Security requirements should also be achieved on XML documents since XML is adopted in data-publishing applications. Sensitive information could be leaked during XML publishing if common knowledge (constraints) is not considered carefully. Hence, the goal of the work in [START_REF] Yang | Secure XML publishing without information leakage in the presence of data inference[END_REF] is to protect sensitive information in the presence of data inference with common knowledge. This work formulates the process that users can infer data in XML publishing by using 3 types of common XML constraints: a child constraint, a descendant constraint, and a functional dependency. Towards that goal, one of the major challenges is finding a partial document of a given XML document without causing information leakage, while allowing publishing as much data as possible. It is also proved that there is a unique maximal document users can infer using the constraints, which contains all possible inferred documents.

II.4 Discussion

In Table 1, we summarize the different categories of inference control approaches with a description of the input and data types they use and the output they generate. [START_REF] Ninghui | t-Closeness: Privacy Beyond k-Anonymity and l-Diversity[END_REF][START_REF] Machanavajjhala | /i>-diversity: Privacy beyond k-anonymity[END_REF][START_REF] Samarati | Protecting Respondents' Identities in Microdata Release[END_REF][START_REF] Sweeney | k-anonymity: a model for protecting privacy[END_REF] to prevent information disclosure and eliminate possible linking attacks that are used for individual re-identification.

These techniques assume that identifiable information and adversaries' background knowledge are stored in structured relational datasets. Specifically, they address linking attacks that can be established between (quasi)-identifying and sensitive attributes of individuals stored in schema-based relational tables without referring to multimedia objects such as images and videos. In addition, ontology-based techniques use only domain-specific ontologies to capture inferable information.

Web based techniques propose web-based solutions to control undesired inferences. However, their ability to handle multimedia documents is limited. The only few techniques [START_REF] Geng | Privacy Measures for Free Text Documents: Bridging the Gap between Theory and Practice, in Trust, Privacy and Security in Digital Business[END_REF][START_REF] Ma | Preserving privacy on the searchable internet[END_REF] proposed to handle identity anonymization in multimedia documents assume textual data with no reference whatsoever to multimedia objects such as images and videos. To summarize, two main challenges are involved when addressing linking attacks on social networks:

1-Multimedia data: Social networks have become a popular way to disseminate different types of information (e.g., text, video, audio, photo, etc.). To prevent information leakage, privacy-preserving algorithms should consider multimedia data and their associated metadata.

2-Background knowledge: Data shared on social networks (e.g., photos, videos, statuses, etc.) may include private and sensitive information. Such information may be helpful to the adversary to discover additional information that the user did not provide in his online profile for privacy reasons, and thus may be needed for inference detection.

In our work, we cover not only personal information provided as text directly by the user on his profile. Our approach includes other information that are most likely related to the user but in an indirect way through multimedia content in his own posts as well as those shared by his contacts online.

III. Online Identity

The concept of identity on the Web is not new, although terminologies vary [START_REF] Irani | Large Online Social Footprints--An Emerging Threat[END_REF][START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF][START_REF] Malhotra | Studying User Footprints in Different Online Social Networks[END_REF][START_REF] Thiel | Navigating the Personal Information Sphere. 1st International Workshop on Model-based Interactive Ubiquitous System 2011[END_REF].

In the following, we first review some of the existing definitions of digital identity. Then, based on our classification, we describe data types used in each approach and show how they construct an individual's online identity and some of their limitations.

III.1 Online Identity Terminologies

The authors of [START_REF] Irani | Large Online Social Footprints--An Emerging Threat[END_REF] introduced the concept of Online Social Footprint (OSF). A user's online social footprint is the total amount of personal information that can be gathered about an online identity by aggregating his online social networks (OSN) profiles. To illustrate, Figure 12 shows a user named Bob Smith and his online social footprint, which was constructed with information from three social networking sites. The combination view of Bob's information from all these sites represents his online social footprint. The authors of [START_REF] Chen | Is more always merrier?: a deep dive into online social footprints[END_REF] used the concept of OSF to quantify the amount of information revealed across multiple OSNs. This description was followed by a definition of a Digital Footprint [START_REF] Malhotra | Studying User Footprints in Different Online Social Networks[END_REF] which represents the set of all personal information related to a user on online social sites. Information can be either provided by the user directly or extracted by observing the user's interaction within several social sites.

Based on background knowledge from Web 2.0 platforms, the authors of [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF] adopted a disambiguation approach to build a unary set of web resources containing personal information that refer to a given person and which they call Identity Web References.

The authors of the di.me project 32 defined the Personal Information Sphere (PS) which refers to all the digital forms of information related to a user and that are available on the user's personal devices and online accounts (e.g., files, folder structures, contact lists, photo albums, status messages, etc.).

Cortis et al. [START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF] define a user's online profile as the set of personal data stored on distinguishable online accounts (networking platforms, email, instant messaging, calendaring, task management, etc.). Personal data in these accounts vary from static identity-related attributes (e.g., name, location, picture-url, phone-number, etc.) to more dynamic information such as online posts on social networks. It also includes information about the user's contacts in his social network: their identities and posts they shared are also retrieved. All these kinds of personal data from one of the user's accounts form his "online profile", whereas the aggregation of his personal data from all his accounts forms his "super profile".

After reviewing various definitions found in the literature, we classify existing methods of building online identity for into three categories:  Statistical approaches,  Learning-based approaches, and  Ontological approaches.

III.2 Statistical Approaches

First, we present some statistical approaches that deal with the problem of online identity. In [START_REF] Irani | Large Online Social Footprints--An Emerging Threat[END_REF], the authors used online identity management sites to construct users' social footprints.

Furthermore, the authors of [START_REF] Chen | Is more always merrier?: a deep dive into online social footprints[END_REF] explored 40 types of attributes representing personal information available in public online profiles (e.g., age, email, links, location, name, religion, user name, etc.). The authors of [START_REF] Abel | Interweaving public user profiles on the web[END_REF] included images in their crawled profile attributes and tagging activity data. In the following, we will outline the methodology used in each of these approaches.

III.2.1 Online Social Footprint

First, online identity management sites are social networking sites where users can create unique profiles and provide links to all their social network profiles in order to manage their online identity (e.g., ClaimId 33 , FindMeOn34 , FreeYourID 35 , MyOpenID 36 ). The aim of the work in [START_REF] Irani | Large Online Social Footprints--An Emerging Threat[END_REF] is to show the threat of private information leakage by combining information about a user from multiple sites. Briefly, the authors aggregate personal information of a user from various social networks to construct his online social footprint. They assume the attacker has prior knowledge of one or more pseudonyms of the user. If not, the attacker attempts to infer it from the user's real name with a pseudonym guessing method. With this prior knowledge, they investigate the threat associated with an online social footprint by measuring (i) its size and (ii) the ability of an attacker to reconstruct it based on the user's pseudonym(s).

This study shed the light on the link between a user's pseudonym(s) and his personal information. The authors showed that an attack on targeted individuals is possible using their pseudonyms without having to use network based de-anonymization techniques [START_REF] Narayanan | Robust De-anonymization of Large Sparse Datasets[END_REF][START_REF] Narayanan | De-anonymizing Social Networks[END_REF].

Pseudonyms have the potential to reconstruct a user's social footprint. They can also be linked to the profile attributes of the user. While multimedia content is emerging on social networks, their study has the limitation of excluding images published on social networks. The generation of online social footprints includes only text-based attributes (e.g., birthday, homepage, hometown, location, name, sex, etc.). Hence, the information leakage model does not measure information that can be discovered from images. The process of matching profiles also ignores different types of posts such as free text entry fields, location and images.

In addition, the authors focus on inferring the pseudonym from the real name only. They don't consider the potential of inferring pseudonym using other personal attributes (e.g., location, hometown, birth year, etc.).

III.2.2 Increased Online Social Footprint

To further account for the threat to privacy in social networks, the authors of [START_REF] Chen | Is more always merrier?: a deep dive into online social footprints[END_REF] investigate various factors that contribute to increased online social footprint, previously introduced in [START_REF] Irani | Large Online Social Footprints--An Emerging Threat[END_REF]. They explored 40 types of attributes representing personal information available in public online profiles (Figure 13). The aim of this work [START_REF] Chen | Is more always merrier?: a deep dive into online social footprints[END_REF] is to show how aggregation of information from multiple OSNs can contribute to privacy loss. Three analyses have been done to examine and measure the amount of information that can be obtained by combining multiple OSN profiles.

First, the authors measured the average number of publicly available attributes in each of the OSN individually and when aggregated across multiple OSNs. Their findings show that the average online social footprint size steadily increases with the number of OSN profiles. This is due to three aspects:



The variety of distinct attributes among OSNs;



The diversity of privacy practices applied by the same user to the same information on different OSNs;



The correlation between the practice of having many online profiles and the tendency to share information online.

Second, the authors examined the online social footprints based on four user demographic information: i) gender, ii) country of residence, iii) age and iv) occupation. The authors did not find a significant correlation between footprint size and the first three attributes.

Nevertheless, the present study shows that the users' age can influence the tendency to share specific attributes 37 . The user's occupation can also influence the amount of sharing (e.g., an increased disclosure of personal information by users with customer-facing occupations).

Third, and similarly to [START_REF] Irani | Large Online Social Footprints--An Emerging Threat[END_REF], the authors of [START_REF] Chen | Is more always merrier?: a deep dive into online social footprints[END_REF] investigated the footprint size based on the use of pseudonyms. To do this, they applied string matching and regular expression techniques with Jaro-Wrinkler 38 distance metric to identify pseudonyms. According to their observations, users who use their real names (namely users on Facebook, Google and Flickr) have a larger footprint than those who use pseudonyms (namely users on Youtube, Blogger 39 and LiveJournal 40 ). 37 Older users are more likely to provide religion and political view on Facebook, whereas younger users are more interested in sharing their favorite music and books. ii. Consistency of information revelation: users have preferred patterns when revealing information across different OSNs.

iii. Consistency of attribute values: users' profiles have a significant matching between their attribute values at different levels of granularity.

The drawback of this method is that it focuses only on information entered by the user in his profile and does not capture personal information from his personal images. Exploring images in profiles would contribute to richer online social footprint, based on the metadata of the imageobjects, associated text and data. Adding metadata of the image to the cross-OSN analysis allows gathering further information.

III.2.3 Digital Footprints

In [START_REF] Malhotra | Studying User Footprints in Different Online Social Networks[END_REF], the authors generate the user's online digital footprints by aggregating personal information related to him across social networks. To do so, they propose an automated supervised classification approach for matching profiles belonging to the same user from the Twitter and LinkedIn profiles. Their system architecture is depicted in Figure 14. Matching profiles is computed using feature-specific similarity measures: string, location and image matching. The problem of this approach is that their image analysis is limited to lowlevel visual features (i.e., color), thereby ignoring objects in the image (e.g., faces, places, objects of interest, text, etc.), metadata and textual information accompanying the image.

Consequently the classification engine matches the pixel values between two images rather than information embedded in the image content and context (metadata files, description text, comments, tags, etc.). The matching criterions they use are restricted to statistical and engineering measures such as the Mean Square Error41 , Peak Signal-to-Noise Ratio 42 and the Levenshtein distance 43 . Thus, their classification does not rely on any semantic interpretation of the image.

In addition, they considered the number of connections of users for disambiguating the user profiles across social networks. This follows the intuition that a same user should have a similar number of friends in different social networks. This is generally not true due to the different nature of the services. However, examining online posts of the connections (friends) of a user is not considered in their study. Actually, information leakage is not strongly related to self-disclosure, as pointed in [START_REF] Lam | Involuntary Information Leakage in Social Network Services[END_REF]. Friends may reveal a user's identifying information unintentionally no matter how much the user tries to protect his identity online. Thus, attackers do not just look at the user's profiles but also at friends of the user [START_REF] Friedland | Sherlock holmes' evil twin: on the impact of global inference for online privacy[END_REF] (friends' profiles and posts, and more particularly friend's photos). Second, the limitation of the comparison method is that it does not involve cross-feature comparison: they compare values only between two same features from two profiles and do not associate values from different features across the profiles. For instance, the "location" feature extracted from a first profile can be matched with an element in the "description" feature in the second profile. Third, in their first analysis on data collected from Twitter, Youtube, Flickr and LinkedIn, they found that at least 80% of the profiles contain a profile picture. Results show much higher percentages (99%) of Twitter and Flickr users providing their profile images due to the nature of these services. The high percentages suggest that profile images would be needed for the digital identity construction, which is in line with the assumptions in our study.

III.3 Learning-based Approaches

Image features were used in several learning-based approaches [START_REF] Minder | Social Network Aggregation Using Face-Recognition[END_REF][START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF][START_REF] Malhotra | Studying User Footprints in Different Online Social Networks[END_REF]. For instance, in the Digital Footprint approach [START_REF] Malhotra | Studying User Footprints in Different Online Social Networks[END_REF], a classification engine is trained with feature vectors including string attributes, location attributes and images. As for the authors of [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF], they present a semi-supervised approach to automatically disambiguate the identity web references for a specified person. We will review their approach in what follows.

III.3.1 Identity Web References

A web resource, as defined by the authors, is any document on the web which is accessible by a URI such as web pages or data feeds. A web resource that contains personal information of a web user is an identity web references of the user. The aim of the approach proposed in [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF] is to find all the web resources residing on the web which are identity web references of a given person. They rely on metadata descriptions from web resources (e.g., name, email, website, location, etc.) with no consideration of image metadata that can also contain personal information.

After retrieving the web resources that point to a particular person, it is important to know if they actually refer to that person. In order to classify the web resources as positive or negative sets, the authors of [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF] proposed two disambiguation techniques: The second disambiguation technique applies a Random Walks [START_REF] Lovász | Random walks on graphs: A survey[END_REF] approach on the interlinked graph space obtained from the seed data and the set of web resources. Using distance measures on the graph, this technique clusters the web resources into positive and negative in order to find the identity web references. The overview of their approach is depicted in Figure 15: Overview of the approach proposed in [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF] to disambiguate identity web references using Web 2.0 data..

Figure 15:

Overview of the approach proposed in [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF] to disambiguate identity web references using Web 2.0 data.

In this approach, disambiguation techniques achieve consistent accuracy with all web presence levels. Yet, regarding content types (images, videos, audios) of the web presence is essential given the trend of publishing and sharing multimedia information on social web sites and online communities. In addition, the background knowledge used for learning is generated by exporting and combining data from multiple heterogeneous online accounts such as Facebook, LinkedIn and Twitter. However, the use of these Web 2.0 platforms in this approach does not cover the maximum knowledge about a web presence. Indeed, metadata extracted from online images of a given person is able to reproduce larger presence of the person on the web. The disambiguation approach must be able to cope with multimedia resources, such as images as seed data to start the identification process.

III.3.2 Enhanced Identity Merging based on Face Recognition

Another approach [START_REF] Minder | Social Network Aggregation Using Face-Recognition[END_REF] using images shows that profile images can provide additional power in the user re-identification. In [START_REF] Minder | Social Network Aggregation Using Face-Recognition[END_REF], the authors combine traditional text-based and imagebased approaches to enhance available user re-identification systems for social network data aggregation. Their method uses a face-recognition algorithm to re-identify user profiles based on their profile images. The advantage of this method is that it incorporates images with traditional text-based. Nevertheless, this method uses face-recognition software to compare the images uploaded by users without considering image metadata. An overview of their method is illustrated in Figure 16.

Figure 16: Approach used in [START_REF] Minder | Social Network Aggregation Using Face-Recognition[END_REF] for data aggregation.

III.4 Ontological Approaches

Different works [START_REF] Bojars | Towards Semantically-Interlinked Online Communities. Call for Take up Actions[END_REF][START_REF] Scerri | Knowledge Discovery in distributed Social Web sharing activities[END_REF][START_REF] Sauermann | Pimo-a framework for representing personal information models[END_REF][START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF][START_REF] Rowe | Mapping between digital identity ontologies through sism[END_REF] used ontologies to attribute semantics to the online identity. We start by providing a brief overview of ontologies designed for managing user presence on the Semantic Web, then we detail the most relevant ones.

Some ontologies, initially designed for desktop personal data in the NEPOMUK project [START_REF] Groza | The nepomuk project-on the way to the social semantic desktop[END_REF],

are useful for personal data available on the Semantic Web. Examples of these ontologies are:

the NEPOMUK contact ontology 44 (NCO), the Information Element ontology 45 (NIE), the Personal Information Model Ontology 46 (PIMO), the Account Ontology 47 (DAO) and the Annotation Ontology 48 (NAO).

Other ontologies were developed for describing personal profile. The FOAF vocabulary 49 represents individuals and relationships between them. It defines the typical attributes that describe the individuals' identity, their online coordinates and basic web activities [START_REF] Brickley | Foaf vocabulary specification 0[END_REF]. Then, the Semantically-Interlinked Online Communities (SIOC) ontology [START_REF] Breslin | Towards semantically-interlinked online communities[END_REF] reused the FOAF vocabulary for exploring personal profile attributes and social networking information (the content the user produces and the actions of other users on this content).

A mapping approach between digital identity ontologies, named the Social Identity Schema Mapping (SISM) is proposed in [START_REF] Rowe | Mapping between digital identity ontologies through sism[END_REF] to address the problem of information heterogeneity.

The LivePost Ontology (DLPO) is designed in [START_REF] Scerri | Knowledge Discovery in distributed Social Web sharing activities[END_REF] The most relevant ontologies used are detailed below.

III.4.1 The Personal Information Model (PIMO)

The Personal Information Model of a user (PIMO) [START_REF] Sauermann | Pimo-a framework for representing personal information models[END_REF] was proposed in the NEPOMUK project [START_REF] Groza | The nepomuk project-on the way to the social semantic desktop[END_REF] and is now maintained by OSCAF members as a response to the need of having a formal representation of the users Mental Model 50 . The EPOS project uses the PIMO to create the user's own domain ontology, called PIMO-User [60]. Personal concepts, ideas, projects, contacts and other resources that are of direct interest to the user represent the user's Personal Information Models (PIM). The PIMO 51 ontology is simultaneously an RDF vocabulary to express these models and an upper ontology defining basic classes and properties to use. This ontology satisfies the following requirements: (1) precise representation, (2) extensibility, (3) interoperability, (4) reuse of existing ontologies, and (5) data integration [61]. The focus in PIMO is on data that is accessed through a Semantic Desktop or other personalized Semantic Web applications.

III.4.2 The Nepomuk Contact Ontology (NCO)

With the OSCAF/NEPOMUK ontologies, the user can cope with different formats of data.

For instance, contact information is described in the NEPOMUK contact ontology 52 (NCO).

The meaning of the term "Contact" in NCO is quite wide. It is every piece of data that identifies an entity or provides means to communicate with it.

III.4.3 FOAF

FOAF is an ontology 53 defined in the FOAF project 54 to represent individuals and relationships between them. It defines the typical attributes of an agent that describe his identity, online coordinates and basic web-based activities. The FOAF ontology is a potential 50 PIMO is based on the idea that users have a mental model to categorize their environment. tool for the extraction and the fusion of personal information: on one hand, it enriches the expression of personal information and relationships represented in RDF/XML as pointed in [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF], including name, mailbox, homepage URL, friends, and so on. On the other hand, it provides a standardized format [START_REF] Martín-Serrano | Telemaco: Context-aware System for Tourism Guiding based on Web 3.0 Technology[END_REF] using the Ontology Web Language (OWL), thus

providing the capability to aggregate information from different sources.

III.4.4 Social Identity Schema Mapping (SISM)

The Social Identity Schema Mapping (SISM) [START_REF] Rowe | Mapping between digital identity ontologies through sism[END_REF] is a mapping vocabulary between five digital identity ontologies: The Friend of a Friend ontology 55 (FOAF), the ontology for VCards 56 , the XFN ontology 57 , the Personal Information Model Ontology 58 (PIMO) and the Nepomuk Contact Ontology 59 (NCO). It aims at mapping concepts from these ontologies in an interpretable and machine-readable format. The basis mapping in SISM is a subjectpredicate-object triple: a (parsed) source concept has a relation with a (known) target concept, where the relation describes the semantics of the mapping. To fully capture such semantics, they combine mapping constructs from the OWL 60 and the SKOS 61 languages. While the OWL language allows to describe the semantics of knowledge with concepts and relationships between concepts, the SKOS ontology is a vocabulary for organizing concepts with lightweight semantic properties (e.g., skos:narrower, skos:broader, skos:related).

Relations 62 defined in the SISM ontology enable to interpret in a common and precise way, metadata models containing identity information on the Web. SISM supports inference rules to normalize metadata models. Consequently, the SISM vocabulary enables the interlinking of identity fragments distributed on different social web platforms. SISM is used in [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF] for normalization of metadata from different social web platforms for producing the Identity Web

III.4.5 The LivePost Ontology (DLPO)

The LivePost Ontology (DLPO) [START_REF] Scerri | Knowledge Discovery in distributed Social Web sharing activities[END_REF] is an open knowledge representation standard which models online posts shared across social networks. The main purpose of the DLPO is to capture implicit presence knowledge embedded in the social web sharing activities (e.g., online posts, online interactions, and online Social Web practices, etc.). This ontology is part of the di.me project [START_REF] Scerri | digital. me towards an integrated Personal Information Sphere[END_REF] which aims at unifying the user's personal information across various heterogeneous sources (Figure 17). ii. user posts and user presence 68 ;

iii. user posts and global semantic data clouds 69 . Table 2 shows a comparison of some digital identity ontologies with respect to the multimedia dimension and the data covered. The hand symbol in this table (and the others) means that the studied approach takes into consideration the given criteria. FOAF [START_REF] Brickley | Foaf vocabulary specification 0[END_REF] SIOC [START_REF] Breslin | Towards semantically-interlinked online communities[END_REF] SISM [START_REF] Rowe | Mapping between digital identity ontologies through sism[END_REF] DLPO [START_REF] Scerri | Knowledge Discovery in distributed Social Web sharing activities[END_REF]

III.4.6 Integration of multiple ontologies

The authors in [START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF] focus on the integration of user profiles from multiple heterogeneous online accounts such as Facebook, LinkedIn and Twitter. Since it is possible to have multiple occurrences of the same data across different accounts, providing an abstraction for data representation is needed to eliminate duplicate instances. To do so, they propose an ontologybased approach to detect semantically-equivalent user profiles, illustrated in Figure 18. User profiles are mapped into one ontology framework consisting of a set of re-used, extended as well as new vocabularies provided by the OSCA Foundation 70 (OSCAF). More specifically, they used the Personal Information Model (PIM), provided by PIMO, as the knowledge base of their approach. The PIM contains data that is of direct interest to the user. Initially, it is automatically populated by crawling data on the user's online accounts or devices. Further, the user can extend it manually by adding the representations of his own mental model. Since it is the user's own PIM, such knowledge base contains the most valuable information about the user. It is certainly smaller, leading to more accurate results. The approach process proposed in [START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF].

To conclude, this work [START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF] proposes an automated approach for the integration of several identities that are distributed across multiple local and remote sources. This approach facilitates the management of the multiple identities of a user with minimal effort. However, it has three limitations:

The NCO and the DLPO ontologies are able to express information extracted from images.

First, in the NCO ontology, a photo (nco:Image) can be attached to the class Contact. Second, in the LivePost ontology, an image post (dlpo:ImagePost) can generate an instance of the MultimediaPost class.

In addition, the authors consider semantic relatedness only among text entities in the profile attributes to compute the user profile matching. The matching between the user's profile data and the user's PIM is limited to text (e.g. surname, username, address, country, etc.).

Although the NCO and the DLPO ontologies handle image data, this work [START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF] retrieves only textual attributes from a user profile, excluding multimedia content available online. The matching should also be extended to include images and consider semantic information from their content (e.g., faces). Images can be either available in the online profile of the user or his contacts.

Furthermore, metadata matching targets only resources from a user's profile but lacks the exploitation of metadata that is attached to shared items (e.g., location coordinates from a posted image). We believe that adding the Nepomuk Exif Ontology (NEXIF) to the ontology framework would be useful for capturing information from the EXIF metadata of the user's images. Furthermore, NEXIF 71 allows the EXIF metadata to be expressed in RDF.

III.5 Discussion

So far, we presented existing approaches regarding online identity construction, and described both their advantages and limitations. In Table 3, we summarize these approaches with a description of their features and limitations. Online Social Footprint [START_REF] Chen | Is more always merrier?: a deep dive into online social footprints[END_REF] Digital Footprint [START_REF] Malhotra | Studying User Footprints in Different Online Social Networks[END_REF] Identity Web References [START_REF] Rowe | Disambiguating identity web references using Web 2.0 data and semantics[END_REF] Personal Information Sphere [START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF] Tagbased Profile Some methods [START_REF] Minder | Social Network Aggregation Using Face-Recognition[END_REF] use key facial features to incorporate the visual content of images.

However, they do not extract users' semantics from metadata attributes behind images (e.g., location, time, etc.). Most of these approaches ignore the potential of obtaining information from events.

Some approaches [START_REF] Bojars | Towards Semantically-Interlinked Online Communities. Call for Take up Actions[END_REF][START_REF] Scerri | Knowledge Discovery in distributed Social Web sharing activities[END_REF][START_REF] Sauermann | Pimo-a framework for representing personal information models[END_REF][START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF][START_REF] Rowe | Mapping between digital identity ontologies through sism[END_REF] used several ontologies for semantically describing the digital identity. Considering the complex and unstructured information, the use of ontologies is important to explicit the concepts and their relationships between the user data. Ontologies have to include various content types of a web presence such as image representations and metadata.

In conclusion, while existing approaches investigate the ability of obtaining personal information using Web 2.0 data, two key differences distinguish our work:

 We use image metadata to build richer user profile information from social networks.

 We detect personal events in photos shared by the user or his connections in a social network in order to discover additional information that might not be included in the user profile.

IV. Events in Online Social Networks

Photos taken during events might contain references to personal data that the user may have not intended to disclose online. In this section, we review previous definitions in the literature, and then we present the problem of detecting events from photos shared in online social networks.

IV.1 Event Definitions

The first approaches of event detection proposed in the literature were associated with the Topic Detection and Tracking (TDT) [START_REF] Allan | Introduction to topic detection and tracking, in Topic detection and tracking[END_REF]. The objective of TDT is to detect topically related stories within a stream of news media [START_REF] Angajala | ECO: Event Detection from Clickthrough Data via Query Clustering[END_REF]. Basically, an event identifies something that happened in a certain place at a certain time [START_REF] Yang | Learning approaches for detecting and tracking news events[END_REF]. This definition was adopted within the TDT project, then was developed to detect events from news stories coming from social media [START_REF] Nallapati | Event threading within news topics[END_REF][START_REF] Qin | Feature-Rich Segment-Based News Event Detection on Twitter[END_REF]. For instance, Knowle is a news event centrality data management system for organizing news events on the Web [START_REF] Xu | Knowle: A semantic link network based system for organizing large scale online news events[END_REF]. A news event in the Knowle system is defined by a life course and a set of features describing the event.

With the emergency of multimedia applications, the concept of event is becoming one of the most challenging objectives in the semantic Web, especially the social Web.

Many different definitions of events have been proposed to process multimedia data [START_REF] Makkonen | Applying semantic classes in event detection and tracking[END_REF],

most of which are based on the spatial [START_REF] Wang | Social event detection with interaction graph modeling[END_REF][START_REF] Zaharieva | Automated social event detection in large photo collections[END_REF][START_REF] Liu | Using social media to identify events[END_REF] , temporal [START_REF] Xu | Knowle: A semantic link network based system for organizing large scale online news events[END_REF][START_REF] Rattenbury | Towards automatic extraction of event and place semantics from flickr tags[END_REF] or spatio-temporal aspects [START_REF] Mahata | Learning from the crowd: an evolutionary mutual reinforcement model for analyzing events[END_REF][START_REF] Allen | Actions and Events in Interval Temporal Logic[END_REF][START_REF] Valkanas | How the live web feels about events[END_REF]. Currently, there are several ways of defining an event in the literature.

Rattenburt et al. [START_REF] Rattenbury | Towards automatic extraction of event and place semantics from flickr tags[END_REF] regard an event as a specific segment of time. Paniagua et al. [START_REF] Paniagua | Social events and social ties[END_REF] represent an event as a pair of temporal and spatial information. The definitions by Mahata et al. [START_REF] Mahata | Learning from the crowd: an evolutionary mutual reinforcement model for analyzing events[END_REF] and Valkanas et al. [START_REF] Valkanas | How the live web feels about events[END_REF] claim that an event is a real-world incident or phenomenon, that occurred at some specific time (or over a certain period of time), and is usually tied to a location. Sakaki et al. [START_REF] Sakaki | Earthquake shakes Twitter users: real-time event detection by social sensors[END_REF] refer to event as an arbitrary classification of a space/time region.

It might have actively participating agents, passive factors, products and a location in space/time. Gupta et al. [START_REF] Gupta | Evaluating Event Credibility on Twitter[END_REF] identify an event using a set of required core words and a set of optional subordinate words. In [START_REF] Sarma | Dynamic relationship and event discovery[END_REF], an event is defined as a 3-tuple <E, R, t>, where E is a set of entities, R  E x E is a set of dynamic relationships, and t is a continuous time window.

This definition enables discovering and verifying dynamic connections among entities that are connected at a given time period. These dynamic relationships are then used to derive dynamic events by identifying and enriching them with further information.

Other definitions have provided categories for events. For example, events can be classified as local when they refer to personal experiences (e.g., wedding, birthday celebrations, etc.), or global events (e.g., sport competitions, concert, natural disasters, etc.) [START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF]. Global events allow building collective experiences for sharing personal experiences as part of a social phenomenon called collective events [START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF]. Other categories are also provided in [START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF][START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF] such as home and away-from-home events, routine and non-routine events in order to recognize the basic nature of an event. Figure 19 shows a distinction of personal and social event with regard to the memory and experience.

Figure 19: Distinction between personal events and social events, as defined in [START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF].

As a conclusion, there is no common definition of events established for multimedia applications. The multimedia event model presented in [START_REF] Westermann | Toward a Common Event Model for Multimedia Applications[END_REF] shows the need of having a common multimedia event model for a wide diversity of applications, event-centric multimedia, reusable event management infrastructure and tools, and application integration.

A common event model should be able to identify six distinct aspects of event description: temporal, spatial, informational, causal, structural, and experiential. Figure 20 illustrates these aspects as well as the determinants of each aspect. Also, many relationships can be expressed by using a suitable event model such as events' temporal and spatial relationships as well as structural and causal relationships.

IV.2 Event Detection from Photos in Online Social Networks

A large number of studies investigated the detection of events from multimedia data including text [START_REF] Abdelhaq | EvenTweet: online localized event detection from twitter[END_REF][START_REF] Bauer | Event based classification of Web 2.0 text streams[END_REF][START_REF] Becker | Beyond Trending Topics: Real-World Event Identification on Twitter[END_REF][START_REF] Kumar | From Tweets to Events: Exploring a Scalable Solution for Twitter Streams[END_REF][START_REF] Liu | Exacting Social Events for Tweets Using a Factor Graph[END_REF][START_REF] Marcus | Twitinfo: aggregating and visualizing microblogs for event exploration[END_REF], image [START_REF] Liu | Using social media to identify events[END_REF][START_REF] Paniagua | Social events and social ties[END_REF][START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF][START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF][START_REF] Bao | Social event detection with robust high-order co-clustering[END_REF], and video [START_REF] Ballan | Event detection and recognition for semantic annotation of video[END_REF][START_REF] Gkonela | VideoSkip: event detection in social web videos with an implicit user heuristic[END_REF][START_REF] Mertens | Acoustic super models for large scale video event detection[END_REF][START_REF] Steiner | Crowdsourcing event detection in youtube videos[END_REF]. In our study, we are interested in approaches exploring image data. In essence, extracting events from multimedia in terms of photographs or images is much more difficult when compared to text. This is essentially due to two reasons, according to the authors of [START_REF] Yan | A comprehensive study of visual event computing[END_REF][START_REF] Scherp | Survey on modeling and indexing events in multimedia[END_REF]. First, event detection from images requires aggregation of heterogeneous metadata. Second, linking multimedia data to event model aspects is far more challenging than textual data. We focus here on methods for the detection of events from images taken from the real-life of an individual and shared on online social networks (i.e., personal events). In the following section, we discuss the main approaches and how they differ from our approach. We classify them in two groups: i)

clustering approaches [START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF][START_REF] Bao | Social event detection with robust high-order co-clustering[END_REF][START_REF] Chen | Event detection from flickr data through wavelet-based spatial analysis[END_REF][START_REF] Firan | Bringing order to your photos: event-driven classification of flickr images based on social knowledge[END_REF][100][101][102] and ii) non-clustering approaches [START_REF] Xu | Knowle: A semantic link network based system for organizing large scale online news events[END_REF][START_REF] Wang | Social event detection with interaction graph modeling[END_REF][START_REF] Zaharieva | Automated social event detection in large photo collections[END_REF][START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF]103,104].

IV.2.1 Clustering Approaches

New approaches have emerged in the past few years in the area of event detection from images on online social networks. For instance, the authors of [START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF][START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF] use the context allied with social media content for the event detection.

The methodology in [START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF][START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF] uses the context of social media content, user provided tags and significant terms for each event from the Internet as features for event detection. Their method, illustrated in Figure 21, groups the images to events simultaneously. The authors of [START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF] refer to context as any information that can be captured by a camera, or extracted from related images in the same photo collection or any textual description provided by the user. The aim of their work is to detect and arrange events in private photo archives by building a contextual meaningful hierarchy of events. Therefore, they suggest an automatic image clustering that merges visual data with context information (time, space). In a subsequent work [START_REF] Paniagua | Social events and social ties[END_REF], the same authors seek to group events coming from different users into richer social events based on two metadata features (time, space). More specifically, they show how event detection can bring together users who participated in the same event and hence propagate social connections among users.

Similarly, the authors of [START_REF] Bao | Social event detection with robust high-order co-clustering[END_REF] investigate photos posted on social media sites to detect social events. They use four types of heterogeneous metadata related to photos: time-stamp, location, visual data and textual description. First, they represent the social media and their metadata into a star-structured K-partite graph. Second, they model relationships between social media and relationships between metadata sets. Then, they apply a co-clustering method on the star structure.

In the work presented in [105], the authors propose a general and scalable online clustering framework for identifying events and their corresponding social media documents. In essence, social media sites represent a valuable source of event information, but this information is far from uniform in quality and might often be misleading or ambiguous. To address this problem, they use contextual data of social media documents and define similarities tailored to the social media domain. The similarity metric is the cosine similarity metric for textual features (e.g., title, description, etc.), and the Haversine distance for location metadata.

Several existing techniques are applied in this work to learn multi-feature similarity metrics for social media documents. Since individual features might be noisy or unreliable, they suggest that all features, considered collectively, can provide more reliable information about events. Therefore, to improve the efficiency of the document clustering, they use first ensemble-based and then classification-based techniques to learn a combined similarity metric.

IV.2.2 Non-clustering Approaches

The authors of [START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF] refer to context as the dateTaken, title, description, tags (user comments), and latitude and longitude information. In their approach, they propose a method for grouping together photographs of similar events using significant terms for each event from

Wikipedia 72 and Google 73 . However, they do not consider geolocation in their analysis because the geographic information is not always available in real-world dataset.

The authors of [START_REF] Liu | Using social media to identify events[END_REF] detect events by temporally monitoring the social media sharing activity at specific locations. They use metadata attached to photos (tags, description, and geographic location) to enrich the event dataset and infer the topic of events. This work shows similarities with our work because of the metadata interest, but they do not detect links between events.

IV.2.3 Discussion

In Table 4, we compare the above event detection approaches.  Insufficiency in exploring all image metadata: Most of these approaches do not include the creation metadata (e.g., creator name, geographic information, etc.). Some approaches consider visual data [START_REF] Liu | Using social media to identify events[END_REF][START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF][START_REF] Bao | Social event detection with robust high-order co-clustering[END_REF] by extracting only low-level features (e.g., color). Our model is able to make use of objects found in images, particularly faces of persons, since we assume that photos are annotated with person-tags.

 Deficiency of the multi-* property of events: Existing approaches do not apply to particular cases such as events that occurred many days, or events that took place in many cities, regions or countries. In our study, we address these particular cases by using the multi-* property of events, namely multi-day, multi-site, multi-source and multi-participant events.

 Restriction to the user's personal profile: Most of the existing approaches [START_REF] Liu | Using social media to identify events[END_REF][START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF][START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF][START_REF] Bao | Social event detection with robust high-order co-clustering[END_REF] limit the extraction of images to the concerned user's profile only. In our approach, we exploit personal events in the individuals' own posts as well as those shared by their friends/contacts.

 Lack of event links: Existing approaches do not consider the different types of links between events that we propose in our work (temporal, spatial, social, and semantic). In the following, we review different approaches that have been developed to identify relations between events.

IV.3 Events Linking Approaches

In this section, we focus on event linking approaches [START_REF] Sarma | Dynamic relationship and event discovery[END_REF][START_REF] Westermann | Toward a Common Event Model for Multimedia Applications[END_REF]106-109] that allow to identify links between events. We summarize and discuss their limitations and features. Then, we try to highlight the difference between our approach and the existing ones.

The challenges that are faced when modeling real-world events captured in multimedia documents as discussed in [109] are the following: i) Event semantics are dynamic and might change over time, across people, and depending on the context.

ii) People may upload and tag a multimedia document without associating it to events, which creates the problem of resolving the semantics of the document.

iii)

The relationship between the multimedia document and the event is driven by the user's need and hence may be defined by people's own words, languages and expressions.

In order to address these problems, the authors of [109] introduced Eventory, a media archive of events corresponding to activities and events of an academic community. Eventory allows the following operations: i) multimedia document upload, ii) event creation and iii) event Events are entity-based (an entity refers to any real world concept such as a person, a company, etc.), and concerned with dynamic relationships. This leads to the entity dynamic relation graph. Their approach starts by building the temporal profile of each entity by evaluating the number of documents that mention this entity within a specific time window.

Candidate dynamic relationships are then detected using temporal profiling, co-occurrence and peak detection techniques [110]. The techniques employed apply to any granularity of event consolidation, though their experiments are based on weekly time windows. In the next step, they develop event identification algorithms to extract the dynamic events based on entity clusters with two temporal constraints (global and local) that they defined. Finally, they enrich events with three additional information: 1. The entity involvement score that measures the overall entity participation in the event; 2. The event confidence score that measures the probability of connectedness of two entities; and 3. Descriptive information about the events, including event textual description and event popularity 74 . 74 Event confidence refers to the confidence their system has of having correctly produced an event, while, event popularity refers to ranking events based on external importance assuming the event is correct.

In [106], the authors focus on the representation of video events and present a framework for semantic annotation of video, called EDF. The goal of EDF is to capture event semantics that enable storage, inference and retrieval of events from lower level event observations. The authors defined two classes of events: i) primitive event (comprised of a set of actors and actions), and ii) composite event (comprised of other primitive events). This framework allows a hierarchical decomposition of complex events into simpler ones. Based on top-level ontology, it supports reasoning and inference of composite events and relationships from the primitive ones. The ontology presented here consists of a set of predicates for describing spatio-temporal relationships between events and entities. The authors referred to Allen's Interval Algebra [START_REF] Allen | Actions and Events in Interval Temporal Logic[END_REF] to express temporal relations between two events. The spatial relationships in EDF are based on the typical spatial relations between any two objects: topological relationships [111], directional relationships [112], and distances (constraints on spatial metrics such as distance < 100).

EDF E [113] is an extended version of EDF developed to facilitate the event detection process and reasoning about events. The extended version provides two additional features: i) the temporal predicate granularity as a feature of temporal association between sub-events, and ii) the event evidence to capture the full evidence for the detected events. EDF E was applied to model complex events from real world surveillance videos in a retail store.

The authors of [108] focus on composite events based on the Event Model, called Emodel [START_REF] Westermann | Toward a Common Event Model for Multimedia Applications[END_REF]. However, composition operators are here restricted to only two facets: spatial and temporal. The authors subsequently defined a set of temporal and spatial operators independent of domains to represent the possible semantics of a composite event. Then, they applied a union aggregation technique to propagate properties of atomic events to composite events, and defined the attributes of composite events as a function of their sub-events. For example, Figure 23 shows two events e1 and e2 are composed via spatial and temporal operators to form composite event E. In order to adequately provide the framework for composing events, the authors construct two graphs: i) a SpatialGraph using the RCC-8 spatial relationships [114], and ii) a

TemporalGraph using Allen's temporal relationship. Nevertheless, this work does not consider the social aspect of events (e.g., social relationships).

In a recent extension of their work [115], the same authors defined a generic ontological model, built on an event ontology, to infer sub events and their characteristics from personal photos using image metadata. This model is used to describe the vocabulary of a general domain event (such as trip) based on the E* model [116], which is an RDF-based data model to represent spatial, temporal and thematic relationships between data objects. Many properties are found in this semantic event model representing general relationships such as composition with subeventOf, similarity with same-as, spatiotemporal relationships like occurs-during, occurs-at, co-occurring-with, and co-located-with, etc. The authors also constructed a semantic language to model different types of entity properties and relationships related to an event domain on the basis of OWL. They added context information from heterogeneous data sources in order to provide very flexible models for high-level semantics of events, such as complex temporal formulas and spatial constraints and functions that cannot be expressed in OWL.

A number of studies have demonstrated the usefulness of social media in collecting data for emergency management [117][118][119][120][121][122]. For instance, the authors of [107] focus on real large-scale events (e.g., floods), which are important in emergency management. This study is useful for providing general overview information about events during a disaster or for afterthe-fact analysis for training purposes. The authors presented a framework that allows the analysis of multimedia data using metadata (e.g., tags and title) associated with content found on social media platforms (Figure 24). They focus on identifying sub-clusters, which point to different hot spots in the emergency region. Subevents are defined here as events during a disaster which are separated from other events w.r. showed the suitability of their clustering algorithm for detecting sub-events. However, this is an offline approach where the optimal terms are selected before clustering. Therefore, its drawback is that it that cannot be directly deployed in real-time during an emergency, and does not take geo-referenced data or date/time information into account. 75 https://www.youtube.com/?gl=FR In a later work, the same authors suggested a clustering approach to detect crisis-related subevents online in real-time [123]. They have addressed the problem by referring to the topic detection and tracking (TDT) [START_REF] Nallapati | Event threading within news topics[END_REF]. Events are similar to topics (and therefore sub-events to sub-topics), but different in that crisis-related events can evolve quite easily. Thus, the authors extended their first approach and explicitly included data containing geo-referenced information (i.e., longitude and latitude coordinates) to assist disaster management with

Twitter, Flickr and YouTube data in real-time. Their approach relies on an online/incremental feature selection mechanism combined with an online clustering algorithm. The feature selection is dynamic, as a weighting function is used to calculate the importance of a term over different periods. It is based on the learning and forgetting model. The clustering algorithm is adapted to the Growing Gaussian Mixture Models (2G2M) algorithm [124]. The problem of this approach with respect to our goal of identifying relationships between events is that it identifies one specific type of events: sub-events. It has a spatial-temporal focus on the data but does not identify the spatial-temporal relations between events and sub-events.

IV.3.1 Discussion

In Table 5, we summarize the current event linking approaches and compare their characteristics. Social networks have become a large repository for detecting events from real world through user's published data. Previously, different solutions have been suggested to detect events [START_REF] Liu | Using social media to identify events[END_REF][START_REF] Paniagua | Social events and social ties[END_REF][START_REF] Sheba | Event Detection Refinement Using External Tags for Flickr Collections[END_REF][START_REF] Tankoyeu | Event detection and scene attraction by very simple contextual cues[END_REF][START_REF] Bao | Social event detection with robust high-order co-clustering[END_REF]105,125] and few of them try to discover relationships among events [81, 106-109, 115, 123]. We believe that metadata contained in multimedia data, such as photos, could be of great importance for the event-linking task.

The main problems of these solutions with respect to our goal of computing links between events can be summarized as follows:

-Focus on spatio-temporal relations: Existing approaches do not account for all aspects of an event. Indeed, most of them focus on temporal relationships. Spatial relationships are sometimes considered. Nevertheless none of these approaches expresses the relations between events on the basis of the type of the relationships that link the event's participants.

-Lack of granularity: Most of these approaches do not incorporate granularity values for the temporal and spatial relations that could be identified between events.

-Capability of detecting complex events: Some of existing approaches allow the identification of complex events, especially sub-events. They are based on ontologies. Therefore, they do not allow the relations among events to be known if the ontology has not first been defined.

-Possibility of discovering semantic relations: Current approaches do not consider semantic relations at all. For one approach only [109], the system includes semantic constraints. This was done to allow the user to define his own semantic relationships.

V. Conclusion

Many social networks users are privacy conscious and may choose to not disclose all information about them online. However, an adversary is perfectly able to use public information shared by their contacts or embedded in multimedia content to discover hidden information on their profiles. It is therefore crucial to provide solutions for enriching the user profile.

So far, many approaches examine the potential of identifying a person using the Web. But they suppose that the person has a priori related information on the Internet. Inputs, queries and results are text-based. Unlike textual attributes, multimedia content cannot be approached without special processing to reduce uncertain decisions that overcome when similarity operators come to play. Photos shared within online social networks are useful for obtaining better background knowledge needed for preserving individuals' privacy. Moreover, we believe that investigating events and discovering link types between events will serve as a base for future applications that require semantically enriched profiles. In this study, we address these challenges and present our approach for detecting and linking events using photos shared within online social networks.

Chapter 3: Privacy-Preserving Framework for Multimedia

Sharing

Abstract

Multimedia documents' publication in general and photo sharing in particular have become part of the routine activity of many individuals and companies. Such data sharing puts at risk the privacy of individuals, whose identities need to be kept secret, when adversaries get the ability to associate the multimedia document's content to possible trail of information left behind by the individual. In this chapter, we propose de-linkability, a privacy-preserving constraint to bound the amount of information shared that can be used to re-identify individuals. We provide a sanitizing MD * -algorithm to enforce de-linkability along with a utility function to evaluate the utility of multimedia documents that is preserved after the sanitizing process. A set of experiments are elaborated to demonstrate the efficiency of our technique.

I. Introduction

The publication of personal information on social media and blogging sites has resulted in the users' exposure to privacy breaches. In essence, preserving anonymity is becoming increasingly challenging with the tools and information available on the Internet today. It goes without saying that every anonymous multimedia document published can be put at risk and linked back to the individual without appropriate anonymization techniques.

As mentioned in the previous chapter, most of the works done in the literature to preserve anonymity focus on structured relational data [START_REF] Samarati | Protecting Respondents' Identities in Microdata Release[END_REF]126] while the only few techniques [START_REF] Geng | Privacy Measures for Free Text Documents: Bridging the Gap between Theory and Practice, in Trust, Privacy and Security in Digital Business[END_REF][START_REF] Ma | Preserving privacy on the searchable internet[END_REF] proposed to handle identity anonymization in multimedia documents assume textual data with no reference whatsoever to multimedia objects such as images and videos.

In this chapter, we provide a generic framework and efficient algorithm for dealing with any kind of multimedia documents. We propose de-linkability, a novel technique for preserving individual privacy before sharing multimedia documents. De-linkability ensures that individuals' identifiable information composed of both textual and multimedia content cannot be used to infer his/her identity. An evaluation based on a real image dataset demonstrates that our privacy-preserving constraint is able to bound the amount of information that can be used to re-identify the individual.

The main contributions of this chapter can be summarized as follows:

 We formally define the identity anonymization problem in multimedia documents composed of textual and multimedia content.

 We quantify the re-identification threat which is highly dependent on how much information can be acquired from i) adversaries' background knowledge and ii) external sources containing relevant information related to the anonymized individual.

 We present our sanitizing MD * -algorithm that allows to sanitize multimedia documents' content and preserve at the same time their utility in order to achieve the de-linkability.

 We provide a utility measure to determine to which extent a multimedia document remains consistent after the sanitizing process.

The remainder of this chapter is organized as follows. In Section 2, we present the adversary model adopted in our study. Our data model definitions and operators are presented in Section 3. In Section 4, we give a formal definition of the re-identification problem. Section 5 is dedicated to present the de-linkability privacy constraint and to show how it is possible to preserve individual anonymity using a multimedia document sanitizing algorithm (the MD *algorithm) and a utility measure. In Section 6, we evaluate our sanitizing algorithm to finally conclude and discuss our future research directions in Section 7.

II. Adversary Model

In our adversary model, we assume that the adversary, that we call cyberstalker, knows that a given individual, that we call cyberstalkee 77 , is hiding his/her identity (e.g., François Fillon in the Motivation Scenario in Chapter 1). We also assume that the cyberstalker has access to public information enabling him/her to link some personally identifying information, in a shared multimedia document, to the cyberstalkee. More subtle, we assume that the cyberstalker has no prior knowledge of specific values for the stalked individuals. For example, the cyberstalker described in our motivating example does not know a-priori that "Château de Beaucé" is the residence of the cyberstalkee. 77 Both terms cyberstalkee and individual will be used interchangeably in the remainder of this chapter.

III. Data Model

We start by defining the data model and the basic notations (Table 6) used in the remainder of this chapter.  𝐴 𝑚 ⊆ A is a subset of attributes of A whose values are used to identify a multimedia object mo.

 V is a set of values describing the multimedia object. ∀vi ∈ V for (1 ≤ i ≤ |V|) vi ∈ D(aj)
where aj is an attribute of Am.

 O is the raw data of the multimedia object. O ∈ D(ai) where ai is a multimedia attribute of Am. O(mo) denotes the raw data of multimedia object mo.

 MO is a set of multimedia objects contained in mo. In this work, we only consider image data. Therefore, we will not have MO as in the case of video which is constituted of scenes and frames.

 ς ⊆ 𝐴 𝑚 × X = {(aj, xi)|aj ∈ 𝐴 𝑚 , xi ∈ V or
x is O} is an association function that assigns each attribute aj to its corresponding value which is either a textual vi ∈ V or a multimedia raw data O.

For example, Figure 25 shows multimedia objects mobeauce and momanoir representing two images of "Château de Beaucé" where "keywords" is an attribute of mo, O contains the raw data and MO is the empty set of multimedia objects contained in mo.  MO is a set of multimedia objects where ∀moi ∈ MO for (1

≤ i ≤ |MO|), moi is a multimedia object contained in MD.
An example of a multimedia document could be, but not limited to, personal blogs, set of tweets, newspaper articles, etc. Typically, these documents are composed of words and multimedia objects.

Now that we have defined our multimedia document, we present in the following what we call a multimedia document signature (SMD).

Definition 5  Multimedia Document Signature: Let MD be a multimedia document, a multimedia document signature denoted by SMD is a subset of MD composed of textual and multimedia content. SMD is created using SMD = IA(MD,As) where IA is a function used to retrieve from MD relevant words and multimedia objects related to the subset of attributes As ⊆ A.

We assume that not all attributes found in a multimedia object provide meaningful clues that could lead to re-identify the cyberstalkee. In essence, the idea is to generate signatures that are mainly related to the individuals. This could be done by determining the most significant and relevant attributes retrieved from the document's content and/or using some of the attributes from the individuals' profiles. These attributes help reduce the error rate of individual name disambiguation [127], particularly when the individual's profile is considered as a relevant source of attributes. For instance, it is unlikely for an individual working in a Health Care Department to be related to Computer Science. In other terms, some of the words and multimedia objects should more likely be related to the medical field instead of computing.

The followings are three sample multimedia documents' signatures generated based on the attributes Country, Event, Location and Image. 𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 is the anonymous multimedia document signature of Prime Minister François Fillon.

𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 : ((visiting, "Japan"), (visit, "Meeting"), (annotation,"@beauce"), (home, mobeauce))

Both 𝑆 𝑀𝐷 𝛽1 and 𝑆 𝑀𝐷 𝛽2 are publicly available multimedia documents signatures related to Prime Minister François Fillon.

𝑆 𝑀𝐷 𝛽1 : ((name, "François Fillon"), (country, "France"), (visiting, "Japan"), (visit, Meeting"))

𝑆 𝑀𝐷 𝛽2 : ((name, "Françis Fillon"), (annotation, "Home"), (home, momanoir))

III.2 Data Comparison

We provide, in this section, the appropriate operators to address both multimedia and textual content of multimedia documents.

Definition 6  Estimated Equality: Let W1, W2 be two sets of words over which an association function f can be used. Their estimated equality is computed as follows:

𝑒𝑞𝑢(𝑊 1 , 𝑊 2 ) = ϖ(𝑓(𝑤 𝑤 1 , 𝑤 1 2 ), … , 𝑓(𝑤 𝑚 1 , 𝑤 𝑟 2 )) → [0, 1]
where:

 𝑤 𝑖 1 , 𝑤 𝑗 2 are two words of W1 and W2 respectively, m = |W1| and r = |W2|.

 f is an association function defined as:

𝑓(𝑤 𝑖 1 , 𝑤 𝑗 2 ) = { 1 if 𝑤 𝑖 1 ∈ 𝑊 1 is the same as 𝑤 𝑗 2 ∈ 𝑊 2 0 otherwise
 ϖ is an aggregation function (e.g., max, min, avg, etc.) used to aggregate association functions' scores.

In our example, the estimated equality takes, for instance, the set of words in the Wikipedia article about François Fillon and compares them with the set of words on his Twitter page.

For instance, if we use the substring function SUBSTR 80 between words from the two sets, we see that beauce (from the Wikipedia page 81 ) is the substring value we wish to find from fdebeauce (from the Twitter page).

The estimated equality is used to identify the amount of common textual values found in multimedia documents (or any subset of them). Alternatively, multimedia documents contain complex types such as images and videos which cannot be approached using traditional 80 SUBSTRING (expression, start, length ) = SUBSTR(fdebeauce, 4, 6) = beauce 81 http://en.wikipedia.org/wiki/Francois Fillon equality operators. We define in the following, an estimated similarity operator to process multimedia objects.

Definition 7  Estimated Similarity: Let MO1, MO2 be two sets of multimedia objects over which n similarity functions s1, …, s2 can be used. Their similarity score is computed as follows:

𝑠𝑖𝑚(𝑀𝑂 1 , 𝑀𝑂 2 ) = ϖ(𝑠 1 (𝑚𝑜 1 1 , 𝑚𝑜 1 2 ), … , 𝑠 𝑛 (𝑚𝑜 𝑚 1 , 𝑚𝑜 𝑟 2 )) → [0, 1]
where:

 𝑚𝑜 𝑖 1 ,
𝑓(𝑊 1 , 𝑀𝑂 2 ) = ϖ(𝑓 1 (𝑤 1 1 , 𝑚𝑜 1 2 ), … , 𝑓 𝑛 (𝑤 𝑚 1 , 𝑚𝑜 𝑟 2 )) → [0, 1]
where:

o fk is a unit association function used to determine whether there is an association between a word w1 ∈ W1 with a multimedia object mo1 ∈ MO2.

We note that 𝑓 1 (𝑤 𝑖 1 , 𝑚𝑜 𝑗 2 ) determines the association between 𝑤 𝑖 1 and 𝑚𝑜 𝑗 2 based on the attributes and values of the latter. For example, it may associate a GPS coordinates, specified as one of the multimedia objects' metadata, with a word representing the corresponding location. The function fk returns a score between [0, 1], where 1 represents a perfect association of the attributes and 0 represents the absence of association.

o ϖ is an aggregation function used to aggregate the computed association scores.

 λm ∈ [0, 1] allows to assign priorities to f (W1,MO2) and f (W2,MO1), based on the relevance of multimedia objects in the multimedia documents.

To illustrate the influence of the cross-matching, we take as input the set of words from the Wikipedia page about François Fillon and the set of multimedia objects from his Twitter page.

We define a function 83 that maps words from the first set to geographical coordinates in the second set. By doing so in our example, we found that the profile image published on the Twitter account and "Château de Beaucé" found on the Wikipedia page refer to the same location.

We show in the following how multimedia documents intersection can be determined using selective intersection.  a represents an attribute for which an equality, similarity and/or cross-matching score should be computed. Such attributes, defined in the attribute set, can be used to selectively choose relevant content in multimedia documents. For instance, it is 83 For example, compareLocations((lat1, long1),(address2))= 1 if (ConvertFromLatLong(lat1, long1) = address2); 0 otherwise possible to capture the amount of common information related to the attribute Person.

This refers to computing equality, similarity and cross-matching of words and multimedia objects that are related to this attribute for both multimedia documents'

signatures 𝑆 𝑀𝐷 1 , 𝑆 𝑀𝐷 2 .
 wa is the weight assigned to attribute a where its magnitude depends on the normalizing assumptions. We assume, in this case, that the estimated similarity between multimedia objects mobeauce and momanoir in 𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 and 𝑆 𝑀𝐷 𝛽2 returns a 0.8 score. The cross-matching score between multimedia object mobeauce and France in 𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 and 𝑆 𝑀𝐷 𝛽1 respectively returns a 0.3 score for semantic similarity. This matching returns a 0.5 score between 𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 and 𝑆 𝑀𝐷 𝛽2 based on the matching between "keywords" attribute of the multimedia object (momanoir) and @beauce.

Unlike mutual information metric [128] which is based on joint probability measures, our selective intersection is a non-correlation based metric where the count of each value in the signatures has minimum influence on the overall computation score. Specifically and for privacy reasons, this assumption is useful to determine the "minimum" intersection between multimedia documents where weighted attributes reflect relevant association measure if processed efficiently. We will show in the following definition, the premise of multimedia documents' association.

Definition 10  α-association: Let MD1, MD2 be two distinct multimedia documents. We say that an α-association exists between MD1 and MD2 if their selective intersection 𝑆 𝑒𝑙 𝐼 𝑛𝑡 (𝑆 𝑀𝐷 1 , 𝑆 𝑀𝐷 2 ) is greater than α where:

 𝑆 𝑀𝐷 1 and 𝑆 𝑀𝐷 2 represent corresponding multimedia documents signatures.

 α ∈ [0, 1] is the association threshold.

α-association expresses the presence of a possible association between two multimedia documents represented by their signatures. It measures the strength of an association between two multimedia document signatures based on their common information composed of both textual and multimedia content.

Suppose we set the threshold α = 0.4. Since 𝑆 𝑒𝑙 𝐼 𝑛𝑡 (𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 , 𝑆 𝑀𝐷 𝛽1 ) < 0.4 , 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 when combined with 𝑀𝐷 𝛽 1 does not break the anonymity of François Fillon. However, 𝑆 𝑒𝑙 𝐼 𝑛𝑡 (𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 , 𝑆 𝑀𝐷 𝛽2 ) > 0.4 can put at risk the privacy of François Fillon and hence allows his identification.

III.3 Identity Anonymization Problem

In the presence of adversaries with sophisticated tracking abilities, privacy and ownership preserving of shared data tends to be a complex task. Such adversaries, armed with plausible background knowledge and a wide range of accessible web-based social information, compromise anonymization techniques and put at risk individuals' privacy. Here, we express the identity anonymization problem that could arise when sharing or publishing multimedia documents as the amount of information accessible by the adversary and that can be, at the same time, associated with the owner of the published multimedia documents. It is formally defined as follows:

Definition 11  Identity Anonymization Problem: Let MDu be the multimedia document of an individual u. There are two thresholds α and β. We say that an adversary is able to re-identify

u from MDu if ∃ MDβ,

IV. Privacy Preservation Prior to Publication

Preserving privacy requires that the cyberstalker remains unable to detect the anonymized identity of the cyberstalkee, owner of the multimedia document to be published. As we have stated in the previous section, a re-identification threat occurs mainly due to:

 the link between his/her related multimedia document MDu and a multimedia document MDβ accessible by the cyberstalker, and  the amount of information extracted from MDβ and associated with u.

Controlling the latter can be, on one hand, a burden or eventually unrealizable due to accessibility issues while, on the other hand, breaking the link between multimedia documents is achievable and can be done using de-linkability. MOβ are sanitized in MDu. 84 The importance of retrieved Wβ and MOβ is determined based on the priority thresholds prefixed in the selective intersection function.

de-linkability. Given a cyberstalkee u and a multimedia document

IV.3 Utility Estimation

To ensure safety, there is trade-off to be made at the stake of utility in order to meet strong privacy requirements. While this could be limited in general, it is considered an absolute necessity in order to establish trust between data owners and data providers. This issue has been the essence of several works [138-140] that provide data anonymization. Here, we determine to what extent a multimedia document remains consistent after the sanitizing process. In particular, we provide an estimation of utility based on the relevance of both words and multimedia objects sanitized. where 𝑅 𝑤 (𝑊 * ) and 𝑅 𝑚 (𝑀𝑂 * ) are word and multimedia relevance metrics.

U is used to express the trade-off between privacy and utility. It shows at which point a sanitized multimedia document can be considered useless according to the amount of relevant information it contains.

V. Experiments

In this section, we present a set of experiments to evaluate the efficiency of our approach. We implemented the MD * -algorithm code in Java and conducted experiments using a 3.4 GHz Intel Core i7 with 16 GB RAM.

V.1 Dataset configuration

We used 200 individuals of the dataset published 85 by the authors of [START_REF] Podesta | Big Data: Seizing Opportunities Preserving Values[END_REF]. For each individual, we grouped 100 of his/her tweets to form his/her MDu. These MDu have been filtered to remove identifying names. OpenCalais api 86 is used to extract attributes from multimedia documents MDu and MDβ. We actually used the most relevant attributes extracted based on a predefined threshold that we have set to 0.5 (this threshold can be used to fine-tune the evaluation results and include relevant attributes).

We limited our use of multimedia objects to images. We specifically used the Zemanta api 87 to retrieve and associate images with their related words contained in MDu. As a matter of fact, the images that were mainly retrieved from the web, compensate the lack of metadata that could be used to link words to their corresponding images. That being said, the use of the Zemanta api enriched the content of MDu with multimedia objects that could be used to reidentify individuals. Individual profiles pfu were downloaded using the Twitter api 88 . For our assessment, we only focused on four profile attributes namely name, screen name, location and profile image url.

As per cyberstalkee, we retrieved up to 10 relevant multimedia documents MDβ using the Google api 89 applying to the individual name combined to relevant content from his/her related MDu. This way, we can assert that the retrieved multimedia documents MDβ are related to the cyberstalkee at hand at least through their names.

To compare images, we used the phash function 90 and assigned a manual weight of 0.5 to the estimated similarity for the selective intersection SelInt.

V.2 Evaluation Results

We elaborated a set of measurements to evaluate the efficiency of the MD * -algorithm. These measurements can be summarized as follows:

 Evaluating the identity anonymization problem represented by the percentage of individuals re-identified;

 Determining the uncertainty raised after sanitizing multimedia documents;

 Evaluating the utility of multimedia documents after the sanitizing process;

 Determining the computational cost of our MD * -algorithm.

V.2.1 Evaluating Privacy

In this first test, we evaluated the identity anonymization problem represented by the We can see that when the association threshold increases, there is a higher chance of linking individuals to the multimedia documents MDβ retrieved from the external source and eventually leading to their re-identification.

V.2.2 Evaluating Uncertainty

We evaluated the MD * -algorithm to determine the increasing uncertainty raised due to the sanitizing process. To do so, we calculated the average entropy Figure 27 shows that the uncertainty caused by the sanitizing process is relatively small. This uncertainty could get even smaller if sanitizing multimedia objects was approached differently using blurring or pixelizing techniques which might preserve the semantic and coherence of images' content. This requires a more extensive study that we would like to address in the future.

V.2.3 Evaluating Utility

To evaluate the utility of multimedia documents that have been subject of a sanitization process, we sanitized a specific percentage (20, 40, 60 and 80%) of words and multimedia objects chosen randomly from the multimedia document signatures of 100 MDu. Here, the salient objects which are represented using our multimedia object representation have been sanitized. The resulting utility computed in terms of words and multimedia relevance metrics of each of the multimedia documents is shown in Figure 28. As one can notice, the trade-off between privacy and utility is explicitly shown in the results

where the increased percentage of anonymized words and multimedia objects decreases the utility of the multimedia documents. Nonetheless, such decrease of utility remains bounded by the number of words and multimedia objects contained in the multimedia documents signatures where only their content is subject to sanitization.

V.2.4 Evaluating Computational Cost

The MD * -algorithm's time complexity is polynomial and of: This can also be seen experimentally in Figure 27. The resulting computational time depends on:

1) the conjunctive set of words and/or multimedia objects in Eu that are used to query the external source,

2) the external source from which multimedia documents (MDβ) are retrieved (e.g., the Web in our case). This is what we call fetching time which in some cases can be unpredictable as noticed between

VI. Conclusion

In this chapter, we proposed de-linkability, a privacy-preserving constraint that ensures the safe publication of multimedia documents. de-linkability addresses the privacy threat in its broader aspect while considering both textual and multimedia content.

We employed a selective intersection function to quantify the re-identification threat which is highly dependent on how much information can be acquired from i) adversaries' background knowledge and ii) external sources containing relevant information related to the anonymized individual.

In order to quantify common information between multimedia documents, we defined three operators:

1) to compare text content between documents such as "@fdbeauce" and Beaucé in the Wikipedia page about François Fillon,

2) to compare multimedia content according to multimedia objects such as the (geographical coordinates) metadata of the profile image of François Fillon and the metadata of the manor picture from Wikipedia page,

3) to match text and multimedia data such as "@fdbeauce" and the (geographical coordinates) metadata of the profile image of François Fillon.

Furthermore, we provided a sanitizing algorithm (MD*-algorithm) to protect against violating content and preserve at the same time a minimum quality through an adapted sanitization process that takes into consideration the complex nature of multimedia objects.

The evaluation of the efficiency of this algorithm was performed in terms of the uncertainty raised and the utility loss of multimedia documents due to the sanitizing process. With delinkability, we were able to demonstrate that profile information of the Twitter users of the dataset tested cannot be used to infer the users' identities. Additional information about an OSN user can be obtained by exploring events published and shared on his network.

Therefore, the user profile can be updated or completed with information from events experienced by the user. In the next chapter, we will provide a methodology for detecting personal events in photos shared within online social networks, in order to enrich the profile information of the individuals to protect.

Chapter 4: Personal Event Detection from OSN Photo's

Metadata

Abstract

Online social networking has become the predominant activity in the digital world thanks to multimedia data (mainly photos) sharing. Discovering events where users are involved using their own posts and those shared by their friends would be of great importance. In this chapter, we address this issue by providing an original approach able to detect user's events using photos shared within his online social networks. Using metadata, our approach provides a multidimensional gathering of similar photos using their temporal, geographical, and social facets. To validate our approach, we implemented a prototype called Foto2Event and conducted a set of experiments on the MediaEval dataset. Results show that our approach works well for various metadata distributions.

I. Introduction

Since photos are the most popular type of content shared on online social networks today, we propose in our study to exploit photos. Moreover, because photos uploaded online usually represent real stories from users' life, we exploit in this chapter personal events in the individuals' own posts as well as those shared by their friends/contacts. Specifically, we propose a clustering algorithm based on the photo collection's metadata. Our approach is able to detect and link users' elementary events using photos (and related metadata) shared within their online social networks. We particularly consider OSN photo's metadata that can be used to characterize who participated in the event, where the event happened, and when it happened.

There are several applications where events detected from photos may be useful, such as:

 Enriching User profile: User profile can be updated with information from events experienced by the user. For instance, it is common that many social network users do not complete their profiles with all personal information such as marital status and home location. However, information available about events (e.g., event location) can be used to disclose sensitive information for the user, or personal data he did not intend to disclose online (e.g., home location). This information, if found, would result in a richer user profile and thus prevent further the identity leakage in online social networks [142].

 Detecting missing data: Events may help in estimating missing (meta)data when photos do not have social tags or metadata [START_REF] Sarma | Dynamic relationship and event discovery[END_REF]143]. In essence, photos belonging to a particular event should have some identical information such as the location and date

[144]. For instance, when a user is identified in a photo containing all metadata information, a propagation could be performed to estimate untagged faces and infer the date and the location of other photos detected in that same event.

 Relationship discovery: Some researches focused on identifying relationship types between users within the same social network or across different social network sites [145]. Yet, it is still a great challenge to discover "hidden" relationships between users in a social network. For example, the Facebook algorithm presented in [146] uses the individual's network neighborhood to identify people who are dating. By finding people who usually attend the same events, uncovered relationships, as well as new relationships could be discovered.

In this study, we detect personal events in photos shared online to enrich the profile information of users. In essence, metadata of photos attached to events may include sensitive, private information about the user who needs to preserve his/her privacy. Therefore, exploiting users' events would provide better privacy protection to the anonymous users on social networks.

The rest of this chapter is organized as follows. In Section 2, we describe our data model used in our approach. A motivating scenario and key challenges are presented in Section 3. Section 4 gives a brief overview of our methodology. We present the pre-processing method in Section 5. Section 6 presents our event detection approach. Section 7 presents our prototype and experimental tests. Section 8 concludes this chapter.

II. Data Model

II.1 User (u)

u is a social network user who has an account on a social network site (e.g., Twitter, Facebook, etc.). Each user has an OSN user profile including personal information (e.g., name, birth date, gender, etc.), contact information (e.g., email, mobile phone, address, etc.), personal interests, work experiences, etc. (S)he can post content on the social networks in the form of photos, videos and text. Since we are interested here only in photos shared by users, we describe each user as follows:

u: (uId, pf, I)

where:

 uId: is the identifier of a given user. u can be identified on the social network site by his/her name, email address, or any other identifying information;

 pf: is the set of attributes used to describe the profile of a user and their corresponding values. This is defined in the Individual Profile definition (pfu) in Chapter 3;

 I: is the collection of photos published or shared within his/her social network.

We note that this definition can be extended to include any type of multimedia data (e.g., video, audio, etc.).

II.2 Star Social Network (SSN)

We represent the social network of a given user u0 as a star graph composed of the set of users directly connected to u0. In online social networks, people are connected to each other through links that can denote one or multiple relationships. These links can be of various types as in real-life (e.g., colleagues, relatives, friends, etc.). In this work, we assume that two users are linked by only one single type of relationship. Formally, we define the star social network of u0 as follows:

SSNu0: (U, L, fl)
where:

 U: is a set of users in the social network of u0;

 L: is the set of labels describing the relationships between users;

 fl : U  L is an association function mapping a link label to each person linked to u0.

II.3 Social Space ()

The social space () is a coordinate system, representing the environment of a social network user u0, over three dimensions: locations, times, and person tags (Figure 30). It can be formally represented as follows:

: (L, T, S)

where:  L: is the set of locations identified in the social network of u0, ordered according to a given distance dist∆L from u0's hometown. The units of L can be one of the following values: GPS-coordinate, street, city, region, or country. We assume here that dist∆L values between two locations having the same distance from u0's hometown but  satisfies the properties of a metric space (symmetry, non-negativity, and triangle inequality)

since each dimension is associated to a distance (proof is omitted here since it is obvious).

II.4 Metadata (meta)

The metadata is a set of attributes describing a photo. Metadata contains context information about the creation of the photo (e.g., user identifier of the creator user, time and date of creation, place of creation, purpose of creation, description, technique, etc.). In this study, we consider only three of the 5W1H attributes: who, where and when, according to the social space's features. Thus, we represent the metadata information of a photo as a point in the social space : meta: (l, t, s)

where:

 l represents the geo-location of a photo when captured (e.g., city, region, country, etc.);

 t represents the capture date/time of the photo. If the capture date/time is missing, we use the uploading date/time instead;

 s is the name of the creator or the publisher of the photo. In essence, the creator's name is used when (s)he belongs to the SSNu0, otherwise the publisher name is used.

II.5 Image (img)

An image represents a photo posted on the social network of u0. It contains embedded metadata attributes that give information about the captured scene and people identified in it.

In this study, we only focus on photos that indicate a place (i.e., l is not empty) or depict at least one person in SSNu0 within the captured scene. Formally, we represent an image as follows:

img: (imgId, meta, F)

where:

 imgId: is a unique identifier of the photo (e.g., its URI);

 meta: is the metadata describing the photo;

 F: is a set of people, belonging to SSNu0, who are identified and tagged in the photo.

Figure 31 shows an example of a photo with its metadata.

II.6 Elementary Event (e)

We define the elementary event that will be used in this study as follows:

e: (evid, SB, metae)

where:

 evid: is the elementary event identifier;  T'  T: denotes the datetime interval in which the event took place.

This interval is made from dates of the first and the last evidence (i.e., photo) that an event has taken place. Allen and Ferguson [153] presented a definition of temporal intervals, and defined events using interval temporal logic. In Allen's interval temporal logic, a time interval is defined in the linear time line, with a fixed starting point and ending point.

 S' S: denotes the set of people participating in the event. We create bag-of-users for each event, by counting once each person tagged in the photos taken at the event.

' can be visualized in a 3D graph (see Figure 32). The point set represents the photos' position corresponding to the same event.

 metae is the set of representative moments, locations, and people/groups involved in the event. metae is used to make inference information from the event social space.

The inference information can be used in order to further identify the context of the event and thereby perform reasoning tasks which would aid the event semantization.

By mining significant-frequent and significant-rare dates, locations and tags, it would be possible to identify interesting aspects and patterns which will provide a semantic description of events. Frequent and rare event aspects will be addressed in a dedicated study.

III. Trip Scenario

Being an active user on Facebook, Lisa (u0) frequently posts photos taken at different events.

Her current residence is in the city of Biarritz in southern France. She has four hundreds of connections (e.g., family members, friends, colleagues, etc.). She posted photos taken with her colleagues and friends taken during a trip to Sweden in May 2014. Her colleagues are from Biarritz and her friends live in Sweden. To show the multi-* property of events, we use the Facebook photos of that trip. We illustrate here four major properties: i) multi-source, ii) multi-participant, iii) multi-day, and iv) multi-site.

III.1 The multi-* property of events

III.1.1 Multi-source Trip

Ghada, Kouki, Regi and Solomon are four colleagues who traveled with Lisa to Stockholm. They took photos and shared them through Facebook, but each in his/her own account. As a result, multiple albums of the same trip were created by different persons on SSNu0. We show some photos that illustrate this case in Figure 33.

III.1.2 Multi-participant Trip

An event may include several participants; some of them post several photos about it while others do not. For instance, only 4 of the 12 colleagues who traveled with Lisa posted photos of the trip (as shown in Figure 33). The other colleagues did not post anything about it online. As we can see in Figure 34: Saddem is identified in the photos of Lisa, Joseba and Irvin are identified in the photos of Regi, etc.

III.1.3 Multi-day Trip

The trip to Sweden started on the 8 th of May and ended on the 11 th of May 2014. Lisa and her colleagues took photos on all days of the trip. Figure 35 shows photos of Lisa from the trip but with different creation dates.

III.2 Challenges

This scenario shows that photos can include information about events that may be not available using only low-level features (e.g., colors, etc.) or the available description provided by the users. The main challenges that arise from this scenario are:

 Detecting photos depicting the same social events, given the metadata attributes describing photos and the person-tags. Given the large collection of photos shared in the social network of Lisa, the main challenge here is to identify all photos related to this trip, even though they concern many participants, or are published by several sources, and/or occur over many days and in several places. Considering this multi-* property ensures getting people, places and moments of the event that were not captured by the main user for many reasons (e.g., scenes subjectively not interesting for her, camera or cell phone battery running out of charge, weak social ties with some participants, etc.).

 Identifying all participants in the event:

o whether they are identified in the same photos with u0 or not:

This can be due to the type and the strength of their relationships with u0. In essence, the relationship strengths between users in online social networks -as in real world-may vary significantly. This is the case of Joseba as described above.

o whether they posted about the event or not:

Events with many participants may include persons who are not active on social media or who do not share photos of their events online. This is the case of Irvin and Joseba as described above.

 Identifying the wedding event as a sub-event of the trip event to Sweden. This can be deduced from two facts:

o The wedding took place in Örebro contained in Sweden during the same period of time as the trip event.

o Friends of Lisa who are residing in Sweden posted photos of the wedding and tagged Lisa.

Sub-events and other related events will be presented in the next chapter.

IV. Methodology Overview

To overcome these challenges, we propose a clustering method to cluster photos based on their metadata and person-tags associated with them. Since metadata are most of the time automatically available in photos, we chose to rely on metadata to detect events from photos.

Furthermore, metadata provide exact and objective information whereas annotations provided by users might be subjective (based on the user's feeling, interest, etc.). We also note that the processing cost (time and relevance) is low using metadata features.

An overview of our event detection algorithm is shown in Figure 37. The suggested algorithm is composed of two main steps: first, to define metadata and person-tags granularities; and second, to detect clusters of photos. The input is SSNu0, the social network of u0, for which we are interested in detecting events. The output contains linked elementary events of u0. In the following, we present our approach in more details.

Figure 37: Description of our approach to detect elementary events.

V. Metadata Pre-processing

Identifying person-tag and metadata occurrences in the collection of photos would be useful to determine the appropriate parameters for the clustering. This pre-processing step takes into consideration the data distribution of photos we have at hand. Three factors can be analyzed:

(i) the photo capture date/time, (ii) the photo locations and (iii) the people-tags depicted in photos.

V.1 Time Granularity

Firstly, we examine the dispersion of photos over time. The time granularity can be one of the unit values of the temporal axis T defined in the Social Space , (such as a year, a month, a day, or an hour). If the time difference between photos corresponds to one day or more, we use day as the time granularity. Otherwise, we use finer time granularities such as hour.

V.2 Space Granularity

Secondly, we examine the geographic dispersion of the photos' locations. Locations can vary at different levels of granularity as the units used in the geolocation axis (L): country, region, city or street. Therefore, we also choose the space granularity based on the heterogeneity of photos' locations. If photos are taken in different streets, we use street as a granularity. Otherwise, we user finer space granularities such as buildings.

V.3 People-tags Granularity

Some people such as family or best friends are more likely to appear in the photos with u0 than others. For this reason, the third purpose of the pre-processing algorithm is to identify frequent faces observed in the photos of u0. Therefore, we set a minimum value threshold minF. For each user ui in the social network of u0, we measure the portion of photos where ui appears in the collection. Frequent faces correspond to users who appear in the photos with u0 in a portion pi higher than minF. Other faces are considered rare faces. This measure will be more elaborated in a future work. In present work, only the time-space granularities are inputs to the clustering algorithm.

VI. Elementary Event Detection

After identifying the appropriate granularities based on the collection of photos at hand, this step aims at identifying clusters of photos belonging to the same event. In this section, we describe how to group similar photos together to form clusters corresponding to elementary events. In this study, we propose an agglomerative, nonhierarchical clustering algorithm that uses three features based on our Social Space : time feature, location feature, and creator/publisher feature. We define the time-space granularities as obtained from the preprocessing step. We use the Social Space corresponding distances between any two photos, photoi and photoj: i) dist∆T between date/ time values, ii) dist∆L between locations, and iii) dist∆S between names. Our algorithm is based on the following assumption:

Different clustering methods

Assumption 1: Since a user can be only at one place at a time, photos captured by him/her at a specific time within a geographical boundary are considered as belonging to the same event.

Thus, a cluster k contains all photos created/published by a user on a common period and location. Formally: where:

k = {photo}/  i,
 γ is a predetermined threshold related to the distance between the capture date/time values of two photos,  δ is a predetermined threshold related to the distance between the geo-locations of two photos,  ε is a predetermined threshold related to the distance between the creators' names of two photos. In this work, we set ε to 0.

VII. Prototype and Experimentations

In this section, we present the set of experiments we have performed to test the efficiency of our approach. First, we describe the prototype implemented to validate the clustering algorithm. Then, we describe the dataset we used and introduce the evaluation strategy we followed. Finally, we present the experiments results.

VII.1 Prototype System

We developed a prototype system using Java, called Foto2Event, to test, evaluate and validate our event detection framework. In this section, we present the general architecture of our prototype (Figure 38), and we detail the different modules of the system: the profile manager, the metadata manager, the photo parser, the preference manager, and the event builder.

Figure 38: Architecture of our framework to detect events.

The Profile Manager component

The profile manager component manages profile information from one or several social networking websites. The role of this component is to: i) extract and store user information from the social network profile, and ii) extract the contacts of the user in this network. This component allows also to generate from scratch or to complete a user profile within a star social network. As mentioned earlier, the user profile includes information such as name, birthday, home address, interests and contact information. The star social network includes a set of users (i.e. users' names) and their relations (e.g., friends, colleagues, etc.) with the main user.

The Metadata Manager component

This component allows extracting the photo metadata and stores them in XML files to provide input to the clustering algorithm. We chose XML because of its flexibility compared to relational databases. In addition, it is easy to parse and to validate (with XML Schema). This component can automatically generate custom metadata associated with photos such as creator name, location, and creation date, in various formats (e.g., geographic coordinates, cities, countries, etc.). Additional criteria may be applied to the generation of the photo metadata in order to highlight the behavior of our approach on different datasets (e.g., interval

of n days for dates, small or large radius that delimitates the geographical neighborhood area for location, etc.).

The Preferences Manager component

Our system also has a Preference Manager that accepts some parameters of the algorithm such as the thresholds related to the distance between the capture dates or the geo-locations of photos, etc.

The Event Builder component

The event builder component accepts as input a photo metadata set. It first analyzes the distribution of the metadata to determine the corresponding granularities for each dataset.

Then, it clusters photos' metadata based on their time stamps, geo-tags and creators to detect events. Each event is associated with a set of at least one photo.

VII.2 The MediaEval Dataset

The dataset we used is the Social Event dEtection Dataset (ReSEED) which was provided at MediaEval 91 2014. This dataset consists of a collection of 437,370 photos contributed by 4,926 unique Flickr users and assigned to 21,169 events in total.

We chose this dataset since it is a large real-world dataset of images, available online with their metadata for the purpose of detecting social events in large collections of multimedia items, and including labels for ground truth. The metadata includes the following: username of the uploader, date, title, description, tags and geo-location.

Before we get into the experimental evaluation, it is worthwhile to present some interesting and challenging aspects of the MediaEval dataset, as discussed in the ground truth statistics in

[158]:

 Photos represent different types of events;

 The distribution of the events' duration is not uniform: Some technical events such as demonstration and protest events tend to last multiple days, while others like soccer events, last a few hours;

 The size of events varies significantly as well: While 3,598 events include only one single photo, and 1,799 events include 2 photos, there is a small number of events which include over 1,000 photos.

However, there is a significant difference between our case and the application of this dataset.

The dataset contains pictures from the Flickr photo community site that is not a star network.

In order to adapt the dataset to our case, we first chose one main user (u0) from the 4,926

Flickr users. We chose the user who has uploaded more photos (1,497 photos). We parsed his photos and assumed that other users are his contacts in his Star Social Network. Relationships (or link labels) are assigned randomly. In future work, we plan to apply rules to derive relationship type between users. 91 http://www.multimediaeval.org/ One other missing information to complete in the MediaEval dataset is the tags of people in the photos. It is difficult to ask users to provide more information to obtain person tags identified in their photos. Therefore, it might be beneficial to search for this information in the description field provided for each photo. As it is a real-world dataset, the description is available for only a subset of the images (37.8%). For these reasons, we chose to leave for future work the interesting question of person-tags.

VII.3 Ground Truth Creation

Photos have metadata (creator name, latitude, longitude, date and time). Before conducting the experiments, we implemented a pre-processing algorithm in order to get a set of photos clustered into elementary events.

The input to this algorithm is a set of photos to be clustered into elementary events.

The first step is the creator segmentation. It aims at getting distinct creators from the image base. Therefore, we start by separating photos in segments based on their creator.

The second step is the spatial clustering. The goal of this step is to form clusters of photos which are near geographically. To do so, we use a non-supervised density-based spatial clustering using the DBSCAN algorithm. The parameters of the algorithm are: i) epsilon: this parameter specifies how close points should be to each other to be considered a part of a cluster. We fixed epsilon to 1 km. ii) minPts: this parameter specifies how many neighbors a point should have to be included into a cluster. In our case, we fixed minPts to 1. iii) distance:

this parameter specifies the distance metric used. In our case, the distance used is the Euclidean distance.

The third step is the temporal clustering. This step corresponds to dividing clusters into subgroups which are close in time. Therefore, we calculate the time interval between all consecutive photos, the mean M and the standard deviation St of the time intervals. Then, we calculate the threshold t by summing the mean and the standard deviation of the time intervals between consecutive photos. Consecutive photos are separated into different clusters if the time interval is greater than a given threshold t.

This is described in the following pseudo-code fragment.

Input: Photo[],

// a set of photos to be clustered into events DBSCAN, Euclidean_distance, epsilon, minPts // clustering parameters

Output: Event[] // a set of photos clustered into events epsilon ← 1 // 1 Kilometer minPts ← 1 clustererInst ← CreateClustererInstance(DBSCAN, epsilon, minPts, Euclidean_distance) C[] ← GetDistinctCreators(Photo[]) Sort(C[]) for each p ∈ C[] do I[] ← timestamp(pi+1) -timestamp(pi) M ← Mean(I[]) St ← StandardDeviation(I[]) t ← M + St for each c ∈ C[] do c.Segment[] ← getPhotos(c) c.Cluster[] ← CreateSpatialClusters(c.Segment[], clustererInst) for each cl ∈ c.Cluster[] do c.Event[] ← CreateTemporalClusters(cl, t) return Event[]

VII.4 Evaluation Measurements

The main criteria used to evaluate the performance of our event detection algorithm are:

 the number of correctly detected events, and  the number of correct/incorrect photos detected for these events.

Since the MediaEval dataset is accompanied by ground truth data, we compare the obtained sets of events to the ground truth sets of events. We use the two evaluation measures suggested in the MediaEval benchmark of 2014: F-score and Normalized Mutual Information (NMI). In the following, we provide the formula and discuss the role of each measure in details.

F-score is the harmonic mean of Precision and Recall for the retrieved photos.

Precision (PR) identifies the number of correctly retrieved photos (but not the number of retrieved events), w.r.t. the total number of photos (correct and false) retrieved by the system. Recall (R) underlines the number of correctly retrieved photos, w.r.t. the total number of correct photos, including those not retrieved by the system. Having:  A the number of correctly retrieved photos (true positives),  B the number of wrongly retrieved photos (false positives),  C the number of correct photos not retrieved by the system (false negatives), precision and recall are computed as follows:

𝑃𝑅 = 𝐴 𝐴+𝐵 ∈ [0,1] 𝑅 = 𝐴 𝐴+𝐶 ∈ [0,1] 𝐹 -𝑣𝑎𝑙𝑢𝑒 = 2×𝑃𝑅×𝑅 𝑃𝑅+𝑅 ∈ [0,1]
High precision denotes that the clustering algorithm achieved high accuracy in retrieving correct photos, whereas high recall means that very few correct photos were missed by the system. In addition to evaluating precision and recall separately, it is a common practice to consider F-score as a combined measure, representing the harmonic mean of precision and recall. High precision and recall, and thus high F-value indicates in our case high clustering quality.

Another performance measure used to evaluate the clustering accuracy is the

Normalized Mutual Information (NMI).

In information theory, mutual information is a measure of the information overlap between two random variables [159]. The MI between random variables X and Y, whose values have marginal probabilities p(x) and p(y), and joint probabilities p(x, y), is defined as:

𝑀𝐼(𝑋; 𝑌) = ∑ 𝑃(𝑥, 𝑦) ln 𝑝(𝑥, 𝑦) 𝑝(𝑥) 𝑝(𝑦) ∈ [0,1] 𝑥,𝑦
This measure score is also in the range [0, 1]. Higher values indicate a better agreement with the ground truth results. This is because the mutual information between two sets of clusters becomes larger as one set of clusters is more consistent with the other set. A good clustering should have high recall and precision, and also high similarities with the ground truth. F-score measures only the goodness of the retrieved photos but not the number of retrieved events, nor how accurate the correspondence between retrieved photos and events is. Whereas, NMI considers the goodness of the retrieved photos and their assignment to different events [165]. Hence, we use both measures to measure the performance of our algorithm and to compare it with existing algorithms.

VII.5 Experimental Evaluation

In this section, we present the set of experiments conducted to evaluate the efficiency of our approach. The goals of these experiments were:

1) To measure the clustering quality for different cases considering i) the metadata distribution, and ii) the metadata availability, 2) To determine the time performance of our event detection algorithm, and

3) To compare our approach with other event detection approaches.

VII.5.1 Event Clusters' Quality

In order to measure our clustering quality, we run first a series of experiments to test our algorithm on different data distributions. Then, we run another series of tests to evaluate the quality of the clustering considering the availability of metadata items used in our approach.

VII.5.1.1 Experiment 1: Testing on different data distributions

The aim of this test was to study the effect of metadata distribution on detecting events using our approach. To test this, we generated three groups of data from the MediaEval dataset.

Each group satisfies specific criteria based on the distribution of the time, location or uploader information. The characteristics of these datasets are summarized in Table 7.

Table 7: Distribution of metadata in each dataset group. Therefore, we created six groups of datasets, where each dataset consists of a set of images uploaded in one or several years. We tuned the number of distinct years in order to collect photos taken during the same year(s). We fixed the space granularity to city. Then we ran the algorithm using year as the time granularity, followed by month, then day. The results are shown in the graphs in Figure 39 and Figure 40. Results in Figure 39 show that f-measure ranges between 0.75 and 0.77 with the year granularity. It increases when using the month granularity and ranges between 0.78 and 0.79.

Group

However, our method yields the highest value (0.83) with the day granularity. Similarly, results in Figure 40 show that NMI ranges between 0.92 and 0.94 with the year granularity. It increases when using the month granularity and ranges between 0.93 and 0.95. Our method also yields the highest NMI value (0.95) with the day granularity. Hence, the best performing result with different time distributions is obtained when using the day granularity.

Group 2: Photos in the dataset are heterogeneous and their location information may be significantly different. For instance, the location data may be over-close (e.g., when almost photos are taken in one city) or too scattered (e.g., when photos are taken in various countries around the world). Therefore, we created five sub-datasets, where each sub-dataset consists of a set of images captured in one or several countries. We tuned the number of distinct countries in order to collect photos captured in the same countries. We fixed the time granularity to day based on the previous result (Experiment 1). Then we ran the algorithm using country as the space granularity, followed by city, then street. The results are shown in the graphs in Figure 41 and Figure 42. Results in Figure 41Figure 41 show that f-measure values range between 0.76 and 0.81 when using the country granularity. We observe that our method yields better f-measure values with the city granularity and ranges between 0.79 and 0.81. However, when using the street granularity, f-measure is 0.82 for almost all tests. Similarly, our method yields the lowest NMI (0.93) when using the country granularity and almost the same values (0.94 and 0.95) with the city and the street granularities as shown in Figure 42. Hence, we observe the best f-measure and NMI scores achieved for almost all tests when using the street granularity. The graph in Figure 43 shows that NMI and f-measure have nearly the same value for all tests (0.8 and 0.95 respectively). This is because results depends on the values of the "target" variables (time, location, and creator) rather than the number of distinct uploaders.

VII.5.1.2 Experiment 2: Considering metadata availability

The aim of this test was to evaluate the benefit of having the metadata available in photos (creator, time and geographic information). We used the MediaEval dataset described previously. The collection of images was subject to the clustering algorithm based on their metadata. As it is a real-world dataset, there are some metadata that are not present in all images. For instance, the capture time is present for 98.3% of the photos, while the uploader information (i.e., username) is available for every photo. However, the location information is present in only 45.9% of the photos in the collection. We generated 5 groups of datasets based on the availability of metadata. The characteristics of these datasets are summarized in Table 8. SetFull T, G, S=25% 25 [START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF] 25

Setfull dataset: The first group consists of the set of photos that are geotagged with location information (consisting of a pair of latitude-longitude coordinates), uploaded by a specific Flickr user, and provided with a (capture) time information. The objective of this experiment is to show the best case behavior when all photos' metadata are available for the event detection algorithm. Once again we tested our algorithm on all the possible granularities of time and space to confirm our choice of granularities (day and street). A qualitative comparison is shown in Table 9. Because in our experiments, we consider three time granularities (i.e., day, month, year) and three space granularities (i.e., street, city, country), we performed 9 tests. For each test, we used different combination of the granularities (the 2 nd and 3 rd columns), and measure the f-measure and NMI values. The experimental results showed reliable results for different granularities when there is no missing metadata. Specifically, we obtained the best results of f-measure (0.8129) and NMI (0.9519) using the day and street granularities. This experiment confirms the results of the previous experiments by again demonstrating that our method with the day and street granularity produces the best quality of clustering. For this reason, in what follows we test the clustering algorithm using only the day and street granularities.

SetFull (T) dataset:

The second dataset group consists of a set of photos which are geotagged with location information and uploaded by a specific Flickr user. However, the capture time information is present in 75%, 50% and 25% of the photos (SetFull T=75%, SetFull T=50%, and SetFull T=25% respectively). We modified the Setfull dataset to meet these percentages. This group was created to test the usefulness of the capture time information for the event detection. The results are given in Figure 44. The f-measure and NMI values appear to be sensitive to the availability of the time information: f-measure decreases from 0.81 (100%) to 0.74 (75%), then from 0.61 (50%) to 0.44 (25%). Similarly, NMI value decreases from 0. 95 (100%) to 0. 91 (75%), then from 0.88 (50%) to 0.85 (25%).

SetFull (G) dataset:

The third dataset group consists of a set of photos which are uploaded by a specific Flickr user, and provided with capture time information. However, the geographic information is present in 75%, 50% and 25% of the photos (SetFull G=75%, SetFull G=50%, and SetFull G=25% respectively). Similarly, we modified the Setfull dataset to meet these percentages. This group was created to test the usefulness of the geographic information for the event detection. The results are given in Figure 45. Similarly to the time availability, the f-value and NMI levels decrease with the availability of geographic information: f-value decreases from 0.81 (100%) to 0.74 (75%), then from 0.61 (50%) to 0.44 (25%). Similarly, NMI value decreases from 0.95 (100%) to 0.91 (75%), then from 0.88 (50%) to 0.85 (25%).

SetFull (S) dataset:

The fourth dataset group generated consists of a set of photos which are geotagged with location information and provided with capture time information.

However, the uploader information is present in 75%, 50% and 25% of the photos in the datasets (SetFull S=75%, SetFull S=50%, and SetFull S=25% respectively). Similarly, we modified the Setfull dataset to meet these percentages. This group was created to test the usefulness of the uploader information for the event detection. The results are given in Figure 46Figure 46. The same behavior can be observed in the case of the uploader information: fmeasure decreases from 0.81 (100%) to 0.74 (75%), then from 0.61 (50%) to 0.44 (25%).

Similarly, NMI value decreases from 0.95 (100%) to 0.91 (75%), then from 0.88 (50%) to 0.85 (25%).

SetFull (T, G, S) dataset:

The fifth dataset group consists of the three worst cases where the time information, the location information and the uploader information are provided in 75% of the photos, in 50% of the photos, and then in 25% of the photos only (SetFull T, G, S=75%, SetFull T, G, S=50%, and SetFull T, G, S=25% respectively). The objective of this experiment is to show the worst case behavior when some photos' metadata are missing for the event detection algorithm. The results are given in Figure 47.

F-measure levels decrease from 0.81 to 0.74 to 0.61 then to 0.44 where the location, time and uploader information is available in 100%, 75%, 50% then 25% of the photos respectively. Similarly, NMI levels decrease from 0.95 to 0.91 to 0.88 then 0.85 respectively.

The above results emphasize the usefulness of having at least two of the metadata, in order to optimize the clustering process.

VII.5.2 Time Analysis

We implemented the event detection algorithm code in Java and conducted experiments using Intel Core i5 CPU @2.4 GHz with 8 GB RAM.

We experimentally analyzed the execution time complexity of our event detection algorithm. We varied the number of photos in the collection.

In order to evaluate time performance of the system w.r.t. the size of the photo collection, we extracted 10 random samples of the Setfull dataset (photos with metadata). We applied our algorithm 10 times on each sample. Then we measured the average execution time required by our system, including both CPU and I/O time in order to verify our approach's linear time dependency on the size of the photo collection. Figure 48 shows that the time needed to perform clustering of photos in a dataset grows in a linear fashion with the dataset size.

VIII. Conclusion

This chapter provides a clustering algorithm for automatically detecting events using photos shared online on the social network of a specific user.

The Social Space is introduced, with other definitions necessary to present our approach, such as star social network, photo's metadata, etc. Using metadata, our approach provides a multidimensional clustering of similar photos based on their temporal, geographical, and social facets. Firstly, we presented a pre-processing algorithm to determine the appropriate timespace granularities. This allows detecting clusters at a granularity that makes sense to the geographical and temporal distribution of the user's image data. Secondly, we showed how we group photos together based on the three features, related to our Social Space. At this stage, we obtained elementary events. More semantic-meaningful clusters can be obtained by building relations between clusters of events. This is what we will propose in the following chapter.

Chapter 5: Linking Elementary Events

Abstract

Events are increasingly attracting researchers today. Many studies and benchmarks were conducted to detect events from posts people are submitting online at any occasion they have.

However, these approaches do not exploit the links that can exist between events. Links can be related to any aspect of events. Computing links between events on online social networks can help collecting more personal information and enforcing users' privacy. In this chapter, we introduce our methodology for describing links between events. First, we describe our meta-model based on the 4-Intersection Model. Then, we present our methodology to identify basic relations based on three main aspects of events: spatial, temporal and social. We will show how we can combine those basic relations to infer more complex relations such as ontological relations and application-based relations.

I. Introduction

As mentioned in the Event Definitions section in the Related Works chapter, an event is a symbolic abstraction for the semantic segmentation of happenings in a specific spatiotemporal volume of the real world [178]. It includes the presence of entities (i.e. persons who attended or witnessed the same event) throughout the same time and space. Today, people can share photos about their events with many different Web 2.0 tools, such as Facebook,

Instagram, and Flickr. However, there is no tool that allows to link events together based on general characteristics or aspects of events (e.g., events occurring at a particular time), using classical ontological relations (e.g., link events by cause and effect), or via user-defined semantics (e.g., link birthday events). Event detection requires examining events and their inter-relations as well. For this reason, we present in this chapter a meta-model that allows representing, combining and inferring inter-event relations in an expressive and flexible way.

We aim to discover relations that can exist between events detected in photos shared online.

Thanks to our meta-model, we can define three types of event relations: i) basic relations, ii) ontological relations, and iii) application-based relations.

We propose an approach to automatically generate relations that correspond to different aspects of elementary events based on a homogeneous representation. In this work, we focus on the three aspects of elementary events: spatial, temporal and social. In addition, we present a methodology that can combine spatial, temporal and social relations of the event for modeling complex relations (e.g., link sub events that are separated in space but are part of the same event). To reach that goal, we identify a list of ontological relations, such as caused-by, is-a, etc. We model relations with a binary representation, which is supposed to be easy, flexible and extensible. Therefore, our approach can be applied for identifying arbitrary relations that can be defined by the social network user based on her context, his/her applications or his/her preferences as well.

Computing links between events is useful through various Web applications and can be used to support several application needs:

 Collecting personal information from online social networks:

Many studies [179-181] have tried to extract personal information from social media websites to provide better profiling (e.g., tag-based user profile, content-based user profile, etc.). However, the ultimate challenge remains: how to get richer user profiles to estimate missing information? By examining relations between events, we can infer unknown characteristics or traits about a user based on event information in order to construct or complete a user profile. For example, when the user's events always occur in the same neighborhood, we can classify the person as a "stayer". Otherwise, we define the person as a "traveler". Another example is when a user often goes to watch drama movies (as events), we can automatically update his/her preference to this kind of movies.

 Enforcing privacy in online social networks:

Social network users might make careful choices about whom they disclose photos about their events to. These choices might be dependent on the nature and the strength of the relationship they have with their contacts (e.g., friends, close friends, acquaintance, etc.). By computing links between events, social web platforms will be able to offer the user better custom privacy settings that allow each contact to only view events that they experienced with the group they belong to. For example, the user can choose to share photos of events occurring at work only with his/her colleagues.

Therefore, his/her colleagues will not be able to view his/her family-related events by using this kind of privacy setting.

 Understanding user behaviors in online social networks:

Novel methods [182] were used to perform social network analysis (SNA) to understand how users are influenced online. These methods usually provide quantitative measures [183] based on statistical analysis on the available data, and do not focus on the visual representation of data. There are also structural metrics that are applied based on mathematical properties of the social network (e.g., centrality measures, structural groups, etc.). In a recent Facebook experiment in July of 2015, over a million people changed their Facebook profile pictures to a rainbow filter to support the American Supreme Court decision event legalizing gay marriage 92 . By setting up this tool, Facebook was able to track users support for this event and thereby got an unprecedented insight on how to influence their users, as a kind of a psychological testing. This example shows how events detected in photos shared on online social networks can be used to understand their users' behaviors.

 Expressive querying among a large number of photos:

Adding links between events is useful to users who are interested in looking at their photos based on specific criteria such as time, place, or people involved. As the number of photos uploaded per user is becoming increasingly huge, it becomes more difficult to manage and search for particular events. Therefore, establishing links between events would allow the user to answer queries quickly and effectively from large amounts of data. These queries can be, but not limited to the following: a) Which events are co-located in the same place with one specific event?

b) Which events are co-occurring in the same time period as a given event?

c) Which events are experienced by the same users as the ones participating in a given event?

By linking events, we can answer those queries and also solve the multi-* property of events, which is a major problem in the event detection as illustrated in Chapter 4.

In spite of wide range of research on the event detection on online social networks shown in the Related Works chapter, only a limited number of studies were conducted on the event relations. Currently, no one has proposed how event relations can be incorporated into the Social Web.

The rest of this chapter is organized as follows. In Section 2, we present definitions that are used throughout the chapter. In Section 3, we describe how we developed our metamodel. Then, in Section 4, we present basic relations between events and how to identify them. In Section 5, we present a methodology to combine relations according to some constraints that we have formalized. We show how we translate this to a binary representation in Section 6. In Section 7, we suggest a logical expression to represent ontological relations and define some application-based relations that can exist between events. In Section 8, we conclude and summarize the benefits and extensions of this metamodel and approach.

II. Definitions

II.1 Event Context: Who, Where, When

Events in social media can be divided into two types: i) explicitly created events and ii)

implicit events [184]. In the first type, people create event pages (on Facebook, LinkedIn, Google+, etc.) to drive attendance and provide event details (schedules and information), or to promote events before, during and after they take place. Our focus is on events of the second type, where social network users add content (such as photos) during or after attending the event, without explicitly creating the event. In this case, it is important to understand the context in which these events take place. This understanding of the context could serve to identify, among others, the relations between events and to answer many questions like: What other events is this event related to and how? While many definitions of context have been proposed in the literature [185], most include three elements: location, time and people. This corresponds to the previously defined social space  in Chapter 4.

In the following, we introduce our relational social space that we will use in this chapter.

II.2 Social-R Space

Let R denote a set of all possible distinct relations that might exist between two events in a star social network. Each relation r: <name, v∆R> is defined by its name and its value v∆R. v∆R is a three-dimensional value (l, t, s) where l, t and s are values defined in the Social-R space ∆ R defined as follows.

A Social-R Space (∆ R ) is a coordinate system for a three-dimensional space representing the Socio-Spatio-Temporal relations between two events in a social network such as:  : represents the spatial relations that exist between two events;  : represents the temporal relations that exist between two events;  : represents the social relations that exist between two events.

Each point r(l, t, s) in the three-dimensional space represents a particular combination of those three relations. The scales of the three axes are the same as shown in Figure 49. l, t and s are hexadecimal values and l, t, s ≤ F. The choice of this scale will be discussed later.

III.1 Why a Meta-model?

We propose a unique meta-model as a basis for representing relations between events. The meta-model allows to deal with many different aspects of events (temporal, spatial, social, causal, etc.) in similar way. Moreover, it can support textual (e.g., author, description, etc.)

and non-textual features (e.g., creation time, geographical coordinates, etc.). Each feature is associated with its appropriate distance metrics using different distance thresholds, at different levels of granularity. For instance, a pair of features are disjoint if the distance between them is less than a threshold for some distance metric (e.g., the number of minutes elapsed between two date/time values, the Haversine distance between two locations, etc.).

By using a meta-model based approach, our proposal is extensible for more heterogeneous data sets and additional contextual parameters. Thus, it allows us to comprehend events and express links among them in a robust and scalable fashion. Thus, the purpose of a unique meta-model is motivated by two primary needs:

1) Properly represent the heterogeneous aspects of events, and

2)

Represent various relations in a unified and flexible way.

III.2 Topological Models

Several topological models [186,187] exteriors, where the exterior is obtained relative to the embedding space. This model was then introduced to enable the identification of more detailed relations when one or both objects are embedded in higher-dimensional spaces (e.g., a line and a real object).

Relations are described here by the nine set intersections (intersections of the interiors, boundaries, and exteriors), instead of the four set intersections (intersections of the interiors and boundaries only).

In this work, we will not deal with exteriors at all since we are not interested in relations with respect to the embedding space. Furthermore, the 4IM provides a high degree of expressivity in our case, with a small cost of indexation. Subsequently, the 4IM can be a sufficient basis to represent the different relations that might exist between event features. Therefore, we propose to compute the similarity between every pair of events by calculating the four intersection of the interior and the boundary sets of their similar features.

III.3 The 4-Intersection Model

In this section, we show how we adapt the 4-IM to represent spatial, temporal and social relations. We use a unified representation based on the intersection sets (IS) and an intersection matrix (IM).

III.3.1 Intersection Set (IS)

Following the 4IM, we build our meta-model on the notion of a feature F. In this study, F could be a spatial, a temporal, or a social feature. For the purpose of homogeneity, we define the following notions and properties of a feature F as follows:

 The interior I F : is the core of F. It is never empty (I F ≠∅).

 The boundary B F : contains other elements related to F without any intersection with I F (I F ∩B F = ∅). B F may be empty.

 The tolerance ε: defines the distance or condition that separates I F from B F .

The value of ε depends on the (meta)data distribution in the social space of the user at hand. It must take into account the three granularities pre-determined in the pre-processing step of our Event Detection algorithm. Selecting a suitable threshold (for time, location and persons) is important for obtaining accurate (topological) relations. A method of selecting the value of the threshold will be discussed in a future work. In this work, we define it manually.

I F and B F constitute what we call the intersection sets of F. In what follows, we define the intersection matrix of two intersection sets between two events. Then, we describe how to define the interior/border for each feature of events: spatial, temporal and social.

III.3.2 Intersection Matrix (IM)

In order to compute relations or links between events e1 and e2, an intersection matrix IM is formed for each corresponding pair-wise event features F1 and F2. It contains the pair-wise intersection results between all possible combinations of interiors and boundaries. IM can be represented as follows:

𝐼𝑀 (𝐹 1 ,𝐹 2 ) = ⩀ (F1,F2) = ( 𝐼 𝐹1 ∩ 𝐼 𝐹2 𝐼 𝐹1 ∩ 𝐵 𝐹2 𝐵 𝐹1 ∩ 𝐼 𝐹2 𝐵 𝐹1 ∩ 𝐵 𝐹2 )
where:

 I F 1 and I F 2 : represent the interiors of the given feature related to e1 and e2 respectively;

 B F 1 and B F 2 : represent the boundaries of the given feature related to e1 and e2 respectively.

Each matrix cell has a binary value of 0 whenever the intersection between sets is empty, and 1 otherwise. Each relation between two events' features is thus represented by a binary value.

Each computed relation between two features is mutually exclusive: there is one and only one relation between any two features [189]. For example, two events cannot be spatially disjoint and equal simultaneously.

IV. Basic Relations between Events

Following our event social space, two events e1 and e2 are at least related by three basic relations: spatial, temporal and social. We explain each of these relations in this section. First, we define the interior and the boundary characteristics derived from the spatial, temporal and social information of an elementary event. After, we define the corresponding intersection matrix.

In order to illustrate relations described below, let us take the example of Lisa. We select two events shared on her social network.

Trip to Stockholm:

She made to a trip with her colleagues to Sweden in May 2014, as described in the trip scenario in Chapter 4.

Trip to Lebanon:

At the end of May of the same year, Lisa went back to her home country Lebanon to visit her family with her two brothers (Bouli and Elie).

Lisa has two social circles: Family and Colleagues as depicted in Figure 51.

IV.1.2 Spatial Intersection Matrix

To identify the spatial relation between e1 and e2, we define their intersection matrix with respect to L as follows:

𝐼𝑀 (𝐿 1 ,𝐿 2 ) = ( ∆ 𝐿1 ′ ∩ ∆ 𝐿2 ′ ∆ 𝐿1 ′ ∩ ∆ 𝐿2 ′′ ∆ 𝐿1 ′′ ∩ ∆ 𝐿2 ′ ∆ 𝐿1 ′′ ∩ ∆ 𝐿2 ′′ )
Each element of the 2x2 matrix corresponds to the intersection product between 'L1, 'L2, ''L1 or ''L2, i.e., either 0 or 1 depending on whether it is empty or not.

IV.1.3 Topological Spatial Relations

Figure 52 shows the 4-intersection-matrix and the graphical description of the 6 possible topological relations between two spatial intersection sets. The definitions of these relations are given below.

 disjoint: The spatial boundaries and interiors do not intersect.

 equal: The two events have the same spatial boundary and interior.

 meet: The spatial boundaries intersect but the interiors do not intersect.

 contains: The spatial interior and boundary of one event is completely contained in the spatial interior of the other event.

 inside: The opposite of contains.

 intersects: The spatial boundaries and interiors of the two events intersect.

IV.1.4 Illustration

We illustrate this on the example given at the beginning of this section: e1 is the trip to Sweden, and e2 is the trip to Lebanon. We fix the location threshold to country. ' and '' in Table 10 constitute the spatial intersection sets.

Table 10: Spatial intersection sets of two events of the example.

The intersections between these intersection sets are empty. Then, the intersection matrix will be written as follows:

IM (𝐿 1 ,𝐿 2 ) = ( 0 0 0 0 )
The result of ⩀ 𝐿 is read out row by row. For example, the above matrix results in 0000.

Therefore, we can say that these two events are spatially disjoint. 'T2 is obtained by the same methodology using event trigger and break.

 ''T1  T and ''T2  T: represent the temporal neighborhood of e1 and e2 respectively obtained by adding a time threshold ε to the trigger and the break points of each event.

Formally, the temporal boundary is represented as:

''T = [e.meta.ttr -ε, e.meta.ttr [ ⋃ ] e.meta.tbr, e.meta.tbr + ε] ''T2 is also calculated in the same manner.

IV.2.2 Temporal Intersection Matrix

To identify the temporal relation that exists between e1 and e2, we define the intersection matrix as follows:

𝐼𝑀 (𝑇 1 ,𝑇 2 ) = ( ∆ 𝑇1 ′ ∩ ∆ 𝑇2 ′ ∆ 𝑇1 ′ ∩ ∆ 𝑇2 ′′ ∆ 𝑇1 ′′ ∩ ∆ 𝑇2 ′ ∆ 𝑇1 ′′ ∩ ∆ 𝑇2 ′′ )
In order to calculate the elements of this matrix, we follow the same procedure described previously for the spatial feature. The intersection product is computed between 'T1, 'T2, ''T1 or ''T2.

IV.2.3 Topological Temporal Relations

Figure 53 shows the possible 4-intersection-matrix and the graphical description for the 6 relations between two intervals. The definitions of these relations are given below.

 Disjoint: The temporal boundaries and interiors do not intersect.

 Equal: The two events have the same temporal boundary and interior.

 Meet: The temporal boundaries intersect but the interiors do not intersect.

 Contains: The temporal interior and boundary of one event is completely contained in the temporal interior of the other event.

 Inside: The opposite of contains.

 Intersects: The temporal boundary of each event intersects with the temporal interior of the other event.

IV.2.4 Illustration

Again, we illustrate this on the example given at the beginning of this section: e1 is the trip to Sweden, and e2 is the trip to Lebanon. ' and '' in Table 11 constitute the temporal intersection sets.

Table 11:

Temporal intersection sets of two events of the example.

The interiors of e1 and e2 intersect. The same can be seen for the boundaries. Then, the intersection matrix will be written as follows:

IM (𝐿 1 ,𝐿 2 ) = ( 1 0 0 1 )
The result of ⩀ 𝑇 is read out row by row. For example, the above matrix results in 1001.

Therefore, we can say that these two events are temporally equal.

IV.2.5 Spatial vs. Temporal Relations

Figure 52 and Figure 53 show that the 4-intersections are the same for spatial and temporal relations, except for the intersect relation. This is discussed in the literature [188] for line-line relations and region-region relations. Line-line relations are identical to our temporal relations, whereas region-region relations are identical to our spatial relations. According to

[188], the difference is due to the fact that regions have connected boundaries, while lines have disconnected boundaries. This fact can be applied to the spatial and temporal relations as well.

IStmp e1 e2

' where:

 fl = SSNu0.fl is the association function used to identify the relationship type between u0 and a given user in SSNu0.

 Dtie is a distance function that returns the social tie between two users. It usually comes in three varieties: strong, weak or absent. In our work, we describe users with only either strong or weak tie relationships with u0. More details about social ties can be found in [190][191][192][193].

''S2 is calculated in the same manner.

We illustrate the social intersection set by referring to the example of Lisa. We can see the following in Figure 54.

 In the inner circle: the set of people that participated with u0 in the event. This is the social event interior;

IV.3.2 Social Intersection Matrix

Let S1 and S2 denote the respective social features of e1 and e2. To identify the social relation that exists between them, we define the intersection matrix as follows:

𝐼𝑀(𝑆 1 , 𝑆 2 ) = ( ∆ 𝑠1 ′ ∩ ∆ 𝑠2 ′ ∆ 𝑠1 ′ ∩ ∆ 𝑠2 ′′ ∆ 𝑠1 ′′ ∩ ∆ 𝑠2 ′ ∆ 𝑠1 ′′ ∩ ∆ 𝑠2 ′′ )

IV.3.3.2 Socially Meet relation

We say that two events e1 and e2 "socially meet" if the set of participants of one event is different from the others, but both sets belong to only one and same group in the social network of u0. In other terms, two events of u0, belonging to two groups (one with colleagues and the other with family members), cannot socially meet.

Our meta-model allows us to detect five possible cases with the socially meet relation:

Case 1: Participants of both events have strong ties with u0.

Case 2: Participants of both events have weak ties with u0.

Case 3: Participants of one event have weak ties with u0, and the other event includes some people that have strong ties with u0.

Case 4: Participants of one event have weak ties with u0, and the other event includes all people that have strong ties with u0.

Case 5: Both events include people that have strong ties with u0 and also people that have weak ties with u0.

We illustrate these five cases with tables in which we compare the social interior/boundary corresponding to e1 and e2, and compute the corresponding intersection matrix.

Case 1: Consider two events e1 and e2 which are detected in the social network of Lisa. Each event includes distinct groups of colleagues who share strong tie strength with her.

Case 2: Consider two events e1 and e2 which are detected in the social network of Lisa. Each event includes distinct groups of colleagues who share weak tie strength with her.

ISsoc e1 e2

Social relation

'

Rich, Gil Kouki, Naty 0110 strong-Meet-strong

''

Kouki, Naty Rich, Gil Case 3: Consider two events e1 and e2 which are detected in the social network of Lisa.

Each event includes distinct groups of colleagues. One event includes some of the colleagues who share strong tie strength with u0. The other includes colleagues who share weak tie strength with u0.

Or Case 4: Consider two events e1 and e2 which are detected in the social network of Lisa.

Each event includes distinct groups of her colleagues. One event includes all the colleagues who share strong tie strength with Lisa. The other includes colleagues who share weak tie strength with u0. The same intersection matrix is obtained whether the second group has all or some of the colleagues who share weak tie strength with u0. We explain this by the fact that people with weak tie strength will never appear in the boundary set. This can be seen in the following two examples, which have the same intersection matrix. Or Case 5: Consider two events e1 and e2 which are detected in the social network of Lisa.

Each event includes distinct groups of colleagues who share strong tie strength and also weak tie strength with her.

As a result, we can see from these examples that the "socially disjoint" relation has seven binary representations: 0110, 0001, 0101, 0011, 0010, 0100, and 0111.

IV.3.3.3 Socially Equal relation

We say that two events e1 and e2 are "socially equal" if the people who participated in the two events are exactly the same. Our meta-model allows to associate the "socially equal" relation with two representations, depending whether the boundary sets are empty or not. Boundary sets are empty when events include all people of one group who share a strong tie strength where both events include all the colleagues of Lisa, as shown below.

IV.3.3.4 Socially Inside relation

We say that an event e1 is "socially inside" another event e2 if the set of people involved in e1 is completely contained in the set of people involved in e2. Here also, our meta-model can associate the "socially inside" relation with two representations, depending whether the boundary sets are empty or not. To illustrate this, we consider the two cases:

Case 1: Consider two events e1 and e2 which are detected in the social network of Lisa, where e1 includes a subset of the people involved in e2. where e1 includes a subset of the people involved in e2. The only difference with Case 1 is that e2 includes all the colleagues of Lisa who share a strong tie strength with her.

IV.3.3.5 Socially Contains relation

We say that an event e1 "socially contains" another event e2 if the set of people involved in e1 comprises all the elements in the set of people involved in e2. The "socially inside" relation has two binary representations: 1100 or 1101, depending whether the boundary sets are empty or not. When boundary sets are empty, the "socially contains" relation will be 1100.

Otherwise, it will be 1101. It is the opposite of "socially contains". Therefore, we consider the same examples described for the "socially contains" relation, but reversed.

Case 1: Consider two events e1 and e2 which are detected in the social network of Lisa, where e1 includes a subset of the people involved in e2. where e1 includes a subset of the people involved in e2. The only difference with Case 1 is that e2 includes all the colleagues of Lisa who share a strong tie strength with her.

IV.3.3.6 Socially Intersect relation

We say that an event e1 "socially intersects" with another event e2 if there is an intersection in the set of people involved in e1 and e2. The "socially intersect" relation has two representations using our meta-model, depending on the cardinality of IS.

As we can see in Case 2, each event includes three colleagues having strong tie strength with u0. Two of them are common (Rich and Gil). Since there are a total of four colleagues who have strong tie with u0, the boundary will contain only one (and different) colleague in each event. Therefore, there is no intersection between boundaries. This is the only difference with Case 1 where more than one colleague appear in the boundaries. Consequently, the intersection between boundaries is not empty and yields a binary representation of the form: 1111, instead of 1110 in Case 2.

Case 1: Lisa had two events: e1 with two of her colleagues Kouki and Naty; and e2 with two of her colleagues Gil and Naty.

IV.3.4 Spatial vs. Temporal vs. Social Relations

We summarize this in Table 12 where we show the relations' representations.

V. Event Constraints

Our approach goes beyond identifying spatial, temporal and social relations between events.

Each of the spatial (R L ), the temporal (R T ) and the social (R S ) relation can result in one of the 6 different types of topological relations: disjoint, equal, meet, contains, inside, or intersects.

In this section, we propose to group all possible combinations of these three relations. This results in a total of 6 3 combinations. However, not all combinations are valid, as they can lead to impossible cases because of spatial, temporal and social constraints that we discuss here.

V.1 Time-based Constraints

Given two elementary events e1 and e2 that are temporally equal, participants cannot be the same if e1 and e2 take place in two different locations. This Temporality Constraint is based on our knowledge that a person cannot physically be in two places at the same time. Thus, we assume that a person cannot be involved in more than one event at the same time, similarly to [START_REF] Bao | Social event detection with robust high-order co-clustering[END_REF]125,194]. For example, Lisa is in Sweden or in Lebanon, therefore it is not possible that Lisa participates in one event in Sweden and another in Lebanon simultaneously.

V.2 Location-based Constraints

Let e1 and e2 denote two events that involve one or more persons in common, occur at different locations, and a time bound T ∈ N representing the interval between e1 and e2. The Reachability Constraint ensures that e1 and e2 cannot be located at a distance greater than Dmax, where Dmax represents the maximum distance that can be travelled in T by any mode of transport (e.g., roads, railways, inland waterways or airports, etc.).

V.3 Social-based Constraints

In social networks, people are connected by specific relationship (or link) categories like friends, colleagues or family. We assume that a user can have only one relationship with another user within a single social network. For instance, Lisa and Bouli are either relatives or colleagues. This assumption would distinguish event categories using the relationship types that exist in the social network (e.g., family events, professional events, etc.).

V.4 Pruning Relations Combinations

As we mentioned before, when examining event relations between i) people, ii) places and iii) time separately, it is necessary to check if the intersection of these three relations is also valid.

The time, location and social-based constraints are used for pruning the set of possible combinations of relations, so that only the valid ones are selected. All the possible cases are listed in Table 13, where:

 The rows list the 6 temporal relations (R T ), and  The columns list the 6 spatial relations (R L ).

The value of each cell corresponds to the names of the social relation when the combination of the three relations is possible to have in real-life events. Let us consider the first term in Table 13. It contains the set of social relations that are possible when two events are disjoint spatially and temporally. In this case, the 6 social relations are all possible (D, M, E, C, In, It).

Table 13:

The possible combinations of the three relations:

disjoint (D), meet (M), equal (E), contains (C), inside (In) and intersects (It) R L R T D, M, E, C, In, It
Table 13 shows that we can reduce the 6 3 = 216 possible cases to 3 groups: Group 1: There is no constraint on the combinations of relations when the temporal relation between events is disjoint or meet (as shown in the rows 2 and 3 of the above table). When events do not occur at the same time, they can involve whoever and wherever. Therefore, the spatial and the social relations can be any one of the aforementioned relations (D, M, E, C, In or It).

Group 2:

Based on the time constraint mentioned above and using the social constraint imposing the unicity of relationships between users, we can show that there are some impossible cases when two events are temporally equal (as shown in the fourth row of the above table). In this case, when the spatial relation between e1 and e2 is disjoint or meet, the social relation can be only disjoint or meet. We formulate this constraint as follows:

Group 3:

The reachability constraint is necessary for some cases to ensure that there is enough time to get from one event's location to another event's location, when the two events take place in different locations (as shown in the rows 4, 5 and 6 of the above table).

VI. Computing Links between Events

As we mentioned before, every two events are related by three relations: one spatial, one temporal and one social relation. Thus, event relations can be represented as:

R(ei, ej): (R L , R T , R S )
where:

 R L : is an exclusive spatial relation that ei has with ej, and corresponding to the l value of the axis of the social-R Space.

 R T : is an exclusive temporal relation that ei has with ej, and corresponding to the t value of the axis of the social-R space.

 R S : is an exclusive social relation that ei has with ej, and corresponding to the s value of the axis of the social-R space.

Now that we have the three exclusive relations between two events, we can define their Hamard Intersect product.

VI.1 From Intersection Set to Hamard Intersect Product

Let 𝐼𝑆 𝑒1 and 𝐼𝑆 𝑒2 denote the intersection sets of two elementary events e1 and e2.

VI.2 From Hamard Intersect Product to Binary Triplet

Suppose two events e1 and e2 occur in two different sites at the same time and include two different groups of colleagues in SSNu0. Then, the three intersection matrices are as follows:

⩀ 𝐿 = ( 0 0 0 0 ) ; ⩀ 𝑇 = ( 1 0 0 1 ) ; ⩀ 𝑆 = ( 0 1 1 0 )

As we mentioned before, the result of each intersection matrix is read out in our study row by row. This comes down to 12-bit (3×4-bit values) as we have three intersection matrices. Their Hamard product will be: This would help later to give a binary representation of complex relations, as explained in the next section.

VII. Complex Relations of Events

Linking events is not simple to compute when it is about personal events because such events represent human experience. Personal events might be connected in multiple complex ways.

Thus, the relations between them would be complex and need a variety of aspects to be met simultaneously. Complex relations cannot be determined using only temporal, spatial or social data. Combination of two or three aspects with other semantic descriptors (e.g., topic or thematic information, etc.) may be needed to find the correct relations.

In the following, we present first our form to express complex relations between events. Then, we use some classical ontological relations in order to identify semantic relations between events. The purpose here is to provide human-understandable semantics for events detected in shared photos. For example, this will allow us to find the relations between events and subevents or sub-collections, and also to find events that cause other events or inherit from other events.

VII.1 Logical Expression

We build an expression in order to define complex relations among events. We represent the expression denoting the relation in terms of logical operators and predicates. This expression is written as:

expr ⩴ [O] ( R | Predicate | expr)
where:

 expr is the name of the relation;

 O is a logical operator (AND, OR, NOT);

 R is a contextual relation with one, two or three features (spatial, temporal, social);

 Predicate is a predicate that uses user preferences (e.g., her favorite location), relationships between users (e.g., "colleagues"), or event metadata (e.g., "Dijon").

The choice of this particular expression allows the combination of one or more contextual relations to derive complex relations using logical operators. In addition, predicates can help in modelling complex events because they may provide valuable information to the event linking that is not revealed in the relational social graph. Hence, contextual parameters given by the user as predicates could be potentially useful to obtain additional relations that are meaningful to the user.

VII.2 Ontological Relations

In the following, we introduce some ontological relations between events and represent them as binary triplets. Examples will be given in the following definitions.

VII.2.1 isSameAs Relation

In our context and based on the multi-* property of events described in Chapter 4, the isSame- This first representation can be simplified to:

isSame-as ⩴ (1001 1001 100_)

For example, it is common that many people take photos using their smartphones during a wedding event. Photos taken in the event by different creators may be identified as different elementary events by our event detection algorithm. In essence, these multiple events happened in a single day, at the same exact location and with the same people. Therefore, they are linked by the relation: isSame-as.

ii. In some cases, a wedding event can span several days and has to be considered as one event. Therefore, the isSame-as relation holds between two elementary events that temporally meet even if they are related by the equality relation on only two aspects: spatially and socially. Thus, the isSame-as relation will be represented as follows: The second representation can also be simplified to:

isSame-as ⩴ (1001 0001 100_) iii. The following example illustrates a third particularity of some (wedding) events: they can be scattered at different sites, and sometimes people have to move into different cities in the same day. Again, the isSame-as relation holds between two elementary events that are related by the equality relation on these two aspects: temporally and socially. Thus, the isSame-as relation will be represented as follows: The third representation can also be simplified to:

isSame-as ⩴ (____ 1001 100_)

The three representations of isSame-as relation enable us to address the issue of multi-source, multi-day, multi-site and multi-participant events respectively.

VII.2.2 isSubevent-of Relation

The isSubevent-of relation (or the mereological relation in WordNet) reflects how events can be composed of two or more events. It indicates the part-whole hierarchical relations in events. An event e1 isSubevent-of another event e2 when there is: For instance, a keynote session is a sub-event of a big conference. The conference took place at the UPPA University in Bayonne from December 17 to December 19, 2014. The keynote session was scheduled for the first day in one conference room of the University. Over sixty persons attended both the conference and the keynote session. Thus, the following conclusions can be drawn:

i. Firstly, the location of the session is spatially inside the location of the conference.

ii. Secondly, the date of the session is within/ inside the start and end dates of the conference.

iii. Thirdly, the conference's participants include the same participants of the keynote.

Concretely, this isSubevent-of relation results in the following binary triplet:

(1001 1010 1000)

VII.2.3 isSubcollection-of Relation

The isSubcollection-of relation (or aggregation relation) occurs when a larger event can be viewed as a collection of smaller events. Such events cannot be viewed as sub-events of the larger event because they do not involve the same people. However, they could be of common topics of interest, with an aggregation over time and space. A topic is what the event is about.

Recently, several studies have been conducted on event topic detection in social networks [197][198][199][200][201]. We note that in this study we do not address event topic in a particular way. This will be addressed in a dedicated study. Thus, in addition to the topic similarity, an event isSubcollection-of a bigger event if we can observe the following:

 Spatial containment (inside), and  Temporal containment (inside).

Thus, the isSubcollection-of relation is represented as follows:

isSubcollection-of ⩴ (1001 1010 ____) isSubcollection-of ⩴ (1010 1010 ____)

For instance, the "official matches" are a collection of "Fifa World Cup" events. In 2014, the tournament began on 12 June and concluded on 13 July (the date of the championship match) 94 . 12 venues in twelve cities were selected for the tournament, all located in Brazil.

Let us consider two games of the Fifa World Cup that were played in a single day: 12 July.

The first one was located in São Paulo, while the second one was located in Rio de Janeiro.

These games are socially unrelated and cannot be connected to the big event for having common participants. However, they are linked by three relations:

 There is a "inside" relation between the two cities (São Paulo, Rio de Janeiro) and the country where the "Fifa World Cup" event was held (Brazil).

 The temporal relation "inside" holds between the day of the games 12 July and the "Fifa World Cup" event interval (12 June -13 July).

 They share the common topic of the big event: "soccer". In this work, we do not explore how to identify event topic.

Therefore, we conclude that the games played in the two different cities are sub collection of the larger soccer event in Brazil.

VII.2.4 caused-by Relation

Given two events e1 and e2, it is possible that e1 is the cause of e2, or that e2 is the cause of e1. This is the causal relation between two events. It helps to answer questions related to why a particular event took place. Causal relations require the modeling of causes and effects and should support the integration and use of different causal theories [202,203]. Therefore, despite the interesting modeling and properties of causal links, we do not address the "why" question in this work. It will be important for future work to study causal relations between events. 94 http://en.wikipedia.org/wiki/2014_FIFA_World_Cup#Match_summary

VII.2.5 is-a Relation

Like the inheritance relation in the ontology, the is-a relation between events represents the generalization or specialization relation between two events, where one event is a specialized version of another. For instance, the is-a relation between session and a keynote session is a subclass-superclass relation. The inheritance rules are the following:

 The instances of keynote session inherit features and properties present in the parent session.

 An instance of session can be member of only one of its subclasses at all instants of time.

The generalization and specialization relation result in event hierarchy where event types and subtypes are required such as a birthday party, a soccer game, etc. Typically, such relations between events need classifiers that are learned using a set of training images [204]. Such classifiers allow the computation of a decision or a probability that a photo is of a certain known event type. Yet, event types are not involved in our model.

VII.3 Application-based Relations

Some events may not have a common cause or aspect; however, they might be related along some particular aspects that make sense only to the user. Hence, these relations must be defined by the user's preferences using predicates. We consider some examples in order to illustrate this possible extension.

withFriendsGroup: is the relation between two events happening with u0' friends.

withFriendsGroup ⩴ withSameGroup ∧ fl(e.meta.S) = "friends"

where withSameGroup ⩴ (____ ____ ___1) is the relation between two events happening with the same group of participants in SSNu0. This means that the social boundaries of the two events must be identical. Similar expressions can be applied to the withFamilyGroup, withCollegeGroup relations, etc withAmy: is the relation between two events experienced with Amy, one of the friends of u0 and the users on SSNu0.

withAmy ⩴ fl(e.meta.S) CONTAINS "Amy" atAirports: is the relation between two events where pictures are taken in airports. For example, when taking a trip, sometimes people start taking photos at the airports, or in the take-off and landing.

atAirports ⩴ fl(e.meta.L) = "airport" atHome: is the relation between two home events (e.g., a dinner party at home, etc.).

atHome ⩴ fl(e.meta.L) = "home" atWeekends: is the relation between two events occurring at weekends, such as going to saturday parties, or going on a weekend trip.

atWeekends ⩴ fl(e.meta.T) = "saturday" OR "sunday" atWeeekendsWithFamily: is the relation between two events occurring at weekends with u0' family members.

atWeeekendsWithFamily ⩴ withSameGroup ∧ (fl(e.meta.S) = "family") ∧ (fl(e.meta.T) = "saturday" OR "sunday")

VIII. Conclusion

In this chapter, we have presented our matrix-based approach for identifying relationships between events. We described the different relationships that we are interested in. We first detailed the basic ones which are related to the spatial, temporal and social aspects of events.

Then, we showed how to combine the set of basic relations to obtain more complex relations.

As discussed above, event semantization is very important. Figure 56 represents the transitions from multimedia data, to event information, and finally to knowledge (linked events) using our approach. When detecting events from the collection of images in the user's social network, the issue of information semantics is one of the major challenges. This is what we attempted to address in this chapter, by employing a three-dimensional space drawn from the main characteristics of events (Who, Where, When).

The two main benefits that can be obtained from this approach are:

 The identification of equivalent events (i.e. related by the semantic relation isSame-as)

to deal with particular events: multi-source events, multi-participant events, multi-site events, and multi-day events, as described in the previous chapter.

 The enrichment of events with semantic relations to achieve semantically richer event detection.

In addition, the advantage of our approach is that it allows the homogeneous representation of relationships between various aspects: spatial, temporal and social, and takes into account all possible combinations of these relations. In addition to the basic relations, we exploited other complex relations. We modeled relations with a binary representation, which is supposed to be easy, flexible and extensible. Therefore, our approach is expected to be able to be applied for defining arbitrary relations that can be defined by the social network user based on his context or his preferences as well.

For optimal expression in future works, each matrix cell can also have one of the following value (but not limited to this):  A common face (CF): belongs to a person that often appears in the photos of u0


or posts photos containing the face of u0. The Jaccard coefficient [START_REF] Geng | Privacy Measures for Free Text Documents: Bridging the Gap between Theory and Practice, in Trust, Privacy and Security in Digital Business[END_REF] can be applied to capture the degree of co-occurrence of a given person and u0.

 An uncommon face (UF): belongs to a person that has only one occurrence in the same photo with u0 or has repeated occurrences over a specific period.

Chapter 6: Conclusion

"The citizens will divide between those who prefer convenience and those who prefer privacy."

-Niels Ole Finnemann

A professor and director of NETLAB,

DIGHUMLAB IN DENMARK

In Chapter 4, we focused on exploiting photos' metadata and tags shared in online social networks to detect personal events, sometimes of capital importance for privacy protection.

To do so, we introduced the Social Space based on the three main aspects of event: temporal, geographical and social, and which are used in the description of our approach. We also defined an elementary event and proposed a method to cluster photos shared by social network users or by their friends/contacts in a social network. Our algorithm relies on a preprocessing step that defines the spatio-temporal granularities in order to obtain better quality clusters of events. We developed a prototype called Foto2Event to validate and demonstrate the efficiency of our event detection approach presented in this work. We successfully demonstrated that our theoretical analysis is accurate in predicting the appropriate granularities, since it was in good agreement with the experimental observation. We also tested the relevance of our proposal using the MediaEval dataset and compared our results to other algorithms implemented in the Social Event Detection MediaEval Benchmarking Initiative for Multimedia Evaluation 95 .

In Chapter 5, we studied relations that exist between events in order to strengthen further the detection of personal information (directly and indirectly). We designed a meta-model to represent various relations in a unified and flexible way. First, we identified temporal, spatial and social relations that exist between any two events based on an extension of the 4-Intersection Model. Then, we computed these basic relations together to produce more complex relations. Our meta-model is also capable to represent other types of relations between events: a) Basic relations: temporal, spatial and social relations based on topological relations (disjoint, meet, equal, contains, inside, and intersects), b) Ontological relations (e.g., isSameAs, isSubevent, isCausedBy, etc.), and c) Application-based relations (e.g., atHome, atAirport, etc.). 95 http://www.multimediaeval.org/mediaeval2014/sed2014/

II. Future Works

There are several improvements we intend to make to our existing contributions and validation.

II.1 Improving theoretical Approach

First, we hope to improve our sanitizing algorithm using dedicated techniques that anonymize the visual content of multimedia documents while preserving their semantics and coherence (e.g., Repetitive denoising (RD), Adaptive PRNU denoising (APD), etc.).

Second, more investigations are still needed to improve our event detection approach:

 Capture the metadata of an event (representative moments, locations, and people/groups) by reasoning on the significant-frequent and significant-rare items (metadata) from one event photos.

 Filling missing metadata of photos from the obtained events and their links, in order to produce better clustering results.

 Explore the "What", "Why" and "How" aspects of an event, since so far our Social Graph represents the "Who", "Where" and "When" of an event. This extension is expected to give a more exact analysis (clustering, link computing, and reasoning) from events detected on social networks, and therefore can help to discover the semantics of users' events.

II.2 Improving Validation

In the future, we plan to enhance our Foto2Event prototype by generating tags for photos and relationships between users (e.g., friends, colleagues, relatives, etc.). We intend to provide a public web-based version since it could help to test and validate our obtained results. Finally, we plan to implement and test our model of event relations, and then link it to our sanitizing algorithm to integrate all our proposed algorithms and techniques.

In the following, we present some of the possible research directions.

II.3 New Research Directions

As for new directions in research, we need to quantify the quality of information retrieved from events before using it to enrich users' profiles. Measuring quality requires reliable measures of data accuracy, while taking into account the evolution of (event) information over time (e.g., timeliness, freshness, up-to-dateness, etc.).

Another interesting aspect to consider in the future is the user's engagement in events. In other words, we are interested to measure the user's interest degree in a particular event based on his personal information and his personal events and their links as well (events shared by himself or by his connections):

 For a past event, we will compute the likelihood degree of user's involvement in an event, and infer his presence even if he did not appear in any photo of that cluster.

 For an upcoming event, it would be possible to predict if the user might be interested in an event. If so, we can predict if he will be able to participate or not, by calculating the probability of the user being elsewhere when that event will happen (Event prediction).

Another future direction that we would like to explore is sentiment analysis in events detected in online social networks. We are interested in detecting the feeling of a user in a particular event. This would allow classifying events as "happy", "sad" or "neutral", by mining text accompanying photos, recognizing facial expressions, identifying the event type (e.g., party, birthday, sickness, death, etc.).

Figure 1 :

 1 Figure 1: The percentage of Internet users in each age group who use social networking sites, over time, 2005-2013. (Source: Pew Research Center's survey, 2013).

Figure 2 :

 2 Figure 2: The percentage of adults who use Facebook, LinkedIn, Pinterest, Instagram, and Twitter, by year from 2012 to 2014. (Source: Pew Research Center's survey, 2014).

Figure 3 :

 3 Figure 3: A list of the most popular types of content shared on social networks. (Source: IPSOS report, 2013).

Figure 4 :

 4 Figure 4: The number of photos uploaded and shared per day on selected platforms from 2005 to 2013. (Source: KPCB estimates based on publicly disclosed company data, 2014).

Figure 5 :

 5 Figure5: The Twitter account of François Fillon with the fake nickname "@fdbeauce".

Figure 6 :

 6 Figure 6: A fragment of the Wikipedia's page of François Fillon.

Figure 7 :

 7 Figure 7: The profile image on "@fdbeauce" Twitter account.

Figure 8 :

 8 Figure8: Metadata of the profile image on "@fdbeauce" Twitter account.

Figure 9 :

 9 Figure 9: Public information existing online (image and description) of "Château de Beaucé", which is part of the cultural heritage of France.

Figure 10 :

 10 Figure 10: The tweet of "@fdebeauce" on October 23, 2011 and the French News reporting François Fillon's visit to Japan for two days.

Figure 23 :

 23 Figure 23: Illustration of atomic and composite events in [108].

Figure 24 :

 24 Figure 24: Multimedia (metadata) exploration framework presented in [107].

Figure 25 :

 25 Figure 25: Examples of typical image descriptions using our multimedia object representation. Figure 1(a) and Figure 1(b) show similar content, contained in two different multimedia documents.

Figure 26 :

 26 Figure 26: Privacy violation evaluation.

Figure 27 :

 27 Figure 27: Uncertainty evaluation.

Figure 28 :

 28 Figure 28: Utility evaluation.

Figure 29 :

 29 Figure 29: Computational cost evaluation.

Figure 31 :

 31 Figure 31: Example of a photo with its metadata and faces of tagged persons.

Figure 32 :

 32 Figure 32: Representation of the event social space in a 3D graph.

Figure 33 :

 33 Figure 33: Some photos shared on SSNu0 created by five different persons, including u0.

Figure 34 :

 34 Figure 34: Some photos shared on SSNu0 created by Lisa and Regi where new participants (Joseba and Irvin) are identified.

Figure 35 :

 35 Figure 35: Some photos shared on SSNu0 taken over the four days of the trip.

Figure 36 :

 36 Figure 36: Some photos shared on SSNu0 taken in different cities during the trip.

Figure 37 :

 37 Figure37: Description of our approach to detect elementary events.

Figure 38 :

 38 Figure 38: Architecture of our framework to detect events.

Figure 39 :

 39 Figure 39: F-measure scores considering different time distributions.

Figure 40 :

 40 Figure 40: NMI scores considering different time distributions.

Figure 41 :

 41 Figure 41: F-measure scores considering different location distributions.

Figure 42 :

 42 Figure 42: NMI scores considering different location distributions.

Figure 43 :

 43 Figure 43: F-measure and NMI scores considering different uploaders distributions.

Figure 44 :

 44 Figure 44: F-measure and NMI results considering different temporal information availabilities.

Figure 45 :

 45 Figure 45: F-measure and NMI results considering different geographic information availabilities.

Figure 46 :

 46 Figure 46: F-measure and NMI results considering different uploader information availabilities.

Figure 47 :

 47 Figure 47: F-measure and NMI results with different metadata availabilities.

Figure 48 :

 48 Figure 48: Time performance w.r.t. dataset size.

Figure 49 :

 49 Figure 49: Representation of the social-R space in a 3D graph.

Figure 50 :

 50 Figure 50: Transition from the social space (on the left) to the social-R space (on the right).

Figure 51 :

 51 Figure 51: Lisa's social connections. Two social circles: Family: Elie, Bouli; Colleagues: Rich, Gil, Kouki, Naty, Solomon, and Minale.

Figure 52 :

 52 Figure 52: Set of the six possible spatial relations between events (with their graphical description and their corresponding intersection matrices).

Figure 53 :

 53 Figure 53: Set of the six possible temporal relations between events (with their graphical description and their corresponding 4-intersection matrices).

Figure 54 :

 54 Figure 54: Illustration of event social interior and boundary.

Figure 55 :

 55 Figure 55: Set of the six possible social relations between events (with their graphical description and their corresponding intersection matrices).

Figure 56 :

 56 Figure 56: The Event Semantization process.

Chapter 1 :

 1 Introduction "The European Commission says data privacy is a basic human right, and yet Facebook and other social networks have over a billion people publicly broadcasting their personal data every day." -Steve Mattey Chief operating officer VCCPme

Figure 1 :

 1 Figure 1: The percentage of Internet users in each age group who use social networking sites, over time, 2005-2013. (Source: Pew Research Center's survey, 2013).

Figure 2 :

 2 Figure 2: The percentage of adults who use Facebook, LinkedIn, Pinterest, Instagram, and Twitter, by year from 2012 to 2014. (Source: Pew Research Center's survey, 2014).

Figure 3 :

 3 Figure 3: A list of the most popular types of content shared on social networks. (Source: IPSOS report, 2013).

Figure 4 :

 4 Figure 4: The number of photos uploaded and shared per day on selected platforms from 2005 to 2013. (Source: KPCB estimates based on publicly disclosed company data, 2014).

  18 http://iknowwhereyourcatlives.com/cat/e3fb32d4a019 https://www.flickr.com/ 20 http://twitpic.com/

Figure 6 :

 6 Figure 6: A fragment of the Wikipedia's page of François Fillon.

Figure 8 :

 8 Figure 8: Metadata of the profile image on "@fdbeauce" Twitter account.

Figure 9 :

 9 Figure 9: Public information existing online (image and description) of "Château de Beaucé", which is part of the cultural heritage of France.

Figure 10 :

 10 Figure 10: The tweet of "@fdebeauce" on October 23, 2011 and the French News reporting François Fillon's visit to Japan for two days.



  Split: conflicting data is sent online,  Disappearance: authentic data drowned in many other data,  Prosperity: data are seconded by similar data,  Shyness: data are minimized in their content and in their reading,  Deception: data are sent based on previous data,  Punctuality: Data are sent in a regular and precise way.

Figure 12 :

 12 Figure 12: Online Social Footprint defined in [16].

Figure 13 :

 13 Figure 13: Attribute list of personal information explored in [17].

  38 http://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance 39 www.blogger.com/ 40 http://www.livejournal.com/ These measures were computed on the profiles initially collected. Observations showed how aggregation of information across OSNs allows gathering additional information through: i. Complementary: more than half of the set of combined attributes from any two OSN profiles of the same user are complementary to each other.

Figure 14 :

 14 Figure 14: System architecture presented in [21].

  based technique. The first disambiguation technique employs a self-training strategy. The choice of a selftraining strategy is due to the limitation in the amount of the seed data. Instances from the initial seed data are first supplied to the machine learning classifier. Then a classification model is constructed based on the similarity of their features.

  to represent online posts shared across social networks and their embedded knowledge. More importantly, personal information in DLPO is gathered from different types of posts within different social networks: Message, MultimediaPost, WebDocumentPost and PresencePost. DLPO integrates concepts from PIM models and existing standards namely the SIOC ontology and several domain ontologies (i.e., NIE, PIMO, DAO, and NAO).

  Personal information is gathered from different types of posts within different social networks. It consists of four types of posts: Message, MultimediaPost, WebDocumentPost and PresencePost. A post item can occur individually or in conjunction with other post items. The DLPO integrates concepts from PIM models andexisting standards namely the Semantically-Interlinked Online Communities (SIOC) ontology[START_REF] Bojars | Towards Semantically-Interlinked Online Communities. Call for Take up Actions[END_REF] and many domain ontologies (e.g., the Information Element ontology63 (NIE), the Personal Information Model Ontology64 (PIMO), the Account Ontology65 (DAO), the Annotation Ontology 66 (NAO)). More specifically, the SIOC ontology defines primitives for describing a user in social web sites and online communities, the content he produces and the actions of other users on this content. NIE, PIMO, DAO and NAO are part of the NEPOMUK67 ontologies. Through the integration of these ontologies, the DLPO ontology enables to describe dynamic personal information in a concise and semantic way. It defines semantic links between: i. user posts from different social networks;

Figure 17 :

 17 Figure 17: Overview of the distributed di.me system architecture in [63].

  70 http://www.oscaf.org/. The "Open Semantic Collaboration Architecture Foundation" (OSCAF) is an industry led group bringing together organizations and individuals interested in ensuring interoperability between next generation desktops and collaborative environments.

Figure 18 :

 18 Figure 18:The approach process proposed in[START_REF] Cortis | Discovering Semantic Equivalence of People behind Online Profiles[END_REF].

  approaches on digital identity provide solutions for the aggregation of personal information across multiple Web 2.0 platforms. Nevertheless, most studies are restricted to textual data. In the meantime, approaches including images are based on low-level features.

Figure 20 :

 20 Figure 20: Basic aspects of event description from [84].

Figure 21 :

 21 Figure 21: Event detection framework presented in [82, 83].

  72 https://en.wikipedia.org/wiki/Main_Page 73 https://www.google.com/

Figure 22 :

 22 Figure 22: Architecture diagram of event detection approach proposed in [81].

Figure 23 :

 23 Figure 23: Illustration of atomic and composite events in [108].

Figure 24 :

 24 Figure 24: Multimedia (metadata) exploration framework presented in [107].

Figure 25 :Definition 3 Definition 4 

 2534 Figure 25: Examples of typical image descriptions using our multimedia object representation. Figure 1(a) and Figure 1(b) show similar content, contained in two different multimedia documents.

  Selective intersection returns a normalized score ∈ [0, 1] computed based on equality, similarity and cross-matching of multimedia documents content. For instance, let us compute the selective intersection between 𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 and both 𝑆 𝑀𝐷 𝛽1 and 𝑆 𝑀𝐷 𝛽2 . We adopt the max aggregation function to compute the equality, similarity and/or matching scores for each attribute and finally determine their average score. The selective intersection based on the attributes Country, Event and Location is detailed below: 𝑆 𝑒𝑙 𝐼 𝑛𝑡 (𝑆 𝑀𝐷 𝐹𝑖𝑙𝑙𝑜𝑛 , 𝑆 𝑀𝐷 𝛽1 )

IV. 2

 2 Multimedia Document Sanitization: MD * -algorithm MD * -algorithm is used to sanitize a multimedia document and protect the cyberstalkee's identity. As mentioned in the pseudo-code, the algorithm takes a multimedia document MDu, a set of attributes As (used to extract multimedia document signature), the cyberstalkee profile pfu along with Eu and both association and identification thresholds α, β. It returns a sanitized multimedia document (𝑀𝐷 𝑢 * ). The MD * -algorithm extracts in Step 1 the multimedia document signature 𝑆 𝑀𝐷 𝑢 using the extraction function IA. It sanitizes MDu from Step 2 to 10. In Step 3, it extracts the signature of a multimedia document MDβ retrieved from an external source E based on the set of entities Eu related to u. In order to determine the amount of information related to u and that can be obtained from MDβ, we compute the selective intersection on MDβ and the cyberstalkee profile pfu. If their selective intersection 𝑆 𝑒𝑙 𝐼 𝑛𝑡 (𝑆 𝑀𝐷 𝛽 , 𝑝𝑓 𝑢 ) is greater than β, the link between MDu and MDβ should be anonymized as done from Step 5 to 8. That is, as long as they are α-associated, the least 84 significant Wβ and

1 𝛽=association thresholds 1 𝛼= 2 , 4 , 6 , 8 and 10 .

 1124610 percentage of individuals identified according to what they have published in their MDu and 88 https://dev.twitter.com/ 89 https://dev.twitter.com/ 90 http://phash.org/docs/howto.html their related multimedia documents MDβ. We fixed the identification threshold 10 in order to capture a significant number of multimedia documents related to individual u and used various The results shown in Figure 26 show the percentage of re-identified individuals (in Figure 26 (a)) and the number of threatening MDβ (in Figure 26 (b)).

Figure 26 :

 26 Figure 26: Privacy violation evaluation.

  Figure 27.

Figure 27 :

 27 Figure 27: Uncertainty evaluation.

Figure 28 :

 28 Figure 28: Utility evaluation.

  𝑂(|σ 𝐸 𝑢 (𝐸)| × (|𝑊 * | + |𝑀𝑂 * | × 𝑧) ≈ 𝑂(|σ 𝐸 𝑢 (𝐸)| × (|𝑊 * | + |𝑀𝑂 * |) where:  |σ 𝐸 𝑢 (𝐸)| is the number of relevant multimedia documents retrieved from the external source,  |𝑊 * | + |𝑀𝑂 * | is the number of sanitized words and multimedia objects from MDu, and  z is the number of attributes used by the selective intersections (which is limited and low).

1 𝛼= 4

 14 to 6 where the time to retrieve the individuals' data from the external source has increased (Figure29).

Figure 29 :

 29 Figure 29: Computational cost evaluation.

  within different directions are different. Different distances can be used such as the Haversine's distance [125], the Euclidean distance between places [147], the Hausdorff distance between two finite set of points [148] , etc.  T: is the set of the datetime values identified in the social network of u0, ordered according to a given distance dist∆T from u0's date of birth. The units of T can be one of the following values: hour, day, week, month, or year. Exponential decay can be used to measure temporal distance [149, 150]. S: is the set of users' names identified in the social network of u0, ordered by the alphabetical order. u0 is the starting element. We assume here that each person has a unique name. A string-based distance dist∆S can be used to decide whether the names are equal or disjoint.

Figure 30 :

 30 Figure 30: The social space , modeled as a three-dimensional space: locations, times, and people names.

Figure 31 :

 31 Figure 31: Example of a photo with its metadata and faces of tagged persons.



  SB (I', '): is the story board describing the event e where: o I' I:is the collection of photos taken in the event e and published on SSNu0; o '   is the social space of the event e representing the geographical, temporal and social information related to this event such as:  L' L: denotes the minimum bounding polygon (MBP) [151, 152] of the location in which the event took place. A method is disclosed in [152] for determining a MBP for use in a geo-coding system: a first polygon is received defining a geographic region. A minimal number of points is determined corresponding to the first polygon to define a second polygon at least bounding the first polygon.

Figure 32 :

 32 Figure 32: Representation of the event social space in a 3D graph.

Figure 33 :

 33 Figure 33: Some photos shared on SSNu0 created by five different persons, including u0.

  However, it is important to note that Lisa does not appear in any photo with Joseba nor with Irvin. Since Joseba and Irvin are identified in the photos of Regi, and Regi has many photos with Lisa, one can conclude that Joseba and Irvin were involved in the trip event made by Lisa.

Figure 34 :

 34 Figure 34: Some photos shared on SSNu0 created by Lisa and Regi where new participants (Joseba and Irvin) are identified.

Figure 35 :

 35 Figure 35: Some photos shared on SSNu0 taken over the four days of the trip.

Figure 36 :

 36 Figure 36: Some photos shared on SSNu0 taken in different cities during the trip.

  are provided in the literature and can be divided into two broad categories: hierarchical and nonhierarchical. Hierarchical clustering algorithms produce nested sets of data (hierarchies), in which pairs of elements or clusters are successively linked until every element in the data set becomes connected [154, 155]. Nonhierarchical methods group a data set into a number of clusters irrespective of the route by which they are obtained. For instance, the k-means [156] groups the data set into k different clusters based on similarity, while density-based clustering [157] can create clusters with arbitrary shapes based on a metric space.

Chapter 4 :

 4 Personal Event Detection from OSN Photo's Metadata 123 Recent studies [160, 161] have shown that the mutual information measure, being sensitive to the amount of overlap between two sets, can be normalized. Normalized mutual information (NMI) has been widely used in a lot of applications to measure the performance of clustering methods [162-164]. In our experiments, we apply the NMI measure to check the overlap between our clustering result and ground truth clusters.

Group 1 :

 1 2006, 2007, 2008, 2009) 5 years(2006, 2007, 2008, 2009, 2010) 6 years(2006, 2007, 2008, 2009, 2010, 2011) The distribution over time is not constant in the original dataset. It consists of images from Flickr with an upload time between January 2006 and December 2012.

Figure 39 :

 39 Figure 39: F-measure scores considering different time distributions.

Figure 40 :

 40 Figure 40: NMI scores considering different time distributions.

Figure 41 :

 41 Figure 41: F-measure scores considering different location distributions.

Figure 42 :

 42 Figure 42: NMI scores considering different location distributions.

Group 3 :

 3 As already mentioned above, the photos in the dataset were uploaded by different users on Flickr. First, we collected the photos of the main userthe one who has uploaded more photos. Then, we included photos which are uploaded by a specific number of other users chosen randomly. The number of distinct users, including the main user varies from 1 to 1000. Following the previous results, we fixed the time-space granularities to day and street (based on Experiment 1 and Experiment 2). Then, we ran the algorithm. The results are shown in Figure43.

Figure 43 :

 43 Figure 43: F-measure and NMI scores considering different uploaders distributions.

Figure 44 :

 44 Figure 44: F-measure and NMI results considering different temporal information availabilities.

Figure 45 :

 45 Figure 45: F-measure and NMI results considering different geographic information availabilities.

Figure 46 :

 46 Figure 46: F-measure and NMI results considering different uploader information availabilities.

Figure 47 :

 47 Figure 47: F-measure and NMI results with different metadata availabilities.

Figure 48 :

 48 Figure 48: Time performance w.r.t. dataset size.

Figure 49 :

 49 Figure 49: Representation of the social-R space in a 3D graph.

 The 4 -

 4 have been developed, from which the most used are the 4-intersection model and its extensions, such as the 9-intersection model [188]. Intersection model (4IM) represents topological relations by calculating the four intersections of the interior and the boundary of two objects (i.e., emptiness and nonemptiness). This initial model for binary topological relations was developed for 2dimensional objects (e.g., two regions, etc.). It consists on intersection sets and an intersection vector.  The 9-intersection model (9IM) is an extended representation of the 4-intersection model. It adds to the 4-Intersection model additional intersections related to the

Figure 51 :Let

 51 Figure 51: Lisa's social connections. Two social circles: Family: Elie, Bouli; Colleagues: Rich, Gil, Kouki, Naty, Solomon, and Minale.

Figure 52 :

 52 Figure 52: Set of the six possible spatial relations between events (with their graphical description and their corresponding intersection matrices).

Let

  L1 and L2 denote the respective temporal features of e1 and e2. The definitions of their temporal intersection sets are given below:  'T1  T and 'T2  T: represent the date/time interval in which e1 and e2 took place respectively. For instance, 'T is situated between [ttr, tbr] ⊆ ∆T, where: o tr: denotes the trigger data/time of the event extracted from the first photo published in e (∀i  ', ti ≥ e.meta.ttr), and o br: denotes the break point of an event derived from the last photo published in e (∀i  ', ti ≤ e.meta.tbr).

Figure 53 :

 53 Figure 53: Set of the six possible temporal relations between events (with their graphical description and their corresponding 4-intersection matrices).



  In the outer circle: the names of the colleagues of u0 (since Kouki and Naty are linked by the colleague relationship with u0), who have a strong social tie with u0. This is the social event boundary;  In the expanding circle: the rest of the colleagues. In this study, we do not consider the exterior to determine the intersection between features of events.

Figure 54 :

 54 Figure 54: Illustration of event social interior and boundary.

Figure 55

 55 Figure55shows the 4-intersection-matrix and the graphical description of the 6 possible topological relations between two social intersection sets.

Figure 55 :

 55 Figure 55: Set of the six possible social relations between events (with their graphical description and their corresponding intersection matrices).

(

  The terms in the first parentheses correspond to the spatial relations (disjoint, meet) separated by the pipe (|) sign to indicate "or";  The terms in the second parentheses correspond to the temporal relation equal;  The terms in the third parentheses correspond to the possible social relations (disjoint, meet) separated by the pipe (|) sign indicating "or".



  Therefore, when the social relation is neither disjoint nor meet (because of the social constraint), the reachability constraint has to be checked. If the time difference and the distance between the two events satisfy the reachability constraint, the temporal relations can be any one of the aforementioned relations (D, M, E, C, In or It). We formulate this constraint as follows:( ) (D | M | E | C | In | It).RC ( & )where: The terms in the first parentheses correspond to the spatial relations not equal; The terms in the second parentheses correspond to the possible temporal relations separated by the pipe (|) sign to indicate "or". It is indicated that the reachability (RC) constraint has to be checked; The terms in the third parentheses correspond to the social relations (not disjoint, not meet) separated by "&" indicating "neither";E D M



  𝐼𝑆 𝑒1 . ∆ 𝑇 ⩀ 𝐼𝑆 𝑒2 . ∆ 𝑇 : denotes the temporal relation between e1 and e2. It is obtained by applying the intersection matrix between ISe1and ISe2 with respect to the temporal feature T. 𝐼𝑆 𝑒1 . ∆ 𝑆 ⩀ 𝐼𝑆 𝑒2 . ∆ 𝑠 : denotes the social relation between e1 and e2. It is obtained by applying the intersection matrix between ISe1and ISe2 with respect to the social feature S.

𝑅(𝑒 1 ,)

 1 𝑒 2 ) = °⩀ (𝐼𝑆 𝑒1 , 𝐼𝑆 𝑒2 ) We convert the result of the Hamard product to a binary triplet. It will have the following value: 𝑅(𝑒 1 , 𝑒 2 ) = 0000 1001 0110

as

  relation can have four possible values: i. The isSame-as relation (or the synonymy relation in WordNet [196]) corresponds to the relation between two events with the same space-time and social features. An event e1 isSame-as another event e2 if e1 and e2 are related by the equality relation on the three event aspects: spatially, temporally and socially. Thus, the isSame-as relation can be represented as follows: isSame-as ⩴ (1001 1001 1001) OR (1001 1001 1000) such as: The first 4-bit 1001 is for the spatial relation (E); The second 4-bit 1010 is for the temporal relation (E); The third 4-bit 1001 or 1000 are for the social relation (E). We recall that the relation socially equal has two representations.



  isSame-as ⩴ (1001 0001 1001) OR (1001 0001 1000) such as:  The first 4-bit 1001 is for the spatial relation (E); The second 4-bit 0001 is for the temporal relation (M); The third 4-bit 1001 or 1000 are for the social relation (E);



  isSame-as ⩴ (____ 1001 1001) OR (____ 1001 1000) such as: The first ____ is for any spatial relation (not equal); The second 4-bit 1001 is for the temporal relation (E); The third 4-bit 1001 or 1000 are for the social relation (E);



  A temporal containment (inside), and  A social equality, intersection or containment (equal, intersects, contains or inside). Thus, the isSubevent-of relation can be represented in two forms: i. isSubevent-of ⩴ (____ 1010 11__) OR (____ 1010 10__) such as:  The first 4-bit is for the spatial relation;  The second 4-bit 1010 is for the temporal relation (In);  The third 11__: is to simplify four social relations: 1100 (C), 1101 (C), 1110 (It), 1111 (It). ii. isSubevent-of ⩴ (____ 1010 11__) or (____ 1010 10__) such as: The first 4-bit is for the spatial relation; The second 4-bit 1010 is for the temporal relation (In); The third 10__: is to simplify four social relations: 1000 (E), 1001 (E), 1010 (In), 1001 (In).

Figure 56 :

 56 Figure 56: The Event Semantization process.



  The cardinality of the intersection set, instead of only distinguishing empty or nonempty intersections as in the DE-9IM (Dimensional Extended nine-Intersection Model),  A general concept such as:  A common place (CP): represents a location where u0 spends the most of his time. This information may be provided in the user profile (e.g., home and work locations). An uncommon place (UP): represents a location that is away from CP. It can be determined based on the metric distance of geographical locations (gi = gi -gCP).

Table 2 :

 2 Comparison of ontological approaches.

Table 3 :

 3 Comparison of online identity approaches.

Table 4 :

 4 A summary of photo-based event detection approaches.

Table 5 :

 5 Comparison of event relations approaches.

Table 6 :

 6 Notations used in our approach.

Table 7 :

 7 Distribution of metadata in each dataset group.

Table 8 :

 8 Availability of metadata in each dataset.

Table 9 :

 9 F-value and NMI scores considering different granularities of time and space on the

	Setfull dataset.

Table 10 :

 10 Spatial intersection sets of two events of the example.

Table 11 :

 11 Temporal intersection sets of two events of the example.

Table 12 :

 12 The binary and hexadecimal values of the three basic relations (spatial, temporal

	and social)

Table 13 :

 13 The possible combinations of the three relations:

Table 1 :

 1 Comparison of inference control approaches.

			Approaches	Background knowledge Data	Output
					type
	Database inference		k-anonymity protection generalization suppression [36] k-anonymity: a model privacy using and for protecting privacy [30]	External information from structured relational datasets	Text Person-specific that cannot be linked to data other information Text Private data where the subjects of the data cannot be re-identified
	Ontology-based	Inference	Privacy measures for [39] free text documents [38] Preserving privacy on the searchable Internet	Domain specific ontologies	Text Score of sensitivity of information Text Inferrble information about an individual
			Efficient techniques for		Text Minimum terms to be
			document sanitization			removed	for	the
			[48]			document	to	be
				Context of words /		sanitized
			Finding user semantics	entities	Text User's semantics
	Web-based Inference		on the web using word co-occurrence information [44] Web-based inference Detection [31] Large online social	Public web documents, Sensitive keywords User's pseudonym(s)	Text Inferences that can be drawn about the user Text User's information
			footprints [16]			from social networks
			Studying user footprints	User's profile URL	Text	User's	digital
			in social networks [21] different online		Image +	footprints
	Several techniques have been defined in the literature

  It represents the user himself and the fact that he has a Personal Information Model.

http://www.semanticdesktop.org/ontologies/2007/11/01/pimo/v1.1/pimo_v1.1.pdf 51 http://www.semanticdesktop.org/ontologies/2007/11/01/pimo/# 52 http://www.semanticdesktop.org/ontologies/nco/ 53 D. Brickley and L. Miller. FOAF: the 'friend of a friend' vocabulary. http://xmlns.com/foaf/0.1/, 2004.

54

"The friend of a friend (foaf) project," http://www.foaf-project.org/.

Table 2 :

 2 Comparison of ontological approaches.

	Ontology

name Personal Information User-generated content Images Metadata PIMO

[61]

Table 3 :

 3 Comparison of online identity approaches.

	Online
	Social
	Footprint
	[16]

Table 4 :

 4 A summary of photo-based event detection approaches.

	Approach	Domain	Metadata	Text	Visual	Other's	Event	Dataset of
				(description,	content	resources	links	experiments
				tags)
	[82]	Social						Flickr
		media
		content
	[91]	News						www.reuters.com
		events
	[83]	Personal						Private photo
		photo						collection
		collections
	[78]	Social						Event-Media
		networks
		discovery
	[73]	Social						MediaEval SED
		media
		sharing

From the above table, one can see the following limitations:

  t. time or location. The first step of the framework contains a pre-selection of data (e.g., images, videos) from different repositories using user-supplied keywords, and filtered based on a date period inserted by the user. The second step is the clustering of Flickr photos and Youtube 75 videos into clusters representing

sub-events, based on a Self-Organizing Map 76 (SOM). SOM takes as input a subset of relevant words computed based on the tf-idf scores. Results of this work

[107]

Table 5 :

 5 Comparison of event relations approaches.

		Spatial	Temporal	Social	Complex	Ontological	User-
		relations	relations	relations	relations	events	defined
	Approaches					relations
	Eventory [109]
	DROP [81]
	EDF [106]
	EDF E [113]
	ECO [108]
	Event ontology
	[115]
	Crisis	related
	sub-events
	[107]

Table 6 :

 6 Notations used in our approach. Attribute Set: A is a set of attributes where ∀ai ∈ A for (1 ≤ i ≤ |A|), ai can be any attribute of the Dublin core metadata element set78 such that {source, description, date, contributor, format} or the MPEG-7 semantic set79 {semantic_place, concept, state, event, object} or any domain specific attribute (e.g., spatial or temporal domain).We use mai ∈ A to denote a multimedia attribute whose values are of complex structure such as a BFILE/BLOB, an URL/URI, an URL/URI augmented with a primitive to represent a salient object (e.g., Multimedia Object: Let mo be any type of multimedia data such as an image, a video, or a salient object describing an object of interest (e.g., face of a person.). mo is formally represented as:

	u	an individual with anonymized identity
	pfu	an individual profile
	mo	a multimedia object
	MDu	a multimedia document of u to be sanitized
	MDβ	a multimedia document publicly accessible to adversaries extracted from an external source E
	SMD	a multimedia document signature
	E	an external source such as the social website, domain specific database, etc.
	A	a set of attributes relevant to the multimedia document content
	α	an association threshold
	β	an identification threshold
	ϖ	an aggregation function such as average, minimum, max, etc.
	Rw	words relevance
	Rm	multimedia relevance
	U	multimedia document utility
	III.1 Data Definition
	Definition 1 

78

http://dublincore.org/ 𝑚𝑜: < 𝐴 𝑚 , 𝑉, 𝑂, 𝑀𝑂, 𝜍 > where:

  𝑚𝑜 𝑗 2 are two multimedia objects of MO1 and MO2 respectively where m =|MO1| and r = |MO2|. Definition 8  Cross-Matching Score: Let 𝑆 𝑀𝐷 1 , 𝑆 𝑀𝐷 2 be the signatures of two distinct multimedia documents. The cross-matching score between their components (W and MO) is computed as follows:𝑚𝑎𝑡𝑐ℎ(𝑆 𝑀𝐷 1 , 𝑆 𝑀𝐷 2 ) = 𝜆 𝑚 × 𝑓(𝑊 1 , 𝑀𝑂 2 ) + (1 -𝜆 𝑚 ) × 𝑓(𝑊 2 , 𝑀𝑂 1 ) → [0, 1] are association functions to determine the association of a set of words contained in 𝑆 𝑀𝐷 1 (𝑆 𝑀𝐷 2 respectively) with the set of multimedia objects contained in 𝑆 𝑀𝐷 2 (𝑆 𝑀𝐷 1 respectively). f (W1,MO2) is defined as:

	where:
	 f (W1,MO2) and f (W2,MO1)
	 sk is a unit similarity function comparing multimedia objects 𝑚𝑜 𝑖 1 ∈ MO1 and 𝑚𝑜 𝑗 2 ∈
	MO2. We note that sk(𝑚𝑜 𝑖 1 , 𝑚𝑜 𝑗 2 ) compares 𝑚𝑜 𝑖 1 and 𝑚𝑜 𝑗 2 based on their attributes

and raw data. sk returns a score between [0, 1], where 0 expresses a total divergence and 1 a complete similarity.

 ϖ is an aggregation function used to aggregate the computed similarity scores. We give below an example to illustrate the estimated similarity between two sets of multimedia objects. Therefore, we propose a unit similarity function

82

that takes coordinates of two images and returns 1 if the input coordinates belong to the same geographical location. We see in our example that one of the images published on the Twitter account of François Fillon was captured in the same geographical coordinates (Latitude : 48.357483, Longitude : -1.116662) as the image of "Château de Beaucé" found on the Wikipedia page of François Fillon.

  Definition 9  Selective Intersection: Let 𝑆 𝑀𝐷 1 , 𝑆 𝑀𝐷 2 be the signatures of two distinct multimedia documents. Their selective intersection is defined as: 𝑆 𝑒𝑙 𝐼 𝑛𝑡 (𝑆 𝑀𝐷 1 , 𝑆 𝑀𝐷 2 ) = ‖∑ 𝑤𝑎 𝑖 × 𝑒𝑞𝑢 𝑎 𝑖 (𝑊 1 , 𝑊 2 ) 𝑎 𝑖 + ∑ 𝑤𝑎 𝑗 × 𝑠𝑖𝑚 𝑎 𝑗 (𝑀𝑂 1 , 𝑀𝑂 2 ) + ∑ 𝑤𝑎 𝑘 × 𝑚𝑎𝑡𝑐ℎ 𝑎 𝑘 (𝑆 𝑀𝐷 1 , 𝑆 𝑀𝐷 2 )

		‖
	𝑎 𝑗	𝑎 𝑘
	where:

1 Achieving de-linkability

  MDβ that can be α-associated with MDu. Unsurprisingly, multimedia objects need a special interest. Eventually, the objective is to break linkable objects that could contribute in re-identifying the anonymized individual. More subtle is to hide and/or disseminate multimedia objects content while at the same time preserving a minimum semantic or visual coherence. In this study, we do not provide an in-depth details on how multimedia objects content could be protected. This matter is left for future work. We only use traditional

	de-linkability can be achieved in textual documents in a straightforward way using extension of traditional anonymization techniques such as suppression, substitution or generalization relationships between domains and values [29, 36, 37] for textual values in MDu as long as data through flexible low-level adapted security rules, while in [130] object substitution is adopted. In [131], blurring proved efficiency, and objects removal from images and videos were addressed in [132-137]. Here, we refer to this process as document sanitization which there is no techniques to protect salient objects as in [129] where the authors protect textual and image we formally define as follows:

MDu, the de-linkability privacy-preserving constraint is satisfied if ∀ MDβ ∈ 𝜎 𝐸 𝑢 (E) that is β-associated with pfu, MDu cannot be linked to MDβ through an α-association, where 𝜎 𝐸 𝑢 (E) is a selection on an external source ɛ based on a conjunctive set of words and/or multimedia objects (Eu) related to u. de-linkability breaks the link between a multimedia document (to be published) and any other document accessible to a cyberstalker and that can be linked to u. It is important to note that the content of Eu that is used to retrieve multimedia documents MDβ from the external source should be considered carefully in order to reduce the scope of potential error. A straightforward assumption is to consider this content as a subset of the individual's profile including both identifying and quasi-identifying values.

IV.

Definition 12  Multimedia Document Sanitization: Let MDu be the multimedia document related to a cyberstalkee u. Given 𝐺 ̃𝑊 and 𝐺 ̃𝑀𝑂 , two corresponding sanitizing functions, we say that MDu is sanitized, denoted by 𝑀𝐷 𝑢 * = 𝐺 ̃(𝑊,𝑀𝑂) (𝑀𝐷 𝑢 ) if both words and multimedia objects are sanitized 𝐺 ̃𝑊(𝑊 𝑀𝐷 𝑢 ) and 𝐺 ̃𝑀𝑂 (𝑀𝑂 𝑀𝐷 𝑢 ).

Multimedia document sanitization ensures that the specified content (W,MO) is either removed, suppressed, generalized and/or protected in the multimedia document MDu based on the sanitization function.

  j, dist∆T (photoi.meta∆.t, photoj.meta∆.t) ≤ γ and dist∆L (photoi.meta∆.l, photoj.meta∆.l) ≤ δ and

dist∆S (photoi.meta∆.s, photoj.meta∆.s) = ε

[START_REF] Clifton | Tools for privacy preserving distributed data mining[END_REF]

Table 8 :

 8 Availability of metadata in each dataset.

	Groups	Datasets	Capture time	Geographic	Uploader
			(T) %	information	information (S)
				(G) %	%
	Setfull	SetFull T, G, S=100%	100	100	100
	SetFull (T)	SetFull T=75%	75	100	100
		SetFull T=50%	50	100	100
		SetFull T=25%	25	100	100
	SetFull (G)	SetFull G=75%	100	75	100
		SetFull G=50%	100	50	100
		SetFull G=25%	100	25	100
	SetFull (S)	SetFull S=75%	100	100	75
		SetFull S=50%	100	100	50
		SetFull S=25%	100	100	25
	SetFull (T, G, S)	SetFull T, G, S=75%	75	75	75
		SetFull T, G, S=50%	50	50	50

Table 9 :

 9 F-value and NMI scores considering different granularities of time and space on the Setfull dataset.

	Test #	Time granularity	Space	F-measure	NMI
			granularity
	1	day	street	0.8129	0.9519
	2	day	city	0.8101	0.9514
	3	day	country	0.8125	0.9525
	4	month	street	0.7923	0.9498
	5	month	city	0.7818	0.9482
	6	month	country	0.7532	0.943
	7	year	street	0.7923	0.9498
	8	year	city	0.7485	0.9426
	9	year	country	0.6112	0.9136

  S1 S and ''S2 S: represent the social neighborhood of e1 and e2 respectively. ''S1 and ''S2 contains the set of people that are not participating to e1 and e2 respectively but belonging to the same groups of those who are participating and having a social tie strength with u0 larger than a predefined threshold 𝜀 ∈ [0, 1].

	IV.3 Social Relations
	IV.3.1 Social Intersection Sets
	Let L1 and L2 denote the respective social features of e1 and e2. The definitions of their social
	intersection sets are given below:
	 'S1 S and 'S2 S: represent the set of people participating in e1 and e2
	respectively (i.e., e1.meta.S and e2.meta.S)
	May '' April,June  ''Formally, the social boundary is represented as: May April,June
	|∆ 𝑆 |-1
	∆ 𝑆 " = ⋃ 𝑢 𝑖	∈ ∆ 𝑆 -𝑒. 𝑚𝑒𝑡𝑎. 𝑆/ (𝑓 1 (𝑢 𝑖 ) ∈	⋃	𝑓 𝑙 (𝑣)
	𝑖=0

∀𝑣∈𝑒.𝑚𝑒𝑡𝑎.𝑆

) ˄(𝐷 𝑡𝑖𝑒 (𝑢 0 , 𝑢 𝑖 ) ≤ 𝜀)

  Lisa had two events e1 and e2 both with two of her colleagues Rich and Gil. Consider two events e1 and e2 which are detected in the social network of Lisa,

	ISsoc with u0. Otherwise, boundary sets are not empty. To illustrate this, we consider the following e1 e2 Social relation
	' two cases: Minale, (Solomon)	Naty Rich, Gil, Kouki,	0010	weak-Meet-ALL
	Rich, Gil, Kouki, Naty e1 Rich, Gil, Kouki, Naty Case 1: Case 2: '' ISsoc ' ''	e2 Minale, (Solomon) Rich, Gil, Kouki, Naty	0100	strong Social relation ALL strong-Meet-weak
	ISsoc	e1	e2		Social relation
	'	Rich, Solomon	Gil, Minale	0111	Meet
	''	Gil, Kouki, Nat	Rich, Kouki, Naty

  Consider two events e1 and e2 which are detected in the social network of Lisa,

	ISsoc Case 2:	e1	e2		Social relation
	'	Rich, Gil		Rich, Gil	1001	Equal
	''	Kouki, Naty	Kouki, Naty
	ISsoc		e1	e2		Social relation
	'	Rich, Gil, Kouki, Naty	Rich, Gil, Kouki, Naty	1000	Equal ALL
	''

  Consider two events e1 and e2 which are detected in the social network of Lisa,

	ISsoc Case 2:	e1	e2		Social relation
	'	Kouki, Naty	Gil, Kouki, Naty	1011	Inside
	''	Rich, Gil		Rich
	ISsoc		e1	e2		Social relation
	'	Kouki, Naty	Rich, Gil, Kouki, Naty	1010	Inside ALL
	''	Rich, Gil
	ISsoc		e1	e2		Social relation
	'	Gil, Kouki, Naty	Kouki, Naty	1101	Contains
	''	Rich		Rich, Gil

  Lisa had two events with three of her colleagues: e1 with Rich, Gil and Naty; and e2 with Rich, Gil and Kouki.

	Case 2:
	ISsoc	e1	e2		Social relation
	'	Rich, Gil, Kouki, Naty	Kouki, Naty	1100	Contains ALL
	''		Rich, Gil
	ISsoc	e1	e2		Social relation
	'	Kouki, Naty	Gil, Naty	1111	Intersect
	''	Rich, Gil	Rich, Kouki

Table 12 :

 12 The binary and hexadecimal values of the three basic relations (spatial, temporal and social)

	Hexadecimal	Binary	Spatial	Temporal	Social
	values	values	relation	relation	relation
	0	0000	disjoint	disjoint	disjoint
	1	0001	meet	meet	weak-Meet-weak
	2	0010			weak-Meet-ALL strong
	3	0011			weak-Meet-strong
	4	0100			ALL strong-Meet-weak
	5	0101			strong-Meet-weak
	6	0110			strong-Meet-strong
	7	0111			meet
	8	1000			equal ALL
	9	1001	equal	equal	equal
	A	1010	inside	inside	inside ALL
	B	1011			inside
	C	1100	contains	contains	contains ALL
	D	1101			contains
	E	1110		intersects	intersects ALL
	F	1111	intersects		intersects

We can see from Table

12

, and as we mentioned before, each spatial or temporal relation has one binary value, and thereby one hexadecimal value. However, for social relations, we can define ranges of values as follows: disjoint: 0; meet: 1-7; equal:8-9; inside: A-B; contains: C-D; intersects: E-F.

https://en.wikipedia.org/wiki/Arab_Spring

http://vincentdubois.fr/undefined.php

http://www.fastcompany.com/3049569/has-this-man-unlocked-the-secret-to-internet-anonymity

di.me = "digital.me: Userware for the Intelligent, Intuitive, and Trust-Enhancing Management of the Users Personal Information Sphere in Digital and Social Environments" http://www.dime-project.eu/

http://claimid.com/

http://www.findmeon.com/relaunch/

http://freeyourid.com/

https://www.myopenid.com/

http://en.wikipedia.org/wiki/Mean_squared_error

For example, compareGPS((lat1, long1),(lat2, long2))= 1 if (ConvertFromLatLong(lat1, long1) = ConvertFromLatLong(lat2, long2)); 0 otherwise

Remerciements

précieuse pour moi. pour ses encouragements.

notification. The media upload supports the sharing of all types of media files including videos, photos as well as text documents. The event creation follows the standard facets of who, where, when and what. In Eventory, relationships can exist between events, between media, and between media and events. Since in this work [109], the authors refer to an event as a real-world occurrence that unfolds over space and time, the authors focus on temporal (e.g., daily, weekly, monthly and yearly) and spatial relations (e.g., beside). Beyond that, they allow the user to define his own semantic relationships arbitrarily. Thus, the semantics will be meaningful to the user. Constraints are also imposed by the system on the semantics of the relationship's specification.

In [START_REF] Sarma | Dynamic relationship and event discovery[END_REF], the authors distinguished between traditional and dynamic relationships extracted from unstructured or semi-structured data sources. Traditional relationships are static predefined relationships (e.g., spousal relationship) that can be extracted based on language patterns. In contrast, dynamic relationships are not predefined by an existing schema. They are temporally defined (e.g., co-bursting) and are often a product of underlying real world events (e.g., relationships associated with news events). In this work [START_REF] Sarma | Dynamic relationship and event discovery[END_REF], the authors focus on dynamic relationships that are formed due to real-world events. However, the difference with our method is that they generate relationships before deriving events.

Figure 22 shows an architecture diagram of their approach.

Consequently, we can easily move from the first space  to the second space ∆ R in a homogeneous method (see Figure 50). The main interest of this representation is to express clearly exclusive 93 relations that exist between events using their metadata, and thus to move from image raw data to Linked Event Data.

The advantages of our model are as follows:

(A1) A simple and not expensive hexadecimal representation;

(A2) An easy-to-read and expressive fashion;

(A3) An extensible model for more complex events relations.

III. Meta-model of Event Relations

We present a meta-model to derive relations between events. Our meta-model is based on the 4-intersection formalism for topological relations to represent temporal, spatial and social relations between events. In the following, we emphasize on the importance of having a meta-The 4-intersections are then computed between 'S1, 'S2, ''S1 and ''S2.

IV.3.3 Topological Social Relations with Examples

Now, we give short examples illustrating each social relation. We consider a set of events shared on the social network of Lisa. She and her colleagues participate in the same events together all the time and share photos of those moments online. However, each event includes different people.

We show the possible social relations that can link any two events (based on topological relations). Then, we compute the intersection matrix that represents each relation. These relations are valid when the event contains people from many groups (e.g., a birthday party with friends and family). For simplicity, we consider here events with individuals from only a single social group.

IV.3.3.1 Socially Disjoint relation

We say that two events e1 and e2 are "socially disjoint" if the people who participated in e1 are totally different from those who participated in e2, and do not belong to the same group in the social network of u0.

To illustrate this relation, we consider two events e1 and e2 which are detected in the social network of Lisa: in one event, Lisa was with three of her colleagues (Rich, Gil and Naty), and in the other, she was with one of her family members (Bouli).

We recall that ' and '' denote the social event interior and exterior respectively. The "socially disjoint" relation has the following binary representation: 0000.

ISsoc e1 e2

Social relation

'

Rich, Gil, Naty Bouli 0000 Disjoint

''

Kouki Elie

I. Contributions

The study presented in this thesis has mainly been concerned with preserving privacy of anonymous users when publishing multimedia documents online, specifically photos as we explained in Chapter 1. We investigated metadata's photos to acquire more information about the users to protect their anonymity. Particularly in this thesis, we are interested in personal information that can be inferred from events shared through photos on online social networks. The contributions of this work are summarized in the following:

Chapter 2 contained the background review, in which we focused on privacy preservation and inference control. We also discussed various definitions of online identity and how it can be constructed using the social networking sites. Then, we investigated the techniques used for online event detection and linking using multimedia content available on social networks.

In Chapter 3, we focused on the anonymization of multimedia documents published online that can put at risk the privacy of users and violate their anonymity. In our motivating scenario described in Chapter 1, we showed the need to preserve the identity of anonymous users (like in the case of the previous French Prime Minister François Fillon). To address this, we presented a privacy-preserving constraint, called de-linkability, to prevent the leakage of sensitive information (anonymity, pseudonymity, etc.) caused by textual and multimedia content that users produce online. With this constraint, users can be warned of potential violation of their privacy. Then, we provided a sanitizing MD * -algorithm to enforce delinkability along with a utility function to evaluate the utility of multimedia documents that is preserved after the sanitizing process. Thus, the identity of the user who wishes to remain anonymous is kept private. We developed a prototype to evaluate the identity anonymization problem and the proposed sanitizing algorithm. We experimentally demonstrated how the delinkability could break the link between multimedia documents to be published (i.e., tweets)

and any other document accessible to a potential adversary and that can be linked to the user.