Mia Johnson Vioulès

Bilel Moulahi

Jérôme Azé

Sandra Bringay

Detection of Suicide-Related Posts in Twitter Data Streams

Suicidal ideation detection in online social networks is an emerging research area with major challenges. Recent research has shown that the publicly available information spread across social media platforms holds valuable indicators to effectively detecting individuals with suicidal intentions. The key challenge of suicide prevention is understanding and detecting the complex risk factors and warning signs that may precipitate the event. In this paper, we present a new approach that uses the social media platform Twitter to quantify suicide-warning signs for individuals and to detect posts containing suicide-related content. The main originality of this approach is the automatic identification of sudden changes in a user's online behavior. To detect such changes, we combine natural language processing techniques to aggregate behavioral and textual features and pass these features through a martingale framework, which is widely used for change detection in data streams. Experiments show that our text-scoring approach effectively captures warning signs in text compared to traditional machine learning classifiers. Additionally, the application of the martingale framework highlights changes in online behavior and shows promise for detecting behavioral changes in at-risk individuals.

Introduction

According to the World Health Organization (WHO), it is estimated that 800,000 people die by suicide each year worldwide with at least as many suicide attempts [1]. The grief felt in the aftermath of such an event is compounded by the fact that a suicide may be prevented. This reality of suicide has motivated WHO member states to commit themselves to reducing the rate of suicide by 10% by 2020 [START_REF]Preventing suicide: A global imperative[END_REF].

In an effort to educate the public, the American Foundation for Suicide Prevention (AFSP) [START_REF]Mental health action plan 2013 -2020[END_REF] has identified characteristics or conditions that may increase an individual's risk. The three major risk factors are: 1) health factors (e.g. mental health, chronic pain), 2) environmental factors (e.g. harassment, stressful life events), and 3) historical factors (e.g. previous suicide attempts, family history). Additionally, the time period preceding a suicide can hold clues to an individual's struggle. The AFSP categorizes these warning signs as follows: 1) talk (e.g. mentioning being a burden or having no reason to live), 2) behavior (e.g. withdrawing from activities, sleeping too much or too little), and 3) mood (e.g. depression, rage).

Identifying these risk factors is the first step in suicide prevention. However, the social stigma surrounding mental illnesses means that at-risk individuals may avoid professional assistance [START_REF] Corrigan | How stigma interferes with mental health care[END_REF]. In fact, they may be more willing to turn to less formal resources for support [START_REF] Rickwood | Young people's help seeking for mental health problems[END_REF]. Recently, online social media networks have become one such informal resource. Research has shown that at-risk individuals are turning to contemporary technologies (forums, micro-blogs) to express their deepest struggles without having to face someone directly [START_REF] Choudhury | Predicting depression via social media[END_REF][START_REF] Moreno | Feeling bad on Facebook: depression disclosures by college students on a social networking site[END_REF]. As a result, suicide risk factors and warning signs have been seen in a new arena. There are even instances of suicide victims writing their final thoughts on Twitter, Facebook, and other online communities [START_REF] Gunn | Twitter postings and suicide: An analysis of the postings of a fatal suicide in the 24 hours prior to death[END_REF][START_REF] Kailasam | Can social media help mental health practitioners prevent suicides?[END_REF]. We believe that this large amount of data on people's feelings and behaviors can be used successfully for early detection of behavioral changes in at-risk individuals and may even help prevent deaths. Social computing research has focused on this topic in recent years [START_REF] Choudhury | Predicting depression via social media[END_REF][START_REF] Kailasam | Can social media help mental health practitioners prevent suicides?[END_REF][START_REF] Choudhury | Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media[END_REF]. However, few initiatives have been concerned with the real time detection of suicidal ideation on Twitter. Previously proposed detection methods rely heavily on manually annotated speech, which can limit their effectiveness due in part to the varying forms of suicide warning signs in at-risk individuals [START_REF] Choudhury | Predicting depression via social media[END_REF][START_REF] Choudhury | Predicting postpartum changes in emotion and behavior via social media[END_REF][START_REF] Lehrman | Detecting distressed and non distressed affect states in short forum texts[END_REF]. Many of these methods also focus on the messages published by individuals at a specific time independently of the whole context, which may be represented by the sequence of publications over time.

In this article, we address the challenge of real-time analysis of Twitter posts and the detection of suicide-related behavior. To process the stream of an individual's online content we implement a martingale framework, which is widely used for the detection of changes in data stream settings. The input into this framework is a series of behavioral features computed from each individual Twitter post (tweet). These features are compared to previously seen behavior in order to detect a sudden change in emotion that may indicate an elevated risk of suicide.

The main contributions of this article are twofold. First, using research from the field of psychology, we design and develop behavioral features to quantify the level of risk for an individual according to his online behavior on Twitter (speech, diurnal activities, size of social network, etc.). In particular, we create a feature for text analysis called the Suicide Prevention Assistant (SPA) text score. Secondly, we monitor the stream of an individual Twitter user and his behavioral features using an innovative application of a martingale framework to detect sudden behavioral changes.

Literature review

The definition and identification of risk factors and warning signs lie at the core of suicide prevention efforts. In this paper, we have chosen to reference the risk factors defined by the American Pyschiatric Association (APA) [START_REF] Psychiatric | Practice guideline for the assessment and treatment of patients with suicidal behaviors[END_REF] and the warning signs identified by the American Association of Suicidology (AAS) [START_REF] Rudd | Warning signs for suicide: Theory, research, and clinical applications[END_REF]. These resources represent a level of consensus between mental health professionals and also provide a rich discussion of the differences between suicide risk factors and warning signs. For further reading, we direct the reader to the work of [START_REF] Rudd | Warning signs for suicide: Theory, research, and clinical applications[END_REF].

As highlighted by [START_REF] Rudd | Warning signs for suicide: Theory, research, and clinical applications[END_REF], warning signs signify increased imminent risk for suicide (i.e., within minutes, hours, or days). According to the APA suicide warning signs may include talking about dying, significant recent loss (death, divorce, separation, broken relationship), change in personality, fear of losing control, suicide plan, suicidal thoughts, or no hope for the future. As discussed in the following, recent research has shown the emergence of such signs on social networking sites.

Most of the research at the intersection of behavioral health disorders and social media has focused on depression detection in online communities, specifically Major Depressive Episodes (MDE). However, the risk factors for suicide defined by the APA [START_REF] Psychiatric | Practice guideline for the assessment and treatment of patients with suicidal behaviors[END_REF] go far beyond depression alone. It is important to remember that depression does not necessarily imply suicidal ideation. Rather, suicide should be thought of as a potential end symptom of depression.

While mental health issues such as depression, suicidal ideation, and self-mutilation are defined medically as separate illnesses with overlapping symptoms, the approaches proposed to detect them online can be quite similar. Where the approaches vary is in the data they are treating, i.e. Facebook posts, Twitter tweets, Reddit forum threads, etc. and the specific event they are attempting to predict. In [START_REF] Moreno | Feeling bad on Facebook: depression disclosures by college students on a social networking site[END_REF], Moreno et al. first demonstrated that social networking sites could be a potential avenue for identifying students suffering from depression. The prevalence rates found for depression disclosed on Facebook corresponded to previous works in which such information was self-reported. On a larger scale, Jashinsky et al. [START_REF] Jashinsky | Tracking suicide risk factors through Twitter in the US[END_REF] showed correlation between Twitter-derived and actual United States per-state suicide data. Together, these works established the presence of depression disclosure in online communities and opened up a new avenue for mental health research.

De Choudhury et al. [START_REF] Choudhury | Predicting depression via social media[END_REF] explored the potential to use social media to detect and predict major depressive episodes in Twitter users. Using crowd-sourcing techniques, the authors built a cohort of Twitter users scoring high for depression on the CES-D (Center for Epidemiologic Studies Depression Scale) scale and other users scoring low. Studying these two classes, they found that what is known from traditional literature on depressive behavior also translates to social media. For example, users with a high CES-D score posted more frequently late at night, interacted less with their online friends, and had a higher use of first-person pronouns. Additionally, online linguistic patterns match previous findings regarding language use of depressed individuals [START_REF] Rude | Language use of depressed and depressionvulnerable college students[END_REF]. More recently, De Choudhury et al. [START_REF] Choudhury | Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media[END_REF] have shown that linguistic features are important predictors in identifying individuals transitioning from mental discourse on social media to suicidal ideation. The authors showed a number of markers characterizing these shifts including social engagement, manifestation of hopelessness, anxiety and impulsiveness based on a small subset of Reddit posts.

Coppersmith et al. [START_REF] Coppersmith | Exploratory Analysis of Social Media Prior to a Suicide Attempt[END_REF] examined the data published by Twitter users prior to a suicide attempt and provided an empirical analysis of the language and emotions expressed around their attempt. One of the interesting results found in this study is the increase in the percentage of tweets expressing sadness in the weeks prior to a suicide attempt, which is then followed by a noticeable increase in anger and sadness emotions the week following a suicide attempt. In the same line of research, O'Dea et al. [START_REF] O'dea | Detecting Suicidality on Twitter[END_REF] confirmed that Twitter is used by individuals to express suicidality and demonstrated that it is possible to distinguish the level of concern among suiciderelated tweets, using both human coders and an automatic machine classifier. These insights have also been investigated by Braithwaite et al. [START_REF] Braithwaite | Validating Machine Learning Algorithms for Twitter Data Against Established Measures of Suicidality[END_REF] who demonstrated that machine learning algorithms are efficient in differentiating people who are at a suicidal risk from those who are not. For a more detailed review of the use of social media platforms as a tool for suicide prevention, the reader may refer to the recent systematic survey by Robinson et al. [START_REF] Robinson | Social Media and Suicide Prevention: A Systematic Review[END_REF].

These works have shown that individuals disclose their depression and other struggles to online communities, which indicates that social media networks can be used as a new arena for studying mental health. Despite the solid foundation, the current literature is missing potential key factors in the effort to detect depression and predict suicide. Currently, few works analyze the evolution of an individual's online behavior. Rather, the analysis is static and may take into consideration one post or tweet at a time while ignoring the whole context. Additionally, an individual's online speech is often compared to other individuals and not to their own linguistic style. This is a disadvantage because two individuals suffering the same severity of depression may express themselves very differently online.

A general framework for detecting suicide-related posts in social networks

In this section, we present the proposed framework for the analysis and real-time detection of suicide-related posts on Twitter. First, we introduce the real-time detection problem. Then we define our online proxy measurements (behavior features) for suicide warning signs. Finally, we describe the approach we implement for detecting behavioral change points.

Problem statement

Sudden behavioral change is one of the most important suicide warning signs. As reported by the AFSP, a person's suicide risk is greater if a behavior is new or has increased, especially if it is related to a painful event, loss, or change. Considering this in conjunction with social media, where users constantly publish messages and deliberately express their feelings, we address suicide warning sign detection as a real-time data stream mining problem. Given a series of observations over time (tweets, messages, blog posts), the task is to detect an abrupt change in a user behavior that may be considered as a suicide warning sign. In the field of data stream mining, this can be specifically seen as change point detection problem [START_REF] Adams | Bayesian online change point detection[END_REF][START_REF] Garnett | Sequential Bayesian prediction in the presence of change points and faults[END_REF]. However, unlike retrospective detection settings [START_REF] Basseville | Detection of Abrupt Changes: Theory and Application[END_REF][START_REF] Liu | Change-point detection in time-series data by relative density-ratio estimation[END_REF], which focus on batch processing, here we are interested in the setting where the data arrives as a stream in real-time.

To tackle this challenge, we chose an approach employing a martingale framework for change point detection [START_REF] Vovk | Testing exchangeability on-line[END_REF]. This algorithm has been successfully applied to detecting changes in unlabeled data streams, video-shot change detection [START_REF] Ho | A martingale framework for detecting changes in data streams by testing exchangeability[END_REF] and more recently, in the detection of news events in social networks [START_REF] Babakhani | A martingale approach to detect peak of news in social network[END_REF]. To the best of our knowledge, this is the first attempt to apply the martingale framework on a multi-dimensional data stream generated by Twitter users.

In the following, we start by introducing and describing the proxy measurements for suicide warning signs that we use to assess the patient's level of suicide risk. As previously mentioned, these warning signs will be the input into the martingale framework.

Suicide warning signs in online behavior

To identify online behaviors that may reflect the mental state of a Twitter user, we established two groups of behavioral features: user-centric and post-centric features [START_REF] Choudhury | Predicting postpartum changes in emotion and behavior via social media[END_REF][START_REF] Choudhury | Social media as a measurement tool of depression in populations[END_REF]. User-centric features characterize the behavior of the user in the Twitter community while post-centric features are characteristics that are extracted from the properties of a tweet. These features have been shown to successfully aid in determining the mental health of a user [START_REF] Choudhury | Predicting depression via social media[END_REF]. Table 1 shows a detailed description of the features we selected.

The American Association of Suicidology (AAS) identifies withdrawing from friends, family, or society as one of the warning signs of suicide. With the user-centric behavioral features, we aim to capture changes in a Twitter user's engagement with other users. The friends and followers features can quantify an individual's interaction with their online community, such as a sudden decrease in communication. Contrarily, they can also reflect an expansion of an individual's online community. This is relevant, as at-risk individuals have also been shown to increase their time online developing personal relationships [START_REF] Harris | Suicidal and online: How do online behaviors inform us of this high-risk population?[END_REF]. It is important to note that we have chosen the terms friends and followers to represent the unidirectional relationships that are inherent on Twitter. We acknowledge that this term may not apply for certain user accounts such as celebrities and news outlets.

Additional features include volume, replies, retweets, and links, which were all identified by [START_REF] Choudhury | Predicting depression via social media[END_REF] as markers for mental health. These measures can help to quantify the number of interactions a user has with their friends and followers for it could be the case that an individual's social network remains stable while their interactions increase or decrease. The final user-centric feature, questions, may also indicate a user's attempt to engage with others online.

Post-centric behavioral features are characteristics originating from the post itself. One important piece of information is the hour at which the tweet is published (time feature). Late night activity can be an indication of unusual rhythms in sleep (insomnia and hypersomnia) [START_REF] Choudhury | Predicting depression via social media[END_REF] and can predict future episodes of depression. In addition to the time feature, we address the text of the post (text score), which holds the most vital information pertaining to an individual's current mood and mental health [START_REF] Coppersmith | Quantifying mental health signals in Twitter[END_REF].

To classify the text of the post, we propose two different approaches. The first approach is a natural language processing (NLP) method that combines features generated from the text based on an ensemble of lexicons. These lexicons are composed of linguistic themes commonly exhibited by at-risk individuals. The second approach, called the distress classifier, is based on machine learning. Although machine learning is commonly used to classify text, the supervised algorithms require annotated datasets, which may be costly in terms of time and potential annotator error. Additionally, traditional machine learning methods are difficult to apply in this context because of the nature of depression and distress in general. Two individuals suffering from depression may not express their symptoms in the same way, which translates to texts of the same severity having vastly different content. This means it is difficult for the algorithm to find the concept mapping between the textual features and the level of depression/distress.

Feature Extraction for Text Scoring: A Natural Language Processing based approach

To extract and compute features using NLP techniques, we start by creating a new symptom lexicon to identify the most discriminating terms commonly used by distressed or depressed individuals. Instead of manually translating questionnaires and generating synonyms [START_REF] Karmen | Screening internet forum participants for depression symptoms by assembling and enhancing multiple NLP methods[END_REF], we decided to create the lexicon directly from a collection of tweets. For this purpose, we implemented the pointwise mutual information (PMI) [START_REF] Krestel | Generating contextualized sentiment lexica based on latent topics and user ratings[END_REF] measure, that highlights the dependence between two random variables. This measure is formally defined as:

𝑃𝑀𝐼(𝑤, 𝑐) = log( 𝑃(𝑤, 𝑐) 𝑃(𝑤)𝑃(𝑐) )
where, w is a word, c is a class, and P(w,c) is the probability that word w occurs in class c. P(w) is the frequency of the word across all classes and P(c) is the frequency of class c.

Our final PMI symptom lexicon is built using an annotated dataset of tweets belonging to two classes of users, distressed and everyday users. This dataset is discussed in more detail in the evaluation section. In the end, each word in the lexicon is assigned two scores, one reflecting the word's dependence on the class distressed and the other on the class everyday. While a positive PMI score indicates that the word is a discriminating term for the distressed class, a negative score indicates that it is a discriminating term for the everyday class.

Note that we moved from the subject of depression (medical term) to the subject of distress, which is a more encompassing term. In fact, depression is characterized by one or more major depressive episodes, which translates to at least two weeks or more of depressive mood or loss of interest accompanied by at least four other symptoms of depression [START_REF]American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders (DSM-5[END_REF]. We hypothesize that the distressed class includes users that discuss suicidal ideation, depression, and self-harm.

To enrich the lexicon, and go beyond depression-only symptoms, we propose three other features: swear words, intensifier terms (very, extremely), and first-person pronouns (me, myself, I). These features have been shown to carry important information in the context of sentiment analysis [START_REF] Choudhury | Social media as a measurement tool of depression in populations[END_REF] and were similarly used by de Choudhury et al. [START_REF] Choudhury | Predicting depression via social media[END_REF].

To compute the final score reflecting the level of distress in a tweet we aggregate the four features to get our Suicide Prevention Assistant (SPA) text score. The score is calculated using the following linear combination:

𝑆𝑃𝐴 = 𝑓 symptoms + 𝑓 swear + 𝑓 intensifiers + 𝑓 first_pronouns
Where f symptoms represents the sum of PMI scores of every word in a tweet that appears in the symptom lexicon and the f swear, f intensifiers and f first_pronouns components are the frequency of symptoms, intensifying adjectives and the first-person pronouns, respectively, in a tweet. Finally, the SPA text score is normalized for each tweet by dividing by the number of total words. This helped us control for longer tweets that might have an inflated SPA text score because of more symptom words.

Feature Extraction for Text Scoring: Classification with Machine Learning

In general, the challenge of categorizing tweets is traditionally tackled using text classifiers that rely on machine learning. Therefore, we considered it important to also test a classifying algorithm as a benchmark against our NLP approach detailed above. We took inspiration from [START_REF] Lehrman | Detecting distressed and non distressed affect states in short forum texts[END_REF][START_REF] Benamara | Sentiment analysis: Adjectives and adverbs are better than adjectives alone[END_REF] and chose to categorize tweets according to different levels of distress. Again, although distress is not equivalent to suicide ideation and major depressive episodes, it is an important risk factor in suicide and one that is highly observable from micro-blog text [START_REF] Homan | Toward macro-insights for suicide prevention: Analyzing fine-grained distress at scale[END_REF].

In addition to the four features used in the previous section (fsymptoms, fswear, fintensifiers and ffirst_pronouns), we split the text into n-grams, which are commonly used in text classification tasks and is popular as a base feature for sentiment analysis of tweets. The character limitation (140 maximum) of tweets lends itself to a choice of shorter n-grams, particularly uni-grams and bigrams [START_REF] Gil | Combining machine learning techniques and natural language processing to infer emotions using Spanish Twitter corpus[END_REF][START_REF] Pak | Twitter as a corpus for sentiment analysis and opinion mining[END_REF].

Design of a Martingale-based approach for emotion change detection

In the previous sections, we described the user-centric and post-centric behavioral features we extract from a user's online content. In this section, we present the approach we use to process these behavioral features and detect sudden emotional changes. If we consider the features as a series of multi-dimensional observations over time, the challenge of detecting an abrupt change in behavior resembles the classic problem of change point detection often seen in the field of data stream mining.

We implemented a martingale framework [START_REF] Vovk | Testing exchangeability on-line[END_REF], which is an online real-time, non-parametric change point detection model. Let X={x1, x2, x3… xn} be a sequence of unlabeled m-dimensional data points with new data points, xi, arriving in a sequence. The m dimensions correspond to the values of the user-centric and post-centric behavioral features identified in the previous section.

When a new tweet is published, it is first characterized by this set of features. With this information, the tweet is then run through a hypothesis test to determine if its features represent a prominent change in the data stream. More formally, the test is as follows: H0, there is no change in the data stream (i.e, no marked emotional change), and H1 otherwise [START_REF] Mozafari | A precise statistical approach for concept change detection in unlabeled data streams[END_REF].

The full martingale framework can be broken down into three steps. The first step is to calculate the strangeness measure, which quantifies for each specific user how much a tweet is different from previous ones. Next, a statistic is defined to rank the strangeness measures of the tweets. Finally, using this statistic, a family of martingales is defined in order to detect movements in the tweet stream and run the hypothesis test.

Unified Strangeness Measure (USM)

Given that the behavioral features we have chosen are represented as numerical attributes, we chose the Euclidean distance to measure the distance of each data point from the mean of the other data points. For a stream of unlabeled tweets, X={x1, x2, x3… xn-1}, where xn is the most recent tweet in the data stream, the unified strangeness measure (USM) is defined as:

𝑈𝑆𝑀 𝑖 (𝑋, 𝑥 𝑛 ) = ∑|𝑥 𝑖𝑘 -𝜇({𝑥 1𝑘 , … , 𝑥 (𝑛-1)𝑘 } ∪ {𝑥 𝑛𝑘 })| 𝑚 𝑘=1
Where i: 1…n, m is the number of (user-centric and post-centric) behavioral features, xik is the value of the k-th feature for tweet xi and µ is the mean of the previous values and the new observed value with respect the feature k. The USM for a tweet increases as it 'moves away' from the representative of the entire stream of data. For continuous attributes, the representative value is the mean, while for categorical attributes; we use the mode as the representative.

In the second step of the martingale framework, we rank the USM of the new point with respect to the USM of the previously observed points using a statistic. This statistic is denoted as the 𝑝value for each instance xi. Formally, the 𝑝-value of xi for i: 1…n can be calculated as follow:

𝑝î({𝑥 1 , … , 𝑥 𝑛 }, 𝜃 𝑖 ) = #{𝑗: 𝑠 𝑗 > 𝑠 𝑖 } + 𝜃 𝑖 #{𝑗: 𝑠 𝑗 = 𝑠 𝑖 } 𝑖
Where sj is the USM for xj, θi is a random value in [0,1] at instance i and #{} is the number of elements satisfying the bracketed condition [START_REF] Vovk | Testing exchangeability on-line[END_REF]. The random number θi ensures that the p-values are distributed uniformly in [0,1] provided that the input observations are generated by an exchangeable probability distribution in the input space.

When a change in the data stream occurs, this translates to high USM values for the observations following the change point. In turn, the computed 𝑝-value become smaller, which means that the incoming data points are moving away from the representatives of the previously seen observations.

To decide whether there is an abrupt change in the user behavior or not (i.e., reject the null hypothesis H0), a family of martingales is defined based on the derived 𝑝-values. We use the randomized power martingale [START_REF] Babakhani | A martingale approach to detect peak of news in social network[END_REF], defined as:

𝑀 𝑛 (𝜀) = ∏(𝜀𝑝̂𝜀 -1 ) 𝑛 𝑖=1
Where the martingale value Mi measures the confidence of rejecting the null hypothesis of exchangeability and ε is a value in [0,1] that controls the transitions between the current and the previous martingale. Suppose that {Mk : k ≥ 0} is a non-negative martingale. If E(Mn) = E(M0) = 1, then from Doob's Maximial Inequality [START_REF] Vovk | Testing exchangeability on-line[END_REF]:

𝜆𝑃 ( max 0≤𝑘≤𝑛 𝑀 𝑘 ≥ 𝜆) ≤ 𝐸(𝑀 𝑛 )
and for any λ > 0 and n ∈ N, one has 𝑃 (max

𝑘≤𝑛 𝑀 𝑘 ≥ 𝜆) ≤ 1 𝜆 .
This inequality implies that it is unlikely for Mk to have a high value. Given this, the null hypothesis H0 is rejected when the martingale value is greater than λ.

In order to detect the movements of the behavioral features towards or away from their representative we average two martingale sequences [START_REF] Mozafari | A precise statistical approach for concept change detection in unlabeled data streams[END_REF]. The first martingale is as defined above and the second one is calculated using the complement of the 𝑝-value at each observation (i.e., 1 -𝑝̂𝑖). These two martingales sequences denoted M1 and M2 are averaged to generate the final martingale sequence that is monitored for values exceeding the threshold λ. One of the interesting properties of the martingale framework is the recursive property of the martingale sequence. Note that:

𝑀 𝑛 (𝜀) = 𝜀(𝑝̂𝜀 -1 )𝑀 𝑛-1 (𝜀)
This way, it is not necessary to store previous 𝑝-values. We only save the previous M1 and M2 values for the next hypothesis test upon the arrival of a new point.

Our motivation for applying the martingale framework is that it is a flexible approach to analyzing text streams. The algorithm can handle different mixes of features and can be implemented without the use of annotated datasets, which are common to text classification.

Again, to the best of our knowledge, this is the first time that the martingale framework is applied to a stream of Twitter data generated from a single user. De Choudhury et al. [START_REF] Choudhury | Predicting depression via social media[END_REF] used features generated from a user's Twitter history. However, the authors still employed a classifier to group users into two classes. Our approach is an improvement in that it considers an individual's behavior against his or her own history and not against other individuals.

Experimental evaluation Data collection and annotation

Due to the absence of publicly available datasets for the evaluation of suicide detection in social media, we used the Twitter streaming API to collect tweets. To evaluate our methodology, we needed two datasets: a sufficiently large annotated set to create the PMI-scored symptom lexicon (cross-sectional) and another smaller set of selected Twitter users and their history (longitudinal) to test the martingale framework.

We used the Twitter streaming API to collect tweets containing key phrases generated from the APA's list of risk factors and AAS's list of warning signs related to suicide. We randomly investigated the authors of these tweets to identify 60 distressed users who frequently wrote about depression, suicide, or self-mutilation. We also randomly collected 60 everyday users. A professional with expertise in mental health research validated the selection of these distressed and non-distressed users. For each set of users, we collected at most the last 50 tweets to create a database of 5,446 tweets of which 2,381 are from distressed users and 3,065 are from everyday users.

In order to test and compare our two methods for scoring the text of a tweet (NLP and machine learning) we randomly selected 250 tweets from the distressed users and 250 tweets from the non-distressed users for a total of 500 tweets. These tweets were removed from the set of 5,446 tweets and the remaining ones were used to create the PMI-scored symptom lexicon. To create a ground truth set of tweets, eight researchers and a mental health professional manually annotated the 500 tweets using four classes from 0 to 3. The classes are defined as follows:

 0: No distress -text discusses everyday occurrences such as work, going out, weekend activities, etc.

 1: Minimal distresstext expresses distress that could be considered common for most individuals (i.e. exam, presentation for work, argument with friend, etc.)

 2: Moderate distress -text expresses a level of distress that is above what an individual may experience in a normal week (i.e. insomnia, extreme crying, feeling alone, etc.)

 3: Severe distress -text includes mentions of self-harm, suicidal thoughts, gratefulness without direction, apologies, feelings of worthlessness, self-hate, guilt, not being good enough, etc.

Each tweet was reviewed by at least two annotators, with a subset of 55 tweets being validated by the psychologist. The annotated data had a Cohen's kappa statistic of 69.1%, which is considered as a substantial agreement among the annotators [START_REF] Viera | Understanding inter observer agreement: the Kappa statistic[END_REF]. The weighted kappa statistic, which takes into account different levels of disagreement, was around 71.5%. Finally, we used the Fleiss kappa to measure agreement for the 55 tweets with three annotators, which gives 78.3%. Overall, the three statistics show strong agreement between the annotators.

In addition to the cross-sectional dataset, we gathered Twitter history from a collection of users. This dataset, which may be considered as a longitudinal data, was collected from 10 unique real users that demonstrated a serious change in speech or online behavior in their Twitter accounts. Two initial validation cases, which ended with the individual committing suicide, were previously identified by [START_REF] Abboute | Mining Twitter for suicide prevention[END_REF]. We were able to identify a third fatal case and an additional seven other cases where the individuals demonstrated an abrupt change in behavior. These additional cases were manually chosen and the abrupt change in behavioral was principally judged by a change in speech.

In order to analyze the evolution of the behavioral features in this longitudinal data, we first identified the change point status or the tweet where the individual showed a change in behavior (namely speech). Using this as a pivot point, we collected 1,000 tweets before and 100 tweets after the change point status, ending up with a dataset of 11,000 tweets. The stream of tweets from each of these users was then used to test the martingale method for change point detection.

Results and discussion

Evaluation of the NLP based approach

To evaluate lexicon-based NLP approach, we used the cross-sectional set of 500 tweets and looked at the average, maximum, and minimum score given by each distress class. If performing well, the score should separate the tweets according to their annotated class. Ideally, there should be no overlap between the values in each class, i.e. all tweets of class 2 should have an overall score higher than the largest score for a tweet of class 1. To visualize this, we plotted the score distribution for each class using a box-and-whisker plot (Figure 1). The horizontal black line in the box represents the median observation, the box covers the inter-quartiles range (where 50% of the observations lie), and the whiskers are the maximum and minimum SPA text score for each class.

Figure 1 clearly shows the significant differences in the distribution of classes 0, 1, and 2, which means that the SPA text score is capable of differentiating levels of distress. Class 2 and class 3 have largely overlapping distributions, but we consider this acceptable as both classes indicate a high level of distress.

Evaluation of the Distress Classifier

The cross-sectional set of 500 tweets was also used to test the distress classifier for text scoring. We chose to test eight different classification algorithms and compare their performance: multinomial Naïve Bayes, Sequential Minimal Optimization (SMO) with a poly kernel, C4.5 decision tree (J48), nearest neighbor classifier (IB1), multinomial logistic regression, rule induction (Jrip), Random Forest, SMO with a Pearson VII universal kernel function (PUK).

Given that incorrectly classifying a tweet of class 3 is much more costly than incorrectly classifying a tweet of class 0 or 1, we use the recall for class 3 in addition to the other traditional measures to judge the performance of each classifier. The recall tells us the percentage of the class 3 observations that were correctly classified. If this percentage is low, then there is a risk that the classifier does not catch certain high-distress tweets.

Table 2 presents the best performing algorithms among the eight tested. The experiments were run with a 10-fold cross validation. The results show that the best performing model in terms of the weighted precision, is SMO with a PUK kernel function, yielding a value of 66.4% with the features n-grams, symptoms, pronouns, and swear components. However, we noticed that this promising preliminary result comes at the cost of a very low recall value for class 3.

This was seen across all of the eight classifiers that we chose for testing. Noticing this trend, we decided to modify our classification task into a two-step process. In the first step, a tweet is classified as distressed or non-distressed and the second step separates distressed tweets into two levels: class 2 or class 3.

For both steps, all eight classifiers were tested again. We experimented with three combinations of the best performing classifiers for the two-step process. We tested the Random Forest, Simple Logistic, and J48 classifying algorithms with the SMO algorithm as a second step classifier. The SMO algorithm performed the best when classifying tweets as class 2 or class 3. It had the highest recall for class 3 so we chose to keep the SMO as the second classifying algorithm. Table 3 shows the results for each combination.

The experiments show that our two-step classification process continues to perform well on the testing set. Specifically, we were able to reach a distressed class recall of 72% for the first classification step, which is arguably the most vital. The combination of the J48 and SMO classifiers resulted in the least amount of class 2 and class 3 instances being incorrectly classified as non-distressed (2%). This fact along with the weighted precision (71% and 80.7%) and recall for the highest risk class (72% and 69.2%) motivated us to use the J48 and SMO classifiers for our two-step tweet classification model.

Evaluation of the Martingale Framework for Emotion Change Detection

In this section, we present the results of the martingale framework on the set of longitudinal data. We use two of the Twitter users, Max and Frank, who presented marked change points in at least one behavioral feature. Their names have been changed to protect their privacy.

As a first step, we were interested in the effect of the different behavioral features on the martingale values. We examined the values with one dimensionthe SPA text scoreand four dimensionsthe SPA text score, friends, volume, and retweets. We focused on the SPA text score because its values are continuous. We found that the discrete classes output from the text classifier were not smooth enough and did not reflect the change points in online speech in side experiments.

Figure 2 show the series of martingale values for user Max, from observation 900 onward, using the SPA text score dimension as a single feature. The graph of the last 145 values clearly shows an increase in the values just before the true change point (#946: "Done with my life"). We also notice another change point detected with a high score (#950: "Goodbye..."), just after the true change point.

As we add dimensions into the martingale framework for the user Max, the increase in the martingale values after the true change point becomes more apparent. Figure 3 shows the distributions of the martingale values when they are computed using four dimensions -SPA text score, friends, volume, and retweets. We observe an interesting spike around observation (#372). This spike is due to a large number of tweets between Max and another user.

We did the same analysis for our other validation user Frank. Figure 4 shows the martingale values over the stream of tweets using one dimension, the SPA text score. Contrary to the first validation case, the one-dimensional martingale framework does not appear to increase after the indicated change point. In fact, the two peaks seen around observations 343 and 363 heavily affect the martingale values and they inhibit the framework from detecting a change point. Upon further investigation, we found that these two spikes are linked to negative SPA scores (positive emotion) corresponding to birthday wishes that Frank received from other users.

These peaks highlight that our martingale framework will need to be adjusted to interpret negative SPA text scores as a positive behavioral change that do not require an alarm to sound. However, it does indicate that our methodology detects efficiently changes in online speech.

While Figures 234show the evolution of the martingale values, the full martingale framework should detect the true change points. This occurs when the martingale value is greater than λ and the null hypothesis H0. We ran several experiments to determine the appropriate λ threshold and the testing set size for each validation case. The testing set is the set of observations used to establish the representative value for each dimension.

As previously described, after a change point is detected, the testing set is reset to create new representative values. Our experiments showed that after such a reset, the original λ threshold may no longer be valid for the new martingale values. In other words, the threshold may be too high and no change points are detected after the testing size is reached. Inversely, the threshold may be too low and multiple false change points are detected.

We present in Table 4 and Table 5 the best results of the emotion-change detection framework obtained for both users Max and Frank, respectively. The fact that the threshold λ does not necessarily remain relevant throughout the stream of tweets hinders the ability of the framework to detect a behavioral change. The true change point is detected for Max with a minimum delay of 14 observations, which is very satisfying. However, for Frank, the true change point is not detected.

Limitations of Current Study

While the presented methodology and experiments were carefully prepared, we are aware that there are certain limitations. Most notably, we present the results of the full methodology run on only two Twitter users. In order to best summarize the entire methodology from feature generation to change detection in a stream of data, we have decided to present the results of two specific users. One avenue for future research might include further testing on more user timelines. If possible, the testing would be done with a cohort of users identifying as suffering from mental distress.

Furthermore, the parameter setting of the martingale framework could be improved upon. This was one of the major challenges when implementing the framework. Additionally, the combination of behavioral features fed into the martingale framework could be another area of exploration. The features used to determine the mental health of a user may differ between individuals. Given this, it may even be interesting to monitor each behavioral feature individually with the martingale framework.

Conclusion

In this paper, we designed and evaluated a novel approach to monitor the mental health of a user on Twitter. Building off existing research, we worked to translate and quantify suicide warning signs in an online context (user-centric and post-centric behavioral features). In particular, we focused on detecting distress-and suicide-related content and developed two approaches to score a tweet: a NLP-based approach and a more traditional machine learning text classifier.

To detect changes in emotional well-being, we considered a Twitter user's activity as a stream of observations and applied a martingale framework to detect change points within that stream. Our experiments show that our NLP text-scoring approach successfully separates out tweets exhibiting distress-related content and acts as a powerful input into the martingale framework.

While the martingale values react to changes in online speech, the change point detection method needs improvement. We were able to detect the true change point for one validation case, but the approach needs to be more robust to parameter setting and positive changes in speech.

For future research, we plan to further explore the impact of martingale parameters on the change detection effectiveness. We also hope to expand the approach to include image processing and other social media outlets in order to assess the effectiveness in other settings. Another interesting perspective is to consider a more fine-grained emotion classes such as anger, sadness, fear, etc., instead of considering four levels of distress. However, overall, we believe our initial work presents an innovative approach to detecting suicide-related content in a text stream setting.

(jerome.aze@lirmm.fr). Dr. The total amount of friends at the time of post. A friend is defined as another Twitter user that the author is following online (out-link).

Followers

The total amount of followers at the time of the post. A follower is another Twitter user that is following the author online (in-link).

Volume

The number of tweets per hour; Retweets are included in this feature.

Replies

The number of tweets per day directed at another Twitter user. An author may post a tweet destined for another user by including "@" plus the user's screen name.

Retweets

The number of tweets per day that are retweeted. A retweet is defined as a tweet previously composed by another Twitter user and that is re-published or shared.

Links

The number of tweets per day that include a URL Questions

The number of tweets per day posing a question.

Post-centric features Time

The hour at which the tweet was published. Text Score Text Score (based on NLP) / Distress Classifier

Figures and tablesFigure 1

 1 Figures and tablesFigure 1 Distribution of SPA text scores by distress class.

Figure 2

 2 Figure 2 Stream of martingale values for tweets from user Max -The values are computed using only the feature SPA text score. The point represents the true abrupt emotion change point (#946: "Done with my life"). The martingale values may begin to increase prior to the change point, but with a sudden change in behavior, the values may only increase after the change point.

Figure 3

 3 Figure 3 Martingale values distribution over the series of tweets for user Max. The values are computed using four dimensions (SPA text score, friends, volume, and retweets). The point represents the true abrupt emotion change point (#946).

Figure 4

 4 Figure 4 Martingale values distribution over the series of tweets for user Frank. The values are computed using only the feature SPA text score (one dimension). The point represents the true abrupt emotion change point (#930).

  Azé is a professor in the Montpellier Laboratory of Informatics, Robotics and Microelectronics. Previously, he was assistant professor and member of the bioinformatic team at the University of Paris-Sud. Currently, he is the head of the Multimedia and Internet Department at the University of Technology, Béziers, France. His research mainly includes data mining and its applications on e-health, sentiment analysis and suicide detection in social networks.Sandra Bringay LIRMM UM CNRS, UMR 5506, 860 Rue St Priest, 34095 Montpellier, France (sandra.bringay@lirmm.fr). Dr. Bringay is a professor in the Montpellier Laboratory of Informatics, Robotics and Microelectronics and is a member of the toxico-vigilance working group managed by the National Institute of Health Monitoring. She is leading several projects on biomedical research and medical information. Her current research primarily focuses on health data mining, sequential pattern mining, and sentiment analysis.

Table format Table 1

 format1 Description of the behavioral features. (For the Text Score calculation and Distress Classifier mentioned at the bottom of the table, see the "Feature Extraction for Text Scoring" sections of this paper.)

Table 2

 2 Best Performing Classifiers -Four Distress Classes. The bold-font numbers highlight the best performing models in terms of each evaluation measure using the given behavioral features.

			SMO (PUK)			Random Forest
	Features	Precision Recall	F-measure	Class 3 Recall	Precision	Recall	F-measure	Class 3 Recall
	N-grams	0.655	0.413	0.315	0.029	0.483	0.468	0.434	0.265
	+ Symptoms	0.625	0.431	0.334	0.039	0.540	0.553	0.506	0.245
	+ Pronouns	0.614	0.427	0.329	0.029	0.541	0.551	0.507	0.245
	+ Swear	0.664	0.429	0.333	0.039	0.554	0.555	0.518	0.245
	+ Intensifiers	0.615	0.431	0.332	0.029	0.571	0.559	0.516	0.200

Table 3

 3 Two-Step Classification Results

	Step	Classifier	Precision	Recall	F-measure	Highest Risk Recall (Class 3)
	1	RandomForest	0.676	0.676	0.676	0.341
	2	SMO	0.829	0.817	0.823	0.791
	1	SimpleLogistic	0.706	0.706	0.706	0.692
	2	SMO	0.770	0.762	0.766	0.697
	1	J48	0.71	0.71	0.711	0.72
	2	SMO	0.807	0.795	0.793	0.692


MD Lenoir Philippe

Moulahi Bilel

Aze Jérôme

Bringay Sandra

Carbonnel François

Raising awareness about cervical cancer using Twitter: a content analysis of the 2015 #SmearForSmear campaign

Keywords: uterine cervical neoplasms, papanicolaou test, social media, early detection of cancer, health promotion, smear test, twitter National Health Service UK: United-Kingdom

Background: Cervical cancer is the 2 nd cause of cancer among women under age 45. To deal with the decrease of smear test (ST) coverage in United Kingdom, a Twitter campaign called #SmearForSmear has been launched in 2015 for the European Cervical Cancer Prevention Week. Its aim was to encourage women to take a selfie showing their lipstick going over the edge and post it on Twitter with a raising awareness message promoting cervical cancer screening. Estimated audience was 500 million of people. Other public health campaigns have been launched on social medias such as Movember to encourage participation and self-engagement. Their result was unsatisfactory as their aim had been diluted to become mainly a social buzz. Objective: Our objectives were to identify the tweets delivering a raising awareness message promoting cervical cancer screening (sensitizing tweets) and the characteristics of Twitter users posting about this campaign. Methods: 3-step content analysis of the English tweets tagged #SmearForSmear posted on Twitter for the 2015 European Cervical Cancer Prevention Week. Data were collected using Twitter API. Their extraction was based on an analysis grid generated by two independent researchers using a thematic analysis, validated by a strong Cohen's kappa coefficient. 7 themes coded for sensitizing tweets and 13 for Twitter users' status. Verbatims were thematically then statiscally analyzed. Results: 3018 tweets were collected, 1881 analysed. 70% of tweets had been posted by people living in United Kingdom. 57.4% of users were women, sex was unknown in 36% of cases. 54.4% of the users had posted at least one selfie with smeared lipstick. 32.2% of tweets were sensitizing. Independent factors associated with posting sensitizing tweets were: woman who experienced an abnormal ST ], p = 0.0005), female sex .598], p < 0.0001) and live in the United-Kingdom (OR 2.097 [CI95% 1.447-3.038], p < 0.0001). Non-sensitizing tweets were statistically more posted by non-health or non-media company (OR 0.558 [CI95% 0.383-0.814], p = 0.0024). Conclusions: This study demonstrates that the success of a public health campaign using a social media depends on its ability to get its targets involved. It also suggests the need to use social marketing to help its dissemination. The clinical impact of this Twitter campaign to increase cervical cancer screening is yet to evaluate.

Introduction:

Cervical cancer is the second cause of cancer among women under age 45 and leads to significant mortality [START_REF] Ferlay | Cancer incidence and mortality worldwide: sources,methods and major patterns in GLOBOCAN 2012[END_REF]. Cervical cancer is caused by human papilloma virus (HPV) [START_REF] Walboomers | Human papillomavirus is a necessary cause of invasive cervical cancer worldwide[END_REF]. Smear test (Papanicolaou test) detects precancerous changes and earlier stage cervical cancers. Its introduction has allowed a dramatic decline of cervical cancer incidence and death rates in many countries, such as the developed countries [START_REF] Thun | The global burden of cancer: priorities for prevention[END_REF]. In England, an organized national screening program has been established in 1988. Incidence of cervical cancer in women aged 20-79 in England has almost halved from 1982 to 2006, thanks to this program. However, its incidence is now rising in 20-29-year-olds from 1996 onwards in most regions in England [START_REF] Foley | Increasing rates of cervical cancer in young women in England: an analysis of national data 1982-2006[END_REF]. From 1999 to 2013, the number of women that didn't attend their smear test for a 5-year period has progressively increased from 16% to 22% [START_REF] Marlow | Understanding cervical screening non-attendance among ethnic minority women in England[END_REF]. It suggests that organized screening isn't intrinsically strong enough to keep a high coverage rate. Social medias would have a great potential to improve behavior change as interactive tools, encouraging participation and self-engagement instead of a « descending » information [START_REF] Strecher | The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial[END_REF][START_REF] Crutzen | Strategies to facilitate exposure to internet-delivered health behavior change interventions aimed at adolescents or young adults: a systematic review[END_REF][START_REF] Neiger | Evaluating social media's capacity to develop engaged audiences in health promotion settings: use of Twitter metrics as a case study[END_REF]. Facebook, Twitter or Instagram had respectively more than 1.86 billion, 317 million and 500 million monthly active users in December 2016. For Twitter, more than 500 million tweets are traded every day [START_REF] Bagadiya | 151 amazing social media statistics you should know in[END_REF]. These social media have become a valuable source of information for health professional and clinicians to effectively discovering health related topics and behaviors online [START_REF] Zhan | Identifying Topics for E-Cigarette User-Generated Contents: A Case Study From Multiple Social Media Platforms[END_REF][START_REF] Attard | A thematic analysis of patient communication in Parkinson's disease online support group discussion forums[END_REF][START_REF] Lemire | Determinants of Internet use as a preferred source of information on personal health[END_REF][START_REF] Ybarra | Help seeking behavior and the Internet: a national survey[END_REF]. Public health campaigns have already tried to take advantage of the ability of social medias to make a campaign viral. The ALS Ice Bucket challenge's goal was to mediatize and fundraise for the amyotrophic lateral sclerosis. It had involved many celebrities worldwide. On 1 September 2014, more than 17 million videos had been shared on Facebook and had been watched more than 10 billion times by more than 440 million people [START_REF]The ice bucket challenge on Facebook[END_REF]. Thanks to this campaign, more than 100 million dollars had been raised by the ALS association [START_REF] Worland | ALS Ice Bucket Challenge Donations Just Topped $100 Million. Time[END_REF]. Hundreds of thousands of people had tweeted daily about ALS, which is a much higher number of tweets than those emitted about multiple sclerosis, a disease better known to the public [START_REF] Koohy | A lesson from the ice bucket challenge: using social networks to publicize science[END_REF]. Movember is an annual event organized each November since 2003 whose goal is to raise awareness and fundraise about masculine disease such as prostate or testicle cancer. Participants let their moustache grow and post selfies on social medias to raise the awareness of their contacts and show their involvement in this campaign. In Denmark, after the initiation of the 2011 Movember campaign, a significant decline in the PSA level at referral and an increase in the number of patients referred under suspicion of prostate cancer was observed. However, only minor differences in referral patterns and prostate cancer diagnosis were detected [START_REF] Thomsen | The Movember campaign: Impact on referral patterns and diagnosis of prostate cancer[END_REF]. These campaigns may be parasitized by the buzz they seeked to create and may vehicle non-health-related messages. A content analysis of the 2013 Movember Canada campaign on Twitter showed that it did not meet the stated campaign objective of creating conversations about men's health and, specifically, about prostate and testicular cancers [START_REF] Bravo | Tweeting About Prostate and Testicular Cancers: Do Twitter Conversations and the 2013 Movember Canada Campaign Objectives Align?[END_REF]. To deal with the decrease of smear coverage in United Kingdom, a Twitter campaign called #SmearForSmear has been launched in 2015 by the Jo's Cervical Cancer Trust for the European Cervical Cancer Prevention Week. Its goal was to encourage women to take a picture of themselves (selfie) showing their lipstick going over the edge and post them on Twitter with an awareness message promoting cervical cancer screening. Estimated audience was 500 million people [START_REF] Smith | Cervical Cancer Prevention Week : Facts, statistics and the #SmearForSmear awareness campaign[END_REF]. Our objectives were to identify the tweets delivering raising awareness messages about cervical cancer screening (RAM) and the characteristics of Twitter users posting about this campaign.

Methods:

We conducted a 3-step content analysis of the English tweets posted on Twitter during the 2015 European Cervical Cancer Prevention Week.

Data collection and extraction:

To collect the tweets, we used Twitter Application Programming Interface (API). It allows the user to conduct manual searches for keywords in tweets with specific parameters such as hashtags, language, and date range. The ones used for this research were: #SmearforSmear, English language, and tweets posted between 25 January 2015 to 31 January 2015 inclusive (European Cervical Cancer Prevention Week). All tweets have been manually collected and assessed. Only original tweets, rather than retweets, were analyzed. In the tweets, only the verbatim were transcribed. Hashtags and content preceded by "@" were removed if that action didn't make the verbatim unintelligible. We also considered all hypertexts linked to another verbatim on another web-platform (e.g., Instagram). The corresponding verbatim were transcribed only if they were informative.

Data analysis:

A total of 3019 tweets that met the search criteria were imported into Excel for data extraction. An analysis grid had been created based on 200 tweets initially collected and thematically analyzed by two independent researchers to extract the themes (topics) of tweets' verbatim and Twitter users' statuses. Then, this grid had been tested on 50 new tweets. No new themes had been identified, confirming that category saturation was achieved. The thematic analysis methodology used consists of transforming qualitative content into a quantitative form by establishing coding categories. The number of data units that fall into each coding category are counted (such as phrases, messages, responses). Finally, they are categorized based on similar meanings and overt or inferred communication [START_REF] Kondracki | Content analysis: review of methods and their applications in nutrition education[END_REF][START_REF] Hsieh | Three approaches to qualitative content analysis[END_REF]. Themes were not restricted to preexisting themes. They emerged through an inductive process whereby open coding of data revealed themes that moved from the specific to the general [START_REF] Elo | The qualitative content analysis process[END_REF]. The two researchers, both general practitioners and trained in qualitative study, elaborated a 7-themes codebook, based on tweets, to identify if the tweets delivered raising awareness messages about cervical cancer screening: incentive to carry out the smear test, evocation of smear test importance without any precision, reminder of the smear test preventive nature, reminder of the low incidence of smear test, allusion to the mortality or morbidity of cervical cancer, reminder of the incidence of cervical cancer and testimony of an experience related to smear test or cervical cancer. If a tweet had at least one of these awareness-raising messages, it was considered a sensitizing tweet. Reproducibility of the classification of 300 tweets by the two independent researchers was tested and calculated with Cohen's kappa coefficient. The agreement was strong and varied between 0.8842 and 1.

The following information was collected about each tweet: verbatim, posting date, retrieval date, presence of a selfie with lipstick going over the edge, picture or video referring to the campaign, user sex, user location, number of followers at the date of retrieval, presence of a selfie with lipstick going over the edge, and user's status. To classify the users, we used their Twitter status. If it didn't exist or was incomplete, we extracted these information from links on their Twitter profile, whenever possible. The analysis grid enlisted 14 themes regarding Twitter user's status: health company, media company, non-health and non-media company, marketing company, fashion company, blogger or youtuber, health professional, National Health Service (NHS), politician, woman who experienced cervical cancer or relatives, woman with unspecified cancer or relatives, woman who experienced an abnormal smear test, general public and unknown. The "unknown" status was attributed when no information to categorize the user was available. Only the "unknown", "general public" or "NHS" status were exclusive. An initial global description of the sample has been performed, using the frequencies of the different categories for the qualitative variables. As the distribution of quantitative variables wasn't always Gaussian (Shapiro-Wilk test), they were expressed by their mean, standard deviation, median, minimum and maximum values and interquartile. Comparison of means was executed through the Student test when distribution was Gaussian, otherwise based on Mann-Whitney test. Comparison of qualitative variables was executed through the Chi2 test for parametric tests, or Fisher exact test when the conditions for applying Chi2 were not observed. A multivariate logistic modeling process was then carried out to identify the independent factors associated with the presence of a sensitizing message in the tweets and associated with each type of sensitizing message. A "step by step" selection procedure of the variable was used with an input and output variable set at 0.10 and 0.05 respectively. The significance threshold was set to 5%. Statistical analysis had been performed by the Department of Medical Information at Montpellier teaching hospital with SAS version 9. This research did not require ethics approval since we only used publically available Twitter content.

Results:

3018 tweets met the search criteria. 1138 were removed (retweets or copies of tweets) and 1881 were analysed. 608 tweets (32,2%) were sensitizing and included from 1 to 5 raising awareness message (RAM), mean 0.54 +/-0.93 (Table 1). Incentive to carry out the smear test was the most frequent RAM. Main users were people from English-speaking countries. United Kingdom accounted for 70% of the posted tweets, followed by USA (9%) and Australia (1%). Nationality was unknown in 15.2% of cases. 57.4% of users were women, sex was unknown in 36% of cases. Twitter users had a mean number of followers of 44 420.8 +/-420 819. 54.4% of the users had posted at least one selfie with smeared lipstick. 15.6% of the tweets were associated with a picture or a video referring to the #SmearforSmear campaign.

Univariate and multivariate analysis

Significant statistic associations between emitting sensitizing tweets and Twitter user status are detailed on Table 2. The "step by step" selection procedure has allowed to identify independent factors influencing the sensitizing characteristic of tweet (Table 3).

Discussion:

Main results: 32.2% of the tweets of the #SmearforSmear campaign were sensitizing. This result was promising as it goes well beyond the results of the 2013 Movember campaign where only 0.85% of the posted tweets may raise awareness of men's health risks [START_REF] Thomsen | The Movember campaign: Impact on referral patterns and diagnosis of prostate cancer[END_REF]. Many factors may explain this gap. On one hand, this campaign had been created using social marketing in a holistic approach. Its objective was clear, its title referred to its objective. Jo's cervical cancer trust posted key messages reflecting the need to adhere to the screening of cervical cancer, used to fill the content of tweets. A slogan had also been created "Attend your smear, reduce your risk", widely retweeted in this campaign. On the other hand, this campaign created to detect an exclusive feminine cancer was based on elements of two women's social construct, lipstick and selfies [START_REF] Etcoff | Cosmetics as a feature of the extended human phenotype: modulation of the perception of biologically important facial signals[END_REF][START_REF] Fox | Selective Self-Presentation and Social Comparison Through Photographs on Social Networking Sites[END_REF]. This approach was possible because this campaign had been designed for the UK, where the cervical cancer screening is organized. Targeted women automatically receive a letter explaining them what to do to get screened and where. Receiving an invitation letter is an independent sensitizing factor associated with greater likelihood of cervical cancer screening [START_REF] Tavasoli | Impact of invitation and reminder letters on cervical cancer screening participation rates in an organized screening program[END_REF].

As for the Twitter users, our expectations were broadly confirmed. From a general point of view, Twitter users posting sensitizing tweets were people personally involved in cervical cancer screening: women, women concerned by a feminine cancer or their relatives, people living in the UK (where this English-speaking campaign took place), the NHS as a partner of this campaign and women who experienced an abnormal ST. As peer, women raised awareness by insisting on the preventive aspect of ST and directly encouraged other women to attend their ST. Peer influence is known as an important social lever for health-related behaviour change [START_REF] Webel | A systematic review of the effectiveness of peer-based interventions on health-related behaviors in adults[END_REF]. Likewise, women or their relatives who experienced a pathological state (abnormal ST, cervical cancer or an unspecified cancer) had the greatest potential among categories of Twitter users to post a sensitizing tweet. Hashtags, such as #SmearforSmear, tend to create communities behaving as support group [START_REF] Park | Tweeting as Health Communication: Health Organizations' Use of Twitter for Health Promotion and Public Engagement[END_REF]. Unveiling elements of private life is conducive to trust and emotional bond [START_REF] Ussher | What do cancer support groups provide which other supportive relationships do not? The experience of peer support groups for people with cancer[END_REF]. Fashion company was a user status that has a significant potential to post tweets about the importance of smear test without any precision. Actively participating to the campaign by posting selfies and pictures or videos linked to the #SmearforSmear campaign helped encouraging people to attend their ST and disseminate the importance of ST. Women's magazines also act as a guidebook and reinforce women's individual responsibility to create and maintain good health for themselves and their families [START_REF] Roy | Taking charge of your health »: discourses of responsibility in English-Canadian women's magazines[END_REF]. As for the other user categories, the RAM in their tweets was in line with expectations. Politicians broadcasted information about the low incidence of ST and how it helped preventing cervical cancer, in relation with their use of social media to communicate with the press and the public [START_REF] Lyons | How (Not) to Talk on Twitter: Effects of Politicians' Tweets on Perceptions of the Twitter Environment[END_REF]. Health companies' RAM was more direct, encouraging people to attend their ST. The general public cautioned about incidence of cervical cancer. The NHS insisted on the importance of ST without giving more information. It was probably in relation with the fact that NHS was only a partner of this campaign and that it only helped disseminating it. Finally, there was a scotoma of health professionals. This status didn't emerge as a relevant category. Their participation to a health campaign on social medias is interesting as it has been shown that the information contained in their posts are more likely to be true compared to other groups [START_REF] Alnemer | Are Health-Related Tweets Evidence Based? Review and Analysis of Health-Related Tweets on Twitter[END_REF]. This under-representation was probably due to the shortness of the studied campaign period. Conversely, "non-sensitizing" tweets had a much greater probability to be sent either by users not directly concerned by cervical cancer such as men (exclusively feminine cancer) or by users that participated but only broadcasted information, without getting involved: medias, marketing companies and non-health/non-media companies. It questions their participation to this campaign. Was it about an opportunistic appropriation of a viral campaign? It is probably one of the main limitations of the virality of health campaigns on social media. Most tweets posted for the 2013 Movember campaign and the breast cancer prevention month did not spark conversations about prostate and testicular cancer nor promote any specific preventive behavior about breast cancer [START_REF] Bravo | Tweeting About Prostate and Testicular Cancers: Do Twitter Conversations and the 2013 Movember Canada Campaign Objectives Align?[END_REF][START_REF] Thackeray | Using Twitter for breast cancer prevention: an analysis of breast cancer awareness month[END_REF]. They may also be an interesting lever for social stimulation.

Strengths & Limitations:

To our knowledge, no study analyzing the content of the #SmearforSmear campaign on Twitter has been published yet. Our findings are corroborated by the content analysis of others health campaigns on Twitter. We used a content analysis method based on a double analysis of the sensitizing capacity of each tweet, in an exploratory process. We also mined Twitter to gather information about users' characteristics and complete the tweets' content. This highly demanding method made us decide early to restrict our study to one week. This choice was also relevant, in our opinion, as this campaign had been created for the European Cervical Cancer Prevention Week. Compared to other Twitter campaigns, our relatively high results must question its ability to keep a high proportion of sensitizing tweets in other countries (particularly where the cervical cancer screening is not organized) and if it remains high over time.

The choice to collect the tweets based on the hashtag #SmearforSmear may have limited their number, by omitting those not using it. As for the content analysis, two safeguards have been used: analyzing the content of tweets to create the categories before the study and evaluate the reproducibility of the classification by two independent researchers with Cohen's kappa coefficient, which was strong in this study. The shortness of Twitter posts, limited to 140 characters may have created a loss of information as users often used hyperlinks to be exempt from this limit. We then chose to manually mine Twitter to complete the tweets' content and gather information about users' characteristics.

Perspectives:

The #SmearforSmear campaign has allowed to disseminate sensitizing messages about cervical cancer screening and became viral. It was based on a well-designed campaign, on a facilitating audience and a facilitating health system using an organized screening.

Choosing a social media adapted to the target is a major concern for a successful campaign. Twitter is interesting as it is well-suited for appointment campaigns such as #SmearforSmear or the ALS ice bucket challenge. It also is a social media used by young adults to keep up in real-time with news [START_REF] Walton | Twitter vs Facebook vs Instagram : who is the target audience ? Investopedia[END_REF]. But its audience is mainly male, living in urban areas. And although diverse, its percentage of users with college educations and incomes over $50,000 is much higher than those of Facebook or Instagram. Users of Instagram are mainly female but 72% of online American adults use Facebook and its audience is the most engaged with 70% logging on daily [START_REF] Walton | Twitter vs Facebook vs Instagram : who is the target audience ? Investopedia[END_REF]. Health campaigns on social medias must be seen as a way to reduce social inequities in health. In the UK, the main decline in screening was about 25-49-year-old women and black and Asiatic ethnic minorities [START_REF] Worland | ALS Ice Bucket Challenge Donations Just Topped $100 Million. Time[END_REF]. Targeted audience must be on the social media chosen and then adapt to the shift of the evolution of their audiences.

The impact of facilitators is to be studied. As previously shown, many Twitter users of this campaign didn't engage in this campaign as they didn't post sensitizing tweets. But they participated and helped broadcasting to their audience. Models such as Cara Delevingne also posted a selfie to support campaign to her millions of followers (8.5 million in May 2017) [START_REF] Delevingne | [END_REF]. They may boost a campaign as influencers and role model. The present findings show a clear need for studies that are capable of automatically analyze the data and extracting useful insights from the #Smear4Smear Twitter campaign. We propose the use of machine learning to tackle these challenges, and we suggest three perspectives for future directions. First, we plan to undertake a large-scale analysis using a collection of tweets that we are currently crawling since February 2017. Thisanalysis will include (i) the application of the Latent Dirichlet Allocation (LDA) in order to extract the topics [START_REF] Wang | Exploring health topics in Chinese social media: An analysis of Sina Weibo[END_REF][START_REF] Zhan | Identifying Topics for E-Cigarette User-Generated Contents: A Case Study From Multiple Social Media Platforms[END_REF][START_REF] Lu | Health-related hot topic detection in online communities using text clustering[END_REF] emerging from the discussions about the campaign, as well as (ii) the exploration of the linguistic style [START_REF] Zeng | Exploring and developing consumer health vocabularies[END_REF] of the Twitter' users. Second, we could benefit from statistical learning techniques [START_REF] Himmel | Text mining and natural language processing approaches for automatic categorization of lay requests to web-based expert forums[END_REF] to predict automatically the categories of all tweets about the campaign. This study may allow us to assess our findings and generalize our results.

We will learn a model with the One-Vs-the-Rest multi-class classifier based on an annotated data set and we will apply it on all tweets about the campaign. We will compare the results to the manual processing and annotation done so far. Moreover, within a sufficiently large data set, we can take advantage of machine learning models to use features that are more complex to characterize the users tweeting about the campaign. We suggest focusing on user groups including health professionals, celebrities, general public politicians. This will lead us to understand which group of users is prominent, so that it could influence others making them to action by retweeting the messages relevant to the campaign, liking and replying to tweets or more importantly donating money. Third, we plan to investigate the temporal distribution of messages to focus on the campaign dynamics over time. We may study the temporal correlations between the reactions of twitter users and real world events such as media coverage of the campaign. This analysis is exploratory and it could help in identifying the factors contributing to the raising the awareness. For example, a televised promotion of the campaign or a promotion published by a celebrity may stimulate a huge volume of tweets and reactions online.

Beyond this, we can also analyze the geographical distribution of tweets during the campaign. Health campaigns on social networks may raise awareness of public health issues. Their clinical impact is yet to evaluate. It will be a hard task in a hyper-connected world to be able to individualize the effect. This scientific step is important to convince stakeholders, health professionals and general public to get involved and use Web 3.0 as a collective intelligence to drive back chronic diseases [START_REF] Lemire | Determinants of Internet use as a preferred source of information on personal health[END_REF][START_REF] Ybarra | Help seeking behavior and the Internet: a national survey[END_REF], particularly for the most fragile ones.

Aknowledgements:

XXX for his help on the statistics XXX for his help reviewing the article.

Table 1 : Description of tweets and Twitter users

 1

	Variable	Total n (%)
	Number of raising awareness message in a tweet
	-0	1273 (67.68)
	-1	347 (18.45)
	-2	149 (7.92)
	-3	83 (4.41)
	-4	25 (1.33)
	-5	4 (0.21)
	Sensitizing tweet	608 (32.32)
	-Incentive to carry out the smear test	440 (23.39)
	-Reminder of smear test preventive nature	217 (11.54)
	-Allusion to the mortality or morbidity of cervical cancer	134 (7.12)
	-Testimony of an experience related to smear test or cervical cancer	92 (4.89)
	-Smear test importance	63 (3.35)
	-Evidence of the number of cervical cancers	41 (2.18)
	-Low Incidence of smear test	27 (1.44)
	Categories of Twitter users:
	-Unknown	442 (23.5)
	-Non-health and non-media company	396 (21.05)
	-Health company	292 (15.52)
	-Blogger or youtuber	262 (13.93)
	-Media company	240 (12.76)
	-Fashion activity	240 (12.76)
	-Marketing activity	220 (11.70)
	-NHS	79 (4.2)
	-General public	77 (4.09)
	-Woman who experienced cervical cancer or relatives	60 (3.19)
	-Health professional	53 (2.82)
	-Woman who experienced an abnormal ST	33 (1.75)
	-Politician	12 (0.64)
	-Woman who experienced unspecified cancer or relatives	6 (0.32)

Table 2 : Twitter users known characteristics

 2

	Characteristics	Total n (%)	P value
	Characteristics linked to a higher probability of emitting sensitizing
	tweets:
	-United-Kingdom	1316 (82.51)	<0.000001
	-Female sex	1079 (89.62)	0.000002
	-NHS	79 (4.2)	0.020013

Table 3 : Independent factors influencing the emission of sensitizing tweets (* statiscally significant influence on the emission of non-sensitizing tweets)

 3

	Message of tweet, Variables	Adjusted OR (95% CI) P value
	Sensitizing tweet
	-Woman who experienced an abnormal ST	13.456 (3.101, 58.378)	0.0005
	-Female Sex	3.752 (2.133, 6.598)	<0.0001
	-United-Kingdom	2.097 (1.447, 3.038)	<0.0001
	-Non-health or non-media company*	0.558 (0.383, 0.814)	0.0024
	Direct encouraging people to go for ST
	-Female Sex	5.967 (2.606, 13.659)	<0.0001
	-Health company	2.203 (1.042, 4.656)	0.0386
	-United-Kingdom	1.997 (1.320, 3.021)	0.0011
	-Selfie	1.673 (1.228, 2.280)	0.0011
	-Non-health or non-mass media company*	0.481 (0.310, 0.746)	0.0011
	Evocation of the importance of smear test without any precision
	-Woman who experienced an abnormal ST	7.365 (2.314,23.436)	0.0007
	-NHS	4.266 (1.778,10.238)	0.0012
	-United-Kingdom	2.888 (1.015,8.212)	0.0467
	-Fashion	2.724 (1.430,5.188)	0.0023
	-Selfie	2.158 (1.163,4.002)	0.00147
	Reminder of the preventive aspect of smear test
	-Woman who experienced an abnormal ST	4.216 (1.734,10.254)	0.0015
	-Politician	3.545 (1.028,12.221)	0.00451
	-Female sex	2.555 (1.156,5.646)	0.00204
	-Marketing activity*	0.414 (0.211,0.812)	0.00103
	Evocation of the mortal or morbid aspect of cervical cancer
	-Woman who experienced unspecified cancer or relatives	6.359 (1.043,38.776)	0.0449
	-Woman who experienced an abnormal ST	5.591 (2.227,14.035)	0.0002
	-Female Sex	3.396 (1.050,10.982)	0.0412
	-Woman who experienced cervical cancer or relatives	2.598 (1.228,5.495)	0.00125
	-United-Kingdom	2.268 (1.069,4.808)	0.0328
	Reminder of the low incidence of smear test
	-Politician	14.754 (3.074,70.816)	0.0008
	Reminder of the incidence of cervical cancer
	-General public	2.913 (1.002,8.474)	0.0497
	-Picture or a video linked to the #SmearforSmear campaign	2.701 (1.372,5.318)	0.0040
	Statement from people who experienced abnormal smear test or
	cervical cancer
	-Woman who experienced an abnormal ST	65.364 (22.709,188.140	<0.0001
	-Woman who experienced unspecified cancer or relatives	14.371 (2.335,88.417)	0.0040
	-Woman who experienced cervical cancer or relatives	7.641 (3.690,15.822)	<0.0001

Conflicts of interest:

None to declare

Funding: None

Multimedia appendix of supplementary files:

Supplementary file 1


Christophe Reffay
email: christophe.reffay@univ-fcomte.fr

Madeth May
email: madeth.may@univ-lemans.fr

Alain Mille
email: alain.mille@liris.cnrs.fr

French HUBBLE project experience on Ethics in Learning Analytics

Keywords: Ethics, Learning Analytics, HUBBLE, tracking data, traces

This paper discusses a research effort on ethical aspects of the French HUBBLE project that brings together research teams from different areas, including Computer and Human Sciences. Nine cases that make use of e-learning traces are currently running in this project. Seven case studies' team completed a questionnaire focused on ethics perspectives. The answers of this questionnaire are being reviewed by the authors as the responsible persons in charge of the Ethics work package of the HUBBLE project. By placing a special focus on the process designed to handle ethical concerns, and tracking ethics errors in concrete case studies, this paper presents our attempt in making sure that ethics is being fairly treated by the researchers and by other partners of the project.

HUBBLE project

The HUBBLE project (HUman oBservatory Based on anaLysis of eLEearning traces) [START_REF] Luengo | Hubble project website[END_REF] proposes the creation of an observatory of e-learning traces analysis and sharing.

The observatory aims at giving the participants involved in the educational settings (e.g. teachers, researchers, designers, students, administrators and policy makers) the possibility to analyze and explain the teaching and learning phenomena occurring within e-learning environments. The analysis processes we build will therefore accompany the researchers in the decision-making, while also guiding them in the elaboration of analytical constructs, models and indicators.

The project brings together research teams from different areas, including Computer and Human Sciences, to co-design and implement a national observatory. Built upon existing platforms developed by the project partners such as UnderTrack [START_REF] Mandran | UnderTracks[END_REF], kTBS4LA [3], UTL [START_REF] Iksal | Environnement UTL[END_REF], the observatory capitalizes the trace analysis processes and makes them available to the research community in Learning Analytics through xAPI or CSV as common formats.

The main goal of the HUBBLE project is to enable the construction, sharing and management of the analysis processes made on the traces being shared among the researchers of the project. In such a context, ethical concerns have to be taken into account in the design and in the usage of the observatory. The authors of this paper are in charge of a work package that covers the ethical aspects of the HUBBLE project.

The team is treating the ethical questions according to the organization of the whole project based on case studies and analysis scenarios.

• A case study is a specific learning situation, with teachers, learners, managers (administrative, political, educational, technical people) and also learning platforms (Moodle, FUN MOOC platform, etc.). • The analysis scenarios describe the objectives of the analysis, the context of its use and results (indicators, models). They will be described starting from the case studies and are categorized in either researcher or decision maker scenarios.

Table 1 gives a summary of nine case studies being treated in the first project loop. It also gives a glimpse of a huge variety of traces by their nature and volume, which make the study of ethics of each case study even more challenging. NB: the case studies presented here were used in the first loop of the Hubble project (in 2016). Some of them are running every year.

To ensure that the decision makers can reuse most of the analysis processes available in the HUBBLE observatory, an "editorial chain" will be suggested. Ethical concerns include privacy issues, but address broader issues regarding the design, the use of elearning systems and the exploitation of the learning traces. These concerns are also very relevant to the learners, policy makers, tutors, teachers, etc. To address them, an ethics committee has been created within the HUBBLE project. The committee works closely with the representatives of each case study with an objective to ease their workflow and to accompany them when it comes to ethical questions.

The rest of the paper is organized as follows. The next section presents the work process in ethics work package. It focuses on how we treat ethical aspects in each case study through a submitted form. In section 3, we draw a conclusion with the assessment of Ethics for LA from the HUBBLE project standpoint.

2

The ethics treatment of each case in the Hubble project

Nine case studies are considered inside the HUBBLE project (see Table 1). According to the process designed to address ethical issues, we asked the representatives of each case study to submit their questionnaire related to the ethical aspects of different analysis processes being worked on. Thus far, the ethic committee has received seven questionnaires. It is worth mentioning that the questionnaire is expressed in French; but an English version is now available online [START_REF] Reffay | CELA guide and submission form: An English version[END_REF]. It covers specific questions, among which, the purpose and objectives of the research, the recruitment of participants, the protocol, the consent form, the benefits and risks, the treatments on data, the privacy and anonymization, the archive duration, the conditions of data storage and sharing. Below is the summary of the most important aspects pointed out during the review of each submitted questionnaire:

• The description of the stored data is not sufficient to decide if privacy is preserved; • The participants were not informed that their learning interactions had been monitored; • The consent form was missing or was not available in all case studies; • The data collection process on mobile devices (tablets in some case studies) might have kept track of private activities;

Considering that most of the researchers were informed of the ethical aspect of their research for the first time, it is not surprising for us to witness all the emerging anomalies. Nonetheless, the positive note of the whole process of making sure that ethics is being fairly treated by the research colleagues in the HUBBLE project is that they have been sincere with their answers and their declarations. While the outcome of our research effort is encouraging, we have to acknowledge that there are still some milestones before full compliance of our research methods and data treatments with French and European ethics laws or recommendation. Having said that, being fair and honest is the first step in making ethics relevant and highly considered in Learning Analytics.

Last but not least, the review of each case study will be discussed in the French Learning Analytics Ethics Committee (CELA), and the recommendations will be sent to the researchers.

Conclusion

As members of the Ethics work package, we have the feeling that the HUBBLE project opened a concrete and practical way for researchers to consider ethics more seriously.

As a side effect for some HUBBLE participants, this ethics perspective lead to new research works centered on ethics questions [START_REF] May | The Question of Ethics in Learning Analytics[END_REF]. These contributions and external ones [START_REF] Sclater | Learning analytics -A taxonomy of ethical, legal and logistical issues, v1.0[END_REF]- [START_REF] Pardo | Ethical and privacy principles for learning analytics[END_REF]where particularly welcome where and when many data are collected, as it can be the case in some of the HUBBLE case studies. Our first attempt on the questionnaire, inspired by the existing ones, in the medicine fields in particular, is a good illustration of how ethics should be handled in practice. Though, the document alone is not selfsufficient. As a matter of fact, it is important to have someone, better informed from the Ethics work package, that can help the data owners (leaders or representatives of the case study) to build trust and to ensure that ethics is not neglected. A preliminary conclusion the team could draw from this experience is that ethics has to be considered as a chance for more successful research results and not as a set of constraints [START_REF] Mille | Learning Analytics : vers une éthique par construction des EIAH[END_REF].

Table 1 .

 1 List of the case studies in HUBBLE in the first project loop

	Case Study Purpose	Nature of traces	Volume in
				2015
	Classcraft	Free online, educational role-play-ing game to play in the classroom and teach differently.	Logs, custom google analyt-ics, comments	75000 learn-ers, 2000 teachers
	LIF3	Computer science classes for under-graduates in Lyon	Exercises models, content & answers + diagnosis	100 students/
	Met-toi-à-	Serious game in Lyon on Nutrition	Activity traces, timestamp, re-	56 students
	table	(biology), history and geography.	sources	age 14-15
				2500 learners.
	MOOCAZ	MOOC to learn how to make a MOOC (from A to Z).	Form + Logs: Timestamp, ID for video, quiz, forum	5 weeks, 5 hours of video, +
				homework
	Openclass-rooms	MOOCS on Web development: ac-tivities (alone or in groups): data Video, Text, Quiz	Logs, exercises, forums, self-reported data	14000 learn-ers (per month)
	PACES	Blended learning for first year of Health studies in Grenoble	Questions asked by students, MCQ answers, Final results.	1700 students, 20 teachers
	Tactileo-map	A geomedia serious game: (learners go in the field, with tablets)	timestamp, geo-localized ac-tions and products	200 learners
	Tama-gocours	Serious game for learning about copyrights and teaching exceptions	chat (manually coded) group id, action id, resource + logs: timestamp, user id,	200 learners
	TESTL1	Tests in sciences and technologies to assess the level of student enter-ing Grenoble University	Logs, answers	1000 students 10 teachers