Antoine Perquin

Betty Fabre

Cédric Fayet

David Guennec

Lily Wadoux

Clémence Metz

Soumayeh Jafaraye

Meysam Shamsi

Angelique Le Pennec

Joëlle Thepault My

Aditya Arie Nugraha

Arseniy Gorin

Anastasiia Tsukanova

Emilie Doré

Ilef Gaêlle Vidal

Ben Farhat

Imran Sheikh

Raheel Qader

Sunit Sivasankara

Sébastien Lemeguer

Motaz Saad

Manuel Sam Ribeiro

Marie Tahon

Great thanks to the CSTR team at the University of Edinburgh. I would like to address a

List of Figures

5.1

This figure illustrates the workflow guiding the rule-based approach. After the phonetization and forced alignment of the chapter text with the corresponding audio file, the data are segmented into paragraphs/ pseudoparagraphs and stored relying on roots toolkit. The segments follow two annotations process: (i) the manual annotation made by an expert (ii) the automatic annotation which has two phases, the first phase consists of labeling the segments according to typographic criteria as DD, ID, and mixed group. The mixed groups are processed in phase 2 ( Figure 5.2) in order to fine-tune the annotation and label the group according to Direct Discourse (DD), Indirect Discourse (ID), and Incidental Clauses with reporting verbs (IC). The mixed groups annotated, and non-mixed groups form the automatic annotation sequence. The two annotations (manual and automatic) are fused to form the Annotated Corpus (AC). . . . . . . . . . [START_REF] Fayet | FlexEval, creation of light websites for multimedia perceptual test campaigns[END_REF]) used for collecting the subjective assessment of the participants.

Question: asked question was: " For each sample, evaluate how similar it is to the reference (0 completely different, 100 completely similar)" . . . . . . 154 E.1 Block diagram of Tacotran-2 [START_REF] Shen | Natural tts synthesis by conditioning wavenet on mel spectrogram predictions[END_REF][START_REF] Oord | Wavenet: A generative model for raw audio[END_REF] Wrd is the number of words in the chapter and Nbr. Syl the number of syllables. The recording type (P) refers to a professional recording, whereas (A) refers to an amateur record.The Siwis French Speech (SFS) voice is the female voice of The SIWIS French Speech Synthesis Database. PODALYDES is a male voice. The speakers FFR0001, FFR0011, FFR0020, and MFR0019 are included in the MUltispeaker French Audiobooks corpus dedicated to expressive read Speech Analysis (MUFASA) corpus. . . . . . . . . . . . . 33

3.6 Subharmonic-to-Harmonic Ratio distribution of the subcorpus speakers . For each speaker, we select all the voiced frames and calculate the Subharmonicto-Harmonic Ratio frequency distribution. . . . . . . . . . . . . . . . . . . 35 3.7 The frequency of the {/ka/,/ta/,/pa/,/ti/,/ti/,/pi/} in the considered dataset, that have been manually annotated in terms of pitch amplitude. . 36

Synthèse en Français 1 Introduction

Pour obtenir une voix de synthèse de qualitée utilisable dans des contextes particuliers, il est fondamental d'améliorer l'expressivité de la parole car elle transmet les émotions, les intentions et les états d'esprit des locuteurs. Une part importante de l'expressivité de la voix est liée au contexte d'élocution et notamment influencée par le type de texte lu. Tous ces éléments participent à ce que l'on peut nommer des styles de parole. Les poèmes, les contes, les discours politiques ou les journaux télévisés sont des textes dont l'oralisation se fait selon des styles différents; et de nombreux lecteurs, s'ils lisent des documents intégrant plusieurs types de textes ou style de parole, sont capables d'adapter leur élocution aux textes à oraliser. La caractérisation des styles de parole à partir de l'étude de différents paramètres prosodiques (rythme, intonation, etc.) et segmentaux (réalisation des segments, liaisons, etc.) est une étape fondamentale. Les résultats de ces analyses serviront de base à la construction de modèles permettant aux systèmes de synthèse de générer des styles de parole divers. L'objectif est d'améliorer le contrôle et le rendu expressif des systèmes de synthèse de la parole.

Le traitement de l'expressivité dans la parole et l'adaptation de la prosodie à des styles particuliers constituent des questions de recherche importantes à l'heure actuelle. Des études très récentes comme [START_REF] Govind | Expressive speech synthesis: a review[END_REF], mettent en avant le manque de naturel et de qualité dans la parole synthétique expressive. Concernant les styles de parole, [START_REF] Obin | MeLos: Analysis and Modelling of Speech Prosody and Speaking Style[END_REF]] propose un modèle permettant la génération de quelques genres de discours. Dans [START_REF] Avanzi | Towards the adaptation of prosodic models for expressive text-to-speech synthesis[END_REF]] sont présentés quelques résultats d'une étude récente visant à déterminer les principaux éléments caractéristiques de quelques genres en vue de les re-synthétiser. Les changements de styles, comme lors du passage au style direct et l'expression par la parole de certaines émotions sont au coeur des travaux réalisés. Pour cela, dans un premier temps on s'intéressera à l'expression portée par des livres audio car ces types de textes permettent de regrouper certaines de ces caractéristiques.

Dans cette thèse, nous avons exploré l'expressivité de la parole à travers des livres audio. Afin de développer des modèles autorisant un meilleur contrôle de l'expressivité en synthèse de parole, ou d'adapter la prononciation et la prosodie au type de discours (changement dans la perspective du discours, style direct/indirect, etc.), nous avons construit deux corpus de livres audio français complémentaires SynPaFlex-Corpus et MUFASA.

Construction de corpus

SynPaFlex-Corpus [START_REF] Sini | Annotation automatique des types de discours dans des livres audio en vue d'une oralisation par un système de synthèse[END_REF]] est un corpus de livres-audios en français composé de 87 heures de parole de bonne qualité, enregistré par une unique locutrice. Il est constitué d'un ensemble de livres de différents genres. Ce corpus diffère des corpus existants, constitués généralement de quelques heures de parole mono-genre et multi-locuteurs. La motivation principale pour construire un tel corpus est l'exploration de l'expressivité à travers différents points de vue, tels que le style de discours, la prosodie, la prononciation, et en utilisant différents niveaux d'analyse (syllabe, mot prosodique ou lexical, groupe syntaxique ou prosodique, phrase, paragraphe). Le corpus a été annoté automatiquement et fournit des informations telles que les labels et frontières de phones, les syllabes, les mots et les étiquettes morpho-syntaxiques. Pour pouvoir étudier les différentes stratégies de lecture adaptées par différents locuteurs nous avons construit MUFASA qui comprend une vingtaine de locuteurs français et contient environ 600 heures de parole de bonne qualité. Dans le chapitre 3, nous avons montré que sur des données comparables, les enregistrements amateurs1 et professionnels2 présentent des similitudes en matière de propriétés phonétiques et prosodiques. En revanche la qualitée de la parole du corpus MUFASA est légèrement inférieure à celle des corpus professionnels, ceci est dû notamment aux conditions d'enregistrement, néanmoins la quantité et la diversité des données permettent d'explorer de nouveaux horizons de la parole expressive lue et de développer des systèmes de synthèse de la parole plus performants. Ensuite, nous avons mené une expérience dans le but de comparer des extraits du corpus MUFASA avec d'autres corpus français bien connus pour mesurer la similitude d'un point de vue phonétique. Comme nous y attendions, MUFASA présente de grande similitude avec le corpus BREF [START_REF] Larnel | BREF, a large vocabulary spoken corpus for French[END_REF], qui est également un corpus de parole lue. Pour aborder l'expressivité portée par les corpus que nous avons construits, nous avons proposé d'articuler ce travail de thèse sur trois thématiques:

• Les émotions interviennent à des moments précis du discours pour animer le discours et lui donner de la profondeur.

• Pour structurer et apporter une cohérence à l'histoire, les auteurs utilisent différents modes de discours.

• Dans les livres audio, les émotions et les discours dépendent du texte autant que du signal de la parole. Le signal de parole dépend des propriétés du locuteur, qui constitue la troisième thématique abordée dans ce travail de thèse.

Étude émotionnelle de corpus SynPaFlex

Pour étudier les caractéristiques émotionnelles des données, nous concentrons nos efforts sur la voix présente dans le corpus SynPaFlex. Pour mener les expériences, une part significative du corpus a été annoté manuellement pour encoder le style direct/indirect et des informations d'ordre émotionnel.

Pour ce faire, nous avons demandé à un annotateur expert en parole de sélectionner un extrait représentatif et d'annoter le signal de parole. L'annotation manuelle a fournit quatre transcriptions complémentaires:

• La transcription de contour intonatif: cette annotation s'appuie sur les travaux de [START_REF] Delattre | Les dix intonations de base du français[END_REF]]. De cette annotation huit patrons intonatifs principaux sont encodés: question (interrogative), note, nuance, suspense, résolution (autoritaire ou impérative), chant, et nopip (aucun patron intonatif particulier).

• La transcription du discours des personnages3 impliqués dans les livres sélectionnés en attribuant un identifiant unique et une identité vocale en tenant compte des performances du locuteur.

• Pour étiqueter le signal de parole en ce qui concerne les émotions, l'approche catégorique des émotions a été adoptée car c'est celle qui est la plus répandue actuellement. Six émotions de base définies par [START_REF] Ekman | Basic Emotions[END_REF]] sont utilisées : colère, joie, tristesse, surprise, dégoût et peur. Deux étiquettes supplémentaires ont été ajoutées : menace et ironie.

• Transcription de phénomènes complémentaires aux patrons intonatifs et émotifs contenant des événements phonétiques et linguistiques tels que les bruits, césure (notamment liaison sans enchaînement), murmuré ou mi-voisé, langue étrangère, paraverbal et musique.

A l'issue du processus de l'annotation manuelle, nous avons voulu reproduire l'étiquetage émotif à l'aide de techniques d'apprentissages automatiques et des procédures bien établies dans la reconnaissance automatique des émotions. Pour ce faire, nous nous sommes appuyés sur la méthodologie proposée par [START_REF] Schuller | The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism[END_REF]] présentée dans le challenge paralinguistique de 2013. Cette méthodologie s'appuie sur des techniques d'apprentissage supervisées pour construire un modèle de prédiction et d'étiquetage automatique en émotions de segment de parole.

Cette technique comporte deux étapes :

1. étape d'apprentissage; le modèle est entraîné avec des données dont on connaît la réalité du terrain, à l'issue de cette étape un modèle est appris.

2. étape de test; il s'agit de tester le modèle appris lors de l'étape d'apprentissage et d'évaluer le modèle.

Pour éviter le sur-apprentissages 4 , il est d'usage de faire recours à la technique de validation croisée. Cette méthode a été mise en oeuvre en utilisant le sous corpus SynPaFlex manuellement annoté, les résultats des expériences ont mis en évidence la subtilité des émotions dans ce type de données. Sur la base de ce constat, nous avons proposé d'explorer les questions relatives aux émotions par l'analyse des propriétés lexicales et sémantiques des transcriptions de livres audio. Pour réaliser ces expériences, nous avons privilégié les approches non supervisées. Cette seconde expérience est basée sur les techniques d'analyse des sentiments et de traitement du langage naturel. Le processus consiste principalement à trouver une représentation numérique adéquate des textes. Pour ce faire, nous avons choisi le modèle doc2vec pour la numérisation des phrases issue du texte, puis une méthode regroupant automatiquement le texte intégré selon des affinités lexico-sémantiques en utilisant l'algorithme de K-moyennes. Une fois les clusters formés, la dernière étape consiste à interpréter les clusters dans l'espace acoustique. Les résultats montrent qu'il existe une forte corrélation entre la représentation du texte et les caractéristiques acoustiques de la parole.

L'annotation émotionnelle présentée et étudiée est fortement dépendante des propriétés du vocalique et du style du locuteur.

Étude discursif des livres audio

Pour étudier le discours, nous avons d'abord construit un outil d'analyse et d'annotation des livres audio des textes considérant trois types de discours, à savoir le discours indirect, le discours direct et les incises de citation. Cet outil comprend deux approches. La première approche est basée sur des règles, qui consiste en un ensemble de règles dérivées de l'analyse des données et élaborées par des experts à l'aide des propriétés morpho-syntaxiques et typographiques du texte. Le second s'appuie sur des techniques d'apprentissage automatique; nous avons obtenu de meilleurs résultats avec les modèles d'apprentissage automatique et plus précisément les modèles réseaux de neurones récurrents. Pour mettre en évidence les propriétés prosodiques lors des changements de discours et comment les locuteurs gèrent les perspectives de changements discursif et de personnages. Nous avons proposé d'analyser ce phénomène à travers un ensemble d'indices prosodiques dérivés de l'Unité InterPausale (UIP) que nous considérons comme pertinents pour mesurer et étudier le discours. Nous avons expérimenté avec deux locutrices du corpus MUFASA lisant un seul et même texte. Les résultats confirment que les deux locuteurs marquent bien le changement de discours et que l'UIP est unité de parole adéquate pour l'étude des changements de discours; le registre (F0-range) et la durée de l'inter pause sont des indicateurs pertinents pour les changements discursifs.

Identité prosodique d'un locuteur dans un système de synthèse vocale multilocuteurs

Pour étudier les propriétés des locuteurs et l'impact de leurs styles d'élocution dans un système de synthèse vocale, il est important d'avoir une représentation couvrant les propriétés du locuteur indépendant du texte. Dans la littérature l'identité vocalique d'un locuteur donnée est souvent représenté selon des méta-information sous-forme d'encodage one-hot, qui fait souvent référence au genre et l'identité unique du locuteur, D'autres approches, consiste à dériver une représentation unique au locuteur à partir de carac-téristiques acoustiques. X-vector est un exemple de représentation de locuteur à partir d'information acoustique, ce plongement de vecteur acoustique est dérivé d'un modèle de reconnaissance du locuteur pré-entraîné. Dans ce travail nous proposons d'avoir une nouvelle représentation du locuteur, intégrant cette fois-ci des caractéristiques prosodiques. Cette nouvelle représentation est dénommée P-Vecteur (P pour prosodique).

Pour évaluer et comparer ces trois configurations, nous avons mis en place un système de synthèse vocale multi-locuteurs basé sur des réseaux de neurones profond intégrant en entrée une des configurations représentant l'identité du locuteur et les informations linguistiques extraites à partir du texte. Pour évaluer ces différents systèmes de synthèses, nous avons effectué deux évaluations objectives : l'évaluation objective standard qui consiste à comparer les paramètres acoustique prédit et réel à l'aide de métrique adéquate, et l'évaluation objective visuelle, qui consist à projeter les sorties de la première couche cachée du réseau de neurone. En outre, nous avons mené une campagne d'évaluation subjective auprès de 30 natifs français. L'évaluation objective et subjective a montré que l'identité prosodique du vecteur P est capable de guider le système de synthèse vocale multilocuteur basé sur le DNN aussi bien que le vecteur X et le vecteur OneHot bien établis.

Perspectives

Perspective à court terme

Comme perspective, nous souhaitons reproduire le même schéma d'intégration et d'évaluation adapté afin d'étudier l'identité prosodique du locuteur pour les informations émotionnelles et les indices prosodiques liés au discours, qui sont encore au stade de l'analyse statistique et de l'évaluation objective. Ainsi, dans une perspective à court terme, nous visons à intégrer ces deux variables dans le cadre de la boîte à outils MERLIN [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF] en nous appuyant sur la même procédure présentée dans [START_REF] Malisz | Controlling Prominence Realisation in Parametric DNN-Based Speech Synthesis[END_REF]]. Concrètement, nous souhaitons insérer deux nouveaux modules basés sur les réseaux de neurones, l'un pour la construction d'un vecteur intégré de discours et l'autre pour un vecteur intégré d'émotion (EEV). Les deux modules seront insérés entre le module frontal, et les modules de durée et acoustique.

Pour évaluer les effets de ces deux modules (émotionnel et discursif), nous considérons deux modules subjectifs distincts: une évaluation pour chaque module. Pour l'évaluation du discours perceptuel, les stimuli seront des extraits issues du changement de mode de discours (DD, ID, IC), deux questions sont prévues:

• Une question directe qui peut se formuler comme suite "remarquez-vous des changements dans l'échantillon de discours ? (oui/non), pour voir si le sujet a remarqué des changements;

• Une deuxième question "Quel type de changements percevez-vous ?

1. la vitesse de la parole "rapide/lente" 2. l'amplitude de la parole 3. la durée de la pause plus courte/longue.

Un changement similaire Un processus d'évaluation sera mené pour évaluer l'impact du module émotionnel. Les stimuli seront les mêmes que ceux utilisés pour l'évaluation du module de discours, mais les questions ne seront pas les mêmes. Comme dans cette deuxième évaluation subjective, les questions seront "est-ce que vous reconnaissez une émotion dans cet échantillon de parole?" Si le sujet répond oui, une liste d'émotions sera présentée, suivie de l'intensité de l'émotion ou des émotions perçues car on suppose que le sujet peut attribuer pour un même échantillon plusieurs étiquettes d'émotion avec une intensité différente. Au-delà de l'analyse des résultats de l'effet respectif de chaque module, la combinaison des résultats est également considérée comme une perspective car elle permet de mesurer la corrélation entre le discours et l'émotion. s

Perspective à long terme

Dans une perspective concrète à long terme, nous prévoyons de changer d'environnement de développement passant ainsi de Merlin [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF] à un cadre bout-à-bout, plus précisément au Tacotran2 [START_REF] Wang | Uncovering latent style factors for expressive speech synthesis[END_REF][START_REF] Shen | Natural tts synthesis by conditioning wavenet on mel spectrogram predictions[END_REF]] disponible dans la boîte à outils ESPNET [START_REF] Hayashi | Espnet-tts: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit[END_REF], pour obtenir une meilleure qualité de synthèse. Ensuite, nous envisageons de construire un module similaire à celui qui a été développé dans des perspectives à court terme. Toutefois, dans cette nouvelle configuration, nous fusionnerons les deux modules en un module unique reposant sur des réseaux neuronaux multitâches profonds [START_REF] Liu | Multi-task deep neural networks for natural language understanding[END_REF]. Ce module sera formé en même temps que les modèles acoustiques.

Discussion générale

Dans cette thèse, nous avons abordé les caractéristiques prosodiques dans le cas de la synthèse de livres audio à travers trois dimensions :

• Émotions : l'intervention d'un orateur pour situer le contexte de l'histoire et fournir des éléments supplémentaires pour divertir l'attention de l'auditeur.

• La typographie du discours pour mettre en évidence les structures des textes et la corrélation avec des indices prosodiques.

• L'identité de locuteur avec pour objectif à long terme de mettre en évidence la stratégie de lecture.

Une lecture expressive se doit de respecter des contraintes syntaxiques, sémantiques, pragmatiques ainsi que la typologie du texte écrit. À cela s'ajoute la stratégie du locuteur lier au contrainte identitaire ainsi que la réalisation des émotions. La corrélation entre les trois paramètres explorés dans cette thèse rend difficile la mise en place d'un système de synthèse vocale expressif robuste et fiable. Démêler ces "trois paramètre" en utilisant des techniques de factorisation basées sur des algorithmes avancés d'apprentissage profond semble être intéressant, selon [Hsu et al. 2019;[START_REF] Mathieu | Disentangling factors of variation in deep representation using adversarial training[END_REF].

Alors que [START_REF] Brognaux | Expressive speech synthesis : research and system design with hidden Markov models[END_REF]] explore l'expressivité à travers la parole spontanée, nous nous concentrons sur la lecture de textes écrits. Il sera intéressant de faire une comparaison entre parole spontanée, en particulier les commentaires sportifs et les textes lus, notamment les livres audio, pour trouver une représentation commune à la parole expressive. Les principaux résultats présentés dans cette thèse sont basés sur une perspective acoustique de la parole. Ce niveau de représentation de la prosodie est important mais pas suffisant pour caractériser le discours expressif porté par les livres audio. La représentation perceptive et linguistique de la prosodie est cruciale pour avoir une vision complète et pour valider les résultats présentés dans cette thèse.

Introduction General Context

An expressive voice is centered on the listener; it aims to communicate precise information, particular emotions, to relate facts or events. This expressivity is achieved through a vocalic gesture with its intonational modifications of tone, pitch, timbre, the repetition of certain phonemes, the lengthening of other phonemes. These phonetic events allows encoding expressive units by taking into acount cultural habits, and they could be acommpanied by facial mimics or certains body gesture. It is possible for a non-deaf human being to become aware of the expressiveness of the message conveyed solely by the voice. A simple audio recording can be rich enough to capture an entire scene or event. Audiobooks are a concrete example of the ability of an expressive voice to transcribe and convey emotions, the interactions between characters through dialogue, the narration of events, the description of places, and the account of time and space in which the literary work is set. This thesis project aims to characterize the expressivity conveyed by audiobooks to improve speech synthesis systems. Text-to-Speech (TTS) systems aim to supply machines with expressiveness to facilitate human-machine interaction.

Expressive Speech Synthesis

Nowadays, speech synthesis from a text can achieve outstanding levels of quality. The use of large corpora of speech has mostly contributed to this success. Nevertheless, synthetic speech still lacks emotion, intention and style. At present, we are not able to synthesize a voice with the expressiveness needed for audiobook reading without recording a speaker to create a large corpus with this style. Some works in the literature are interested in taking into account phenomena related to expressivity and bring interesting conclusions that partly allow us to characterize the functioning and materialization of these phenomena. Here, we intend to deal jointly with emotion, intention, and style of speech, since these notions are very closely linked in practice. Our goal is really to integrate them into speech synthesis.

SynPaFlex-Project

The SynPaFlex project5 mainly funded this thesis. The objective of the SynPaFlex project is to investigate the different characteristics that contribute to the expressiveness of a voice in order to build a prosody model and a pronunciation model adapted to one or several speakers. The use of these models will be explored in order to integrate expressivity into speech synthesis systems, notably through concatenation or parametric statistical models. The research work focuses on the French language.

The main challenges of the project lie in the feasibility of applications of expressive speech synthesis, applications which are still not very widespread at the moment. In particular, opportunities are to be expected in the field of video games (diversification of synthetic voices, creation of expressive voices adapted to the game situation), language learning (dictation, style of speech), and personal assistance.

Challenges

It is challenging to realize the expressiveness conveyed from a simple text. Information such as the position of the pauses and their duration according to the context, the intonation, the rhythm, and many other parameters are not encoded in the text. However, it is possible to derive this information by analyzing and modeling different prosodic descriptors responsible for a natural voice.

These prosodic descriptors vary according to the context and depend on the text to be read. For instance, poems cannot be read as a simple message. That is where the style of speech comes into play.

Texts in audiobooks have special properties compared to other spoken texts, because the texts are longer, carry the author's style, his intention, and each sentence has a particular context.

In addition to this, some parameters are speaker-dependent as based on the reading strategy of a given speaker. Furthermore, it is not easy to evaluate the quality or judge the strategy of a speaker.

Document organization

Automatic characterization of prosodic descriptors responsible for expressiveness of the voice, and which can be integrated into a text-to-speech system, is still a challenge. This process requires several steps that will be addressed in this manuscript.

Chapter 1 and Chapter 2 are dedicated to state of the art of text-to-speech synthesis systems, and speech prosody modeling, respectively. The collection of audio data in sufficient quantity to highlight the properties of audiobooks, will be discussed in Chapter 3. Manual annotation as well as a quantitative study of certain aspects of the expressivity of audiobooks will be reported in Chapter 4. Chapter 5 will discuss automatic annotation of speech types in audiobooks, followed by a prosodic study of discourse changes, dialogues, and discourse markers in Chapter 6. Chapter 7 deals with the prosodic identity of a speaker in multi-speaker synthesis systems. The manuscript concludes with a general conclusion where we summarize the main contributions of this thesis as well as further issues and the perspectives in the future work.

Chapter 1

Text-to-Speech Synthesis

This chapter provides a general background of the field of TTS Synthesis, and It mainly focuses on issues relevant for this thesis. It is divided into three sections. In the first section, we give an overview of TTS systems, providing a generic technical background.In the second section, we focus on the parametric speech synthesis used in this thesis, including the evaluation methodology. And finally we present the main challenges of expressive speech synthesis.

Text-To-Speech Synthesis System

Speech synthesis systems aim to generate speech from a text. This process of encoding text into speech follows a path that is organized in most cases into modules. In the majority of systems, there are two main modules, back-end, front-end.

Front-End

This first module receives the raw text as input and furnishes an output vector of the linguistic and prosodic specification.

Text Processing and decoding

In most systems, this module is used for tokenizing the raw text. Through a normalization module, each token is converted to an orthographic form, numbers, acronyms and so on being thus replaced. This stage is crucial for the following modules, it furnishes a normalized text.

Phonetic and Phonological

The sequence of words of the normalized text is phonetized either with a rule-based model or a statistical based model so-called LETTER-TO-SOUND (LTS). From the phoneme sequence, syllabification is processed, and pronunciation rules are applied to the chain according to language specification.

•

Phonetization, Grapheme to Phoneme (G2P) [START_REF] Novak | Phonetisaurus: Exploring grapheme-to-phoneme conversion with joint n-gram models in the WFST framework[END_REF] Syllabification in French [START_REF] Swaileh | Un modèle syllabique pour la reconnaissance de l'écriture[END_REF] we can destinc Pronouciation consiste of the way that

Prosody

The most challenging sub-module in the front-end module is the prediction of prosody.

According to [START_REF] Taylor | Text-to-speech synthesis[END_REF]], prosody involves three phenomena:

-Accentuation or stressing is the act of emphasizing a particular speech sequence in the majority of languages, and this prominence appears at the syllable level. Prominences may assume functions such as emphases, stylistic variation. This phenomenon is language-dependent. For instance, there is no phonetic stress but only phrase stress in French, the prominence is on the last syllable of a word, unlike in English where there is lexical stress.

-Phrasing refers to the division of the speech flow into chuncks of different ranks.

-Intonation is the shape of the pitch contour at the sentence/phrasal level (to distinguish with lexical tones in tonal languages)

In [START_REF] Avanzi | La prosodie du français: accentuation et phrasé[END_REF][START_REF] Avanzi | Notes de recheche sur l'accentuation et le phrasé prosodique à la lumière des corpus de français[END_REF]], the authors have investigated the relation between accentuation and phrasing and the relation between intonation and rhythm was compared in the case of British English and French in [START_REF] Patel | Comparing the rhythm and melody of speech and music: The case of British English and French[END_REF].

Despite the fact that speech synthesis is now developed withing Deep Learning paradigms, prosody is still a research area in comparison to text processing and decoding modules.

For more details about prosody see Chapter 2.

Back-End

This module converts the intermediate linguistic specification into a synthetic speech waveform. In literature, this block is called the waveform generator. In most cases, this block is not language-dependent since the front-end has done most of the linguistic processing. Many approaches have been proposed to map the linguistic representation to speech.

Rule-based synthesis

This method is one of the oldest paradigms proposed for generating artificial speech. These approaches typically define a set of rules to artificially generate a waveform from a set of acoustic parameters such as formant frequencies [START_REF] Klatt | Software for a cascade/parallel formant synthesizer[END_REF]]. The main advantage of this paradigm is that it offers a certain control over the synthesized speech. For instance, this paradigm has been used for synthesizing emotional speech [START_REF] Schröder | Emotional speech synthesis: A review[END_REF]]. This technique has two main disadvantages. The first one is due to the fact that systems are based on a set of rules, which are language-specific and typically require knowledge of experts, and the second one is the lack of naturalness and intelligibility.

Concatenative synthesis

This approach is one of the most common technique for waveform generation in industrial systems. This approach consists in concatenating pre-recorded units of speech to generate new waveforms. However, systems aiming for more generic synthesis focus on the concatenation of smaller units. Such units may begin, for example, at the mid-point of a phone and end at the mid-point of the following phone, thus capturing the co-articulation between the two phones. These units are diphones. Systems using a minimal database with a single diphone sample are said to be diphone synthesizers [START_REF] Moulines | Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones[END_REF]. Extensions of this idea vary the type and number of units present in the database. This generalization is referred to as unit selection [START_REF] Hunt | Unit selection in a concatenative speech synthesis system using a large speech database[END_REF][START_REF] Guennec | Study of unit selection text-to-speech synthesis algorithms[END_REF][START_REF] Alain | The IRISA Text-To-Speech System for the Blizzard Challenge 2016[END_REF][START_REF] Lolive | The IRISA text-to-speech system for the Blizzard challenge 2017[END_REF]. Unit selection systems use a very large database with multiple samples of the same unit. The task of the waveform generator is then to select the optimal unit sequence given an input linguistic specification. It is hard to control the generated speech, as these techniques are not very flexible.

Parametric synthesis

This class uses parametric representations of speech waveforms, which are modeled via statistical frameworks. For this reason, these systems are often grouped under the term Statistical Parametric Speech Synthesis (SPSS) [START_REF] Ze | Statistical parametric speech synthesis using deep neural networks[END_REF]. Parametric systems offer several advantages over concatenative systems. For example, it is easy to see how unit selection systems can be limited by their database: larger databases allow the system to be more flexible, but also increase the number of resources needed. Parametric voices are flexible when it comes to the manipulation and control of acoustic parameters. This flexibility makes them attractive for various tasks such as speaker adaptation, multispeaker speech, multilingual systems, voice conversion, and expressive speech. Additionally, parametric systems tend to benefit from a very small footprint when compared to standard unit selection systems. However, parametric voices suffer from various disadvantages.

Hybrid synthesis

This approach represent a class of techniques that combine unit selection and parametric methodologies. The most common hybrid approach [START_REF] Tiomkin | A hybrid text-to-speech system that combines concatenative and statistical synthesis units[END_REF]] uses a statistical framework to generate a sequence of acoustic parameters that are then used to guide the selection of units from the database.

Statistical Parametric Speech Synthesis

This thesis is mainly concerned with the statistical parametric approach for speech generation, more precisely on the Deep Neural Network based technique. Therefore, Section 2.1 will provide a further overview of this class of techniques.

Overview

In the context of speech synthesis, a vocoder, or voice encoder, extracts from the speech waveform a set of parameters that may be modeled statistically. Common approaches are based on the source-filter model of speech production. This model makes the assumption that speech is produced by first generating a source signal, which can be intuitively understood as air exiting the lungs and passing through the vocal folds. The positions of the vocal tract articulators (tongue, lips, oral, and nasal cavities) then act as a filter on the source signal. The source-filter model assumes that these two components are independents and vocoders aim to find representations that separate the effects of source and filter. Vocoders extract parameters over speech windows, referred to as a speech frame, and may, in common implementations, span 25ms. Each frame is assigned source (or excitation) parameters such as fundamental frequency and voicing information. Some vocoders include extra excitation parameters, such as band aperiodicities: this is the case of WORLD [START_REF] Morise | WORLD: a vocoderbased high-quality speech synthesis system for real-time applications[END_REF] and STRAIGHT [START_REF] Kawahara | Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds[END_REF] vocoders.

For the filter (or spectral envelope) parameters, Mel-cepstrum coefficients [START_REF] Fukada | Speech recognition based on acoustically derived segment units[END_REF]] are often used. Alternatively, one can use Mel-generalized cepstral coefficients [START_REF] Tokuda | Mel-generalized cepstral analysis-a unified approach to speech spectral estimation[END_REF] or line spectral pairs [START_REF] Itakura | Line spectrum representation of linear predictor coefficients of speech signals[END_REF]]. The speech waveform can be analyzed and reconstructed with minimal error via these speech parameters.

Recent approaches using neural networks for SPSS aim to overcome the Hidden Markov Model (HMM) based speech synthesis systems. There has been a considerable amount of earlier work using neural networks for speech synthesis [Wang et al. 2017a;[START_REF] Ping | Deep voice 3: Scaling text-to-speech with convolutional sequence learning[END_REF][START_REF] Tachibana | Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention[END_REF]. However, recent improvements in software, hardware, and data availability have caused a huge interest in these methods.

In this thesis, we use a framework such as the one described in [START_REF] Ze | Statistical parametric speech synthesis using deep neural networks[END_REF]. This method was implemented in the Merlin Neural Network Toolkit [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF]. During the data preparation stage, we have used the JTrans [START_REF] Cerisara | JTrans, an opensource software for semi-automatic text-to-speech alignment[END_REF] software to force align the data at the phone-level, from which phone alignment can be inferred. Given this alignment between linguistic features and acoustic parameters, a Feed-Forward DNN (FF-DNN), called the acoustic model, can be trained using mini-batch Stochastic Gradient Descent (SGD). An additional neural network, called the duration model may be trained in a similar fashion to model phone durations.

Finally, a vocoder is used to synthesize the waveform. This framework is used for DNN-based speech synthesis.

Evaluation

Two metrics are used to evaluate a synthetic speech system: objective and subjective assessments. The objective evaluation consists of measuring the acoustic and duration distance between the synthetic speech and natural one relying on different metrics, calculating the recognition rate using a speech recognition technique. These measures are not reliable for measuring the intelligibility and naturalness of synthetic speech. For this reason, a subjective assessment is somehow mandatory for validating and reinforcing objective assessment.

Objective Evaluation

In statistical parametric speech synthesis, objective evaluations compare a sequence of acoustic parameters generated from a model with a reference sequence extracted from a waveform. Most objective metrics are distance measures between the two sequences. The underlying assumption is that the distance between the sequences is meaningful in terms of the quality of the model. That is, the smaller the distance between generated and reference parameters, the better the model. However, it is not always the case that objective measures are representative of the quality of the acoustic parameters. Averaging over datasets might dilute otherwise perceptible acoustic differences between systems. Objective measures can still be useful as they are fairly easy to compute and they facilitate comparisons over a large number of systems.

In this section, we give a brief overview of the main measures used in this work. When appropriate, these are computed according to the Merlin Neural Network Toolkit [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF] and the equations presented here reflect that implementation. Note that during this thesis we keep the default configuration proposed by the framework.

Objective metrics are sensitive to the vocoder used. In this thesis, we use [START_REF] Morise | WORLD: a vocoderbased high-quality speech synthesis system for real-time applications[END_REF] and these measures are computed accordingly. Mel-Cepstral Distortion (MCD) measures the distance between two sequences of Mel-Frequency Cepstrum Coefficient (MFCC). We are given a reference vector x and a generated vector x of MFCC coefficients. MCD is then computed as an extension of the standard Euclidean distance:

M CD = α T D d=1 (x d (t) -xd ) 2 α = 10 √ 2 ln 10 with : α = 10 √ 2 ln 10
where T is the total number of frames in the data set and D is the dimensionality of the MFCC extracted at each frame. In this thesis, we use 60 coefficients per speech frame. Following [START_REF] Kominek | Synthesizer voice quality of new languages calibrated with mean mel cepstral distortion[END_REF], the constant α is included for historical reasons. Note that we exclude the first coefficient, commonly associated with the energy of a speech frame. This prevents the distance measure from being influenced by loudness, which may affect some datasets, such as non-professional audiobooks [START_REF] Kominek | Synthesizer voice quality of new languages calibrated with mean mel cepstral distortion[END_REF].

Band Aperiodicity Parameter (BAP) distortion follows the same intuition (and notation) as MCD. For each frame, a D-dimensional vector of parameters is extracted to represent the source excitation signal. In this thesis, we extract 25 band aperiodicities and we compute the distortion between natural and predicted parameters.

BAP = 1 10T T t=1 D d=1 (x d (t) -xd ) 2
In terms of objective measures related to the f0 signal, we have used the root-meansquare error and Pearson's product-moment correlation. These are standard measures in the literature, although alternatives have been suggested [START_REF] Clark | Objective methods for evaluating synthetic intonation[END_REF]. For the purpose of this thesis, these measures are computed at utterance-level on voiced-frames only and the average of all utterances in the test set is reported.

For a given utterance u, the root-mean-square error of the f0 signal is determined as

RM SE u = 1 N N n=1 (x u (n) -xu (n)) 2 (1.1) RM SE = 1 U U u=1 RM SE u (1.2)
Similarly, the correlation of the f0 signal is determined as:

r u = N n=1 (x u (n) -x u )(x u (n) -xu ) N n=1 (x u (n) -x u ) 2 N n=1 xu (n) -xu ) 2 (1.3) CORR = 1 U U u=1 r u (1.4)
Where x u u andx u u denote the mean value of the reference and the generated f 0 signal for utterance u, respectively. Note that the f0 correlation is here implemented as Pearson's product-moment correlation coefficient. Intuitively, this measure captures the similarity between the overall shape of generated and reference f 0 signals, which is particularly relevant for intonation. While distance-based objective measures aim to be minimized, the signal's correlation aims to be maximized. Finally, in some chapters of this thesis, voicing error is reported as the percentage of frames that were assigned the incorrect voicing label.

Subjective Evaluation

Objective evaluation methodologies are often used as an indication of the quality of synthetic speech, especially when a large number of systems are being developed, and reference acoustic parameters are available. However, it is widely agreed that subjective listening tests still remain the standard method for the evaluation of synthetic speech.

The subjective evaluation of synthetic speech is not a simple task and still quite challenging. The majority of evaluations of systems focus on naturalness and intelligibility. In recent years, with the developments of speaker adaptation, multi-speaker modelling, and voice conversion techniques, speaker similarity has been adopted as a third dimension in the evaluation of speech synthesis systems.

Subjective evaluation methods are able to provide more accurate quality measurements than objective evaluation methods, but they also tend to be costly. Listening tests typically require a large investment in terms of time and resources, as they require well-designed experiments and listeners.

When we design perceptual listening tests several factors should also be considered for the evaluation of synthetic speech. For instance, the type of test, the question being asked, or the type and number of listeners. We briefly provide a review of well-established protocols for the evaluation of naturalness, and some methods used for the evaluation of intelligibility and comprehension.

In [START_REF] De | Suprasegmental representations for the modeling of fundamental frequency in statistical parametric speech synthesis[END_REF], the author grouped the protocols for the evaluation of naturalness into two main classes:

-Referenced methods, in which a synthetic sample is judged against an available natural reference.

-Non-referenced methods, in which synthetic samples do not have an available reference and are instead judged against the listener's expectations.

In both cases, naturalness means close to human voice properties with a perceptual point of view. There are many evaluation methods in literature, we will briefly give the most used for evaluating synthetic speech synthesis.

• Mean Opinion Score (MOS) is a non-referenced evaluation methodology [ITU-T and Recommend 1996]. Listeners are not given a speech reference to anchor their judgments. In a MOS evaluation, listeners are presented with one speech sample at a time. They are then asked to judge that sample on a 5-point scale in terms of quality, where 1 indicates bad and 5 indicates excellent. This methodology has variation such as DMOS (Differential MOS), which is a referenced version of the MOS test. Listeners provide their judgments for individual samples with respect to a reference sample. CMOS (Comparison MOS) presents the listeners with two randomized samples from different conditions.

• Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) [START_REF] Schoeffler | Towards the next generation of web-based experiments: A case study assessing basic audio quality following the ITU-R recommendation BS. 1534 (MUSHRA)[END_REF]: With this approach, listeners are presented with many conditions at once and they are asked to provide a subjective rank of the conditions with respect to each other and to an explicit reference. A copy of the explicit reference is hidden within the remaining experimental conditions, which fixes an upper bound for the listeners' judgments. In the MUSHRA paradigm, listeners provide absolute scores measuring the similarity of synthetic samples with respect to a reference. But because all conditions are rated simultaneously, multiple comparisons across conditions are also provided. This implicitly creates a ranking of systems, which might be preferable over an absolute score. Ranking scores can be interpreted as a preference judgment, while absolute scores can be interpreted as a measurement of that preference. In chapter 7, we use this methodology to evaluate a multi-speaker speech synthesis system.

• In the AB test, also called preference test, listeners are given a randomized pair of samples and are asked to express their preference with respect to some speech attributes. The most common question under this framework asks listeners to select the sample which sounds more natural.

A similarity evaluation aiming at understanding perceptual similarities across multiple systems may also be used [START_REF] Mayo | Multidimensional scaling of listener responses to synthetic speech[END_REF]. Similarity means the given condition and the reference are perceptually similar.

Intelligibility and comprehension evaluation: with this type of methodologies, we focus on the intelligibility and comprehension of synthetic speech as well as natural speech.

Expressive Speech Synthesis

In order to obtain a synthetic voice of good quality that we can use in particular contexts, it is fundamental to improve the expressiveness of speech, which is a vector of emotion, intention, or state of mind.

3.1 What do we mean by "expressive speech synthesis"? [START_REF] Brognaux | Expressive speech synthesis : research and system design with hidden Markov models[END_REF]] defines expressive speech as "any aspect of speech that makes it more natural-sounding and suited for a specific communicative situation other than reading nonemotional laboratory speech (that can be seen as a typical example of what we designate as 'neutral speech')." In this work, we were somehow extending or simplifying this description by considering that human speech is always expressive as long as it is carrying emotion, intention, or state of mind of the speaker. This aspect is manifested and leveraged by the speaker according to the context "circumstances," and speaking style. Furthermore, we consider that expressive speech demands less cognitive load1 from the listener's point of view; it makes the message among interlocutors easy in dialogue, for instance, and understandable when it is a political speech, for example. These aspects contribute to settling the definition of expressive speech as well as the primary motivation for building an expressive speech synthesis system.

For instance, in audiobooks, the reader is constrained by the text "circumstance or context". Consequently, the produced speech has to be expressive to keep the listener's attention and give access to the various information carried by the text read, including story plot and author intention.

Transversal questions

To address the expressivity of speech in audiobooks, we need to define the speaking style and the level of granularity that we consider to study this kind of data.

Speaking Style

A significant part of the expressiveness of the voice is related to the speech context and is influenced in particular by the type of text read. All of these elements contribute to what can be called speech styles. Poems, stories, political speeches, or television news are texts vocalized according to different styles. Many readers, if they read documents integrating several types of texts or style of speech, can adapt their speech to the texts to be verbalized.

The characterization of speech styles based on the study of different prosodic parameters (such as rhythm, intonation) and segmental parameters (such as the realization of segments, links) is a fundamental step. The results of these analyses will serve as a basis for building models that allows synthesis systems to generate speech styles. The goal is to improve the control and expressive rendering of speech synthesis systems.

The treatment of expressiveness in speech and the adaptation of prosody to particular styles are important research questions at present. Recent studies like [START_REF] Govind | Expressive speech synthesis: a review[END_REF][START_REF] Jauk | Unsupervised learning for expressive speech synthesis[END_REF], highlight the lack of naturalness and quality in expressive synthetic speech. In [START_REF] Avanzi | Towards the adaptation of prosodic models for expressive text-to-speech synthesis[END_REF]] are presented some results of a recent study which aimed at determining the main elements of characteristics of four genres, including a reading of fairy tales, dictations, political speeches, and reading of novels to re-synthesize them. Changes in style, such as changing to direct speech and the expression of certain emotions by speech, are at the heart of the work done. For that, at first, we will be interested in the expression carried by audiobooks because these types of texts allow covering some of these characteristics.

Beyond Sentence Level

To capture the style carried by a text, it is necessary to re-consider the linguistic unit used for building synthetic voice. Many information, such as expressivity and speaker attitude, can not be well characterized at a simple sentence level. For example, if we consider audiobook or dialogue, the sentence is not enough to describe the speaker strategy and to extract a consistent prosodic pattern for speech recognition or text-to-speech synthesis.

Studying long text book paragraphs can be an alternative to sentence. According to [START_REF] Farrus | Paragraph-based prosodic cues for speech synthesis applications[END_REF][START_REF] Lai | Automatic Paragraph Segmentation with Lexical and Prosodic Features[END_REF][START_REF] Doukhan | Synthèse de parole expressive au delà du niveau de la phrase: le cas du conte pour enfant: conception et analyse de corpus de contes pour la synthèse de parole expressive[END_REF], prosodic cues assigned to paragraphs seem to be more relevant to study expressivity and speaker's reading strategy in audiobooks, and the authors claim that the speakers tend to reset the prosodic cues between paragraphs. In [START_REF] Vaissière | Prosodic constituents in French: a data-driven approach[END_REF] the authors consider the paragraph as the largest unit defined by F0 fluctuation.

Chapter 2

Speech Prosody

This chapter aims at briefly introducing notions regarding speech prosody, its roles in communication, and at presenting the main paradigms used for modeling speech prosody in Text-To-Speech systems.

What is prosody?

There is no consensus on the definition of prosody; it depends on the level of analysis and representation. In this work, we have chosen the most relevant for the purpose of this thesis, which is the characterization and the generation of an adequate prosody to synthesize audiobooks. From this perspective and as defined by [START_REF] Cristo | Adaptive dimension reduction using discriminant analysis and k-means clustering[END_REF], prosody can be defined as a mechanism which supervises the management of a set of parameters that are:

-Fundamental frequency (F 0 ): The frequency of a sound corresponds to the number of vibrations per second, namely period: if there are few vibrations per second, we hear a low tone if there are more vibrations per second a high tone. The frequency is expressed in Hertz (Hz). This definition is valid for all types of periodical signals. However, speech is a complex signal, and it is not strictly periodic. In human speech, the principal frequency called F 0 corresponds to the frequency of the vocal folds. The F 0 is commonly referred to as pitch, which is the perceptual representation of F 0 . In this work, these two denotations (F 0 and pitch) are interchangeable.

-The intensity depends on the amplitude of the vibration induced by the speech signal: the higher the amplitude, the louder the sound; the lower the amplitude, the weaker the sound. It is commonly expressed in decibels (dB).

-The duration depends on the time during which a speech unit is produced. The unit used is the second (s). Suppose we consider speech units such as phones and pauses. Their duration depends on the context in which they were generated.

From these basic parameters, prosodic elements are derived such as:

-Tone and intonation are elements of prosody that refer mainly to pitch patterns. A tone is a pitch contrast that is limited to the syllable or word, and is manifested by a relative difference in pitch between syllables or words that follow each other. This element is specific to tone languages such as Chinese. On the other hand, intonation is a pitch pattern imposed by the utterance for the purpose of expression other than the pitch difference between words or syllables. Intonation is sensitive to emotions, politeness, context, and in general, to speaking style. Unlike tones, intonation is present in all languages.

-Prominence: In [START_REF] Büring | Intonation and meaning[END_REF]], prominence is defined as "local valleys and peaks in the voice's fundamental frequency are perceived as prosodic prominence, understood as emphasis, and modeled as pitch accents." Prominence is probably one of the broadest subjects of prosody, as many underlying notions are related to it (emphasis, accent, rhythm, and metrics). An utterance is a sequence of syllables. These syllables are not perceived as having a similar pitch level or energy. Some syllables appear more prominent because they are longer, or because they receive a particular prosody. Therefore, the terms of accent or accentuation relate to a phenomenon of prominence or local salience, which can assume in the language a metrical or pragmatic function.

A distinction is thus made between metrical accentual phenomena and accentual phenomena with emphatic value. The study of metrical accentual phenomena should make it possible to account for the distribution of stressed syllables. In French, they play an essential role in the demarcation of prosodic phrasing, and therefore, also in interpreting the utterance. The study of metrical phenomena is based on a distinction between meter and rhythm. At the meter level, the study of a language's metrical functioning relies on defining the syllables that are likely to receive an accent in the language. Rhythm, on the other hand, is built from the syllables stressed in a given utterance.

-Phrasing and prosodic structure: An analysis of the speech flow highlights the establishment of chuncks composed of syllables or words. These phrases can be delimited by a pause or not. In the same way, these newly formed phrases can participate in building a larger phrase. This structure is referred to as a prosodic structure of utterance. The analysis of the prosodic structure requires to take into account syntactic and semantic information. However, in some instances, prosodic groupings do not respect certain syntactic boundaries. Phrasing should help the interpretation and analysis of an utterance.

Roles of speech prosody

According to [START_REF] Cristo | Adaptive dimension reduction using discriminant analysis and k-means clustering[END_REF], many roles are attributed to prosody, including lexical, demarcative, pragmatic, behavioural, emotional, identifying, stylistic and many other roles.

In [START_REF] Christodoulides | Effects of cognitive load on speech production and perception[END_REF]] three prosody roles are reported: linguistic, para-linguistic, and extra-linguistic.

2.1 Linguistic [START_REF] Vaissière | Language-independent prosodic features[END_REF]], claim that some prosodic features fulfill similar roles across languages. For instance, pauses, some fundamental frequency features (declination tendency, resetting or baseline), durational features (final lengthening) and intensity. The author highlights the prosodic differences among languages (differences in timing, different orders of priorities, different relationships between F0, duration and intensity).

In [START_REF] Cohen | The role of prosody in discourse processing[END_REF], the authors have shown through two subjective assessments, including twenty subjects each, that altered prosody and punctuation similarly affect performance and seriously impair text comprehension and word recognition. The authors claim that linguistic prosody supplies redundant cues for judging sentence structure and manages attentional resources to help with the semantic encoding of lexical units and with the organization of linguistic information in long-term memory. [START_REF] Veenendaal | The role of speech prosody and text reading prosody in children's reading comprehension[END_REF] found that speech prosody contributes significantly to the construction of the meaning of written texts. This result was founded by performing reading and language assessments over 106 subjects (Dutch fourth-grade primary school children) using storytelling task and oral text reading performance such as decoding skills, vocabulary, syntactic awareness, and reading comprehension.

Para-linguistic

Prosody is involved in several paralinguistic parameters, including speaker attitude, emotional state, affective, and cognitive states. [START_REF] Liscombe | Prosody and speaker state: paralinguistics, pragmatics, and proficiency[END_REF]] has explored the primary information that are carried by prosody from three distinct speaker state-related perspectives: a) Paralinguistic: Pitch contour shape seems to discriminate emotions in terms of positive and negative affect. b) Pragmatics: The phrase-final rising of the intonation pattern plays a crucial role in the studied case of student questions in a corpus of one-on-one tutorial dialogues. c) Proficiency: There is a significant correlation between intonational features, including syllable prominence, pitch accent, and boundary tones, and language proficiency assessment scores at a strength equal to that of traditional fluency metrics.

According to [START_REF] Levin | The prosodic and paralinguistic features of reading and telling stories[END_REF], it is possible to differentiate between story-reading and storytelling, relying on intonation prosodic features and paralinguistic features such as speakers' behavior in both situations.

Extra-linguistic

Extra-linguistic dimension is related to the speaker's physiological characteristics, and also to idiolectal, and geographical (ethnicity) information. Furthermore, all these pieces of information appear in speech communication through non-verbal vocalization. [START_REF] Labov | The study of language in its social context[END_REF]] describes "social speech registers" such as styles or modes of social speech as extra-linguistics variables that contribute to the construction of a social situation. In [START_REF] Levin | The prosodic and paralinguistic features of reading and telling stories[END_REF], story-telling and story-reading are seen as a concrete example of "social speech registers." This example is defined as functions of communication that can designate the speech registers, the topic, the setting, social characteristics of the listeners and the speakers (e.g., age, sex, ethnicity, education, social class) and the relationships between the speaker and the listener [START_REF] Gumperz | The speech community[END_REF]]. [START_REF] Maekawa | Discrimination of speech registers by prosody[END_REF] showed that prosodic parameters such as prosodic label frequency information and speaking rate allow to automatically discriminate four speech registers including academic presentation, simulated public speech, dialogue, and reproduction speech in the Corpus of Spontaneous Japanese (CSJ).

In [START_REF] Trouvain | Laughing, breathing, clicking-The prosody of nonverbal vocalisations[END_REF]], the authors investigate three cues considered as non-linguistic features (that we call extra-linguistic) such as laughing, audible breathing, clicking in conventional speech. The studies highlight the importance of prosody in characterizing and describing the non-linguistic speech information.

3 Prosody Modeling for Text-to-Speech Synthesis [START_REF] Rajeswari | Prosody modeling techniques for text-to-speech synthesis systems-a survey[END_REF] describes the prosody modelling for TTS as "the process of building computational models to produce prosodic variations in synthesized speech automatically". The authors report three approaches that are discussed in the following.

Rule-based methods

These techniques require solid knowledges at the text processing stage as well as at the speech processing stage. Regarding text processing, techniques derive a formal description of the phonetic and phonological properties according to the context and studied language. The corresponding speech has to be consistent with text features. This approach is usually difficult to implement and expensive in terms of time because it requires manual annotation, which generates annotation conflicts between annotators in the case of several annotators. In most cases, the annotations are carried out on a small data set. Furthermore, the prosody models resulting from this method are highly dependent on the language for which they were designed as well as on the type of data studied.

Statistical data-driven methods

This technique relies exclusively on statistical analysis and modeling of the phenomena present in the data. We can cite the likelihood-based prosody model and posterior based prosody modeling. However, the most successful prosody modeling nowadays is the deep learning approach. This success is due to the availability of data and the computational power progresses of modern computers.

Hybrid approach

The hybrid model is a combination of both rule-based and statistical based approaches. The prosody modeling implemented in the Merlin toolkit [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF] is an example of a hybrid model. This model requires the extraction of a set of linguistic features during the pre-processing process. These features are fed respectively to the duration model which makes predictions regarding duration, and an the acoustic model which predicts the prosodic features (mainly F0/pitch and spectral information of the speech).

For exploring and characterizing the expressivity of speech, sentence-based prosodic features are not enough.

What are the topics discussed in this manuscript?

It is clear that to obtain a synthetic voice of good quality that we can use in particular contexts, it is fundamental to improve the prosodic models used text-to-speech synthesis.

The French language prosody will be the focus of our investigations. Aspects related to the variation of parameters and to prosodic phenomena (such as intra-speaker and inter-speaker variability) will be addressed in chapter 4, followed by paralinguistic studies related to emotions conveyed by audiobooks. The relation between prosody and discourse will be studied in the chapters 5 and 6.

Chapter 3

Audiobooks Corpora For Expressive Speech Synthesis

This chapter is an extended version of the work described in "SynPaFlex-Corpus: An Expressive French Audiobooks Corpus Dedicated to Expressive Speech Synthesis" presented at LREC18 [START_REF] Sini | Annotation automatique des types de discours dans des livres audio en vue d'une oralisation par un système de synthèse[END_REF]].

We propose a new corpus of audiobooks, containing about 600 hours of speech (silence and pauses included). We present the annotation methodology and exploratory experiments that we conducted in order to have a clear idea of the expressiveness carried by this kind of data at text level and the corresponding speech. The initial corpus called SynPaFlexcorpus contains a single female speaker, but we found that this data was not sufficient to characterize expressivity in all its complexity. So we decided to extend the corpus to multi-speakers in order to consider speakers reading strategy perspectives. This new version is named MUltispeaker French Audiobooks corpus dedicated to expressive read Speech Analysis (MUFASA).

We designed these corpora by considering three goals. The first goal consists of exploring the text related features such as morpho-syntax, semantics and phonology, discourses types, and literary genres. The second one aims to analyze and to characterize the intra-speaker prosodic patterns related to the phenomena due to reading aloud a long text, and the last goal focused on the inter-speakers variation exploration/characterization.

We compare the MUFASA-Corpus with other existing corpora dedicated to TTS to figure out uncovered aspects of expressivity.

SynPaFlex Corpus

It seems impossible to describe the expressivity of speech by a finite number of rules that cover all the exceptions, factors, and contexts. Data-driven techniques seem to be a well adapted solution for this kind of challenge and the availability of data makes their use possible. Even though finding appropriate data, tidy data is, in some sense, also challenging and raises other difficulties such as the dependency between the model and the data (the data quality impact on the model performance model). In this thesis, we are especially interested in audiobook data.

Motivation

The SynPaFlex corpus is an audiobooks corpus of single female voice. The data were collected according to this criteria:

-Availability of a large quantity of data uttered by a single speaker; -Availability of the corresponding written texts; -Good audio signal quality and homogeneous voice; -Various discourse styles and literary genres; -Conveying emotions in speech.

Relation to previous work

Many corpora dedicated to the synthesis of speech already exist. Most of them are in English. For French, most corpora do not exceed ten hours of read speech by only one speaker. Most professional corpora recorded to build a synthesized voice are often sentenceby-sentence records. Except the GV-LEX [START_REF] Doukhan | The GV-LEx corpus of tales in French[END_REF]] corpus for which the author seeks to characterize expressiveness beyond the sentence. However, this last work is entirely dedicated to a particular genre that is fantastic tales dedicated to young children.

The whole annotation pipeline were handled with the ROOTS toolkit,that allows storing various types of data in a coherent way using sequences and relations. This toolkit [START_REF] Chevelu | ROOTS: a toolkit for easy, fast and consistent processing of large sequential annotated data collections[END_REF] allowed us to incrementally add new information to the corpus.

Once audio data have been selected and the corresponding texts have been collected, a few manual operations have been applied to simplify further processing. Notably, as recordings were performed in different technical and environmental conditions, loudness has been harmonized using the FreeLCS tool 1 . Despite of that, audio data acoustic features remain more or less heterogeneous. Therefore, analyzing the intensity of audio files is now possible.

As texts were coming from diverse sources, their formats were unified. Then the exact orthographic transcriptions of the readings were achieved by inserting the introductions and conclusions the speaker added in the recording, and by placing footnotes and end-of-book notes where they appear in the reading stream.

The next step has been to normalize the texts using rule-based techniques appropriate for the French language, and split them into paragraphs. For the rest of the process, we kept each chapter in a separate file so as to keep long term information accessible.

Data Collection and Pre-processing

Most of the texts collected are in the public domain. Two sources have been mainly used: the Gutenberg 2 project and the Wikisource 3 bookstore. Records have been collected along with the corresponding text in plain text format. Few manual adjustments were performed on the text to insure its correspondence to the audio files. The original text structure is respected. Most of the texts studied were published between the 17th and the 20th century.

In narrative or descriptive texts such as in novels, short stories and tales, the paragraph is considered as basic text unit. On the other hand, poems and fables are structured in verses. Consequently the utterance represents a verse.

Each utterance is tokenized then normalized, which consists of orthographically transcribing numbers and acronyms. This is done using rules set manually by experts. A syntactical analysis of all utterances is performed to establish the syntactic function of the words content.

The original audio files are mostly in MP3 format, with a sampling rate of 22.050 khz or 44.1 khz each of these samples being coded on 16-bit with a bit rate ranging from 64 to 128 kbps. All the recordings were converted to wav format with a sampling frequency of 22.05 khz in order to have a consistent corpus.

Speech Segmentation

The broad phonetic transcription, based on the French subset of Sampa, has been extracted and aligned with the speech signal using JTrans [START_REF] Cerisara | JTrans, an opensource software for semi-automatic text-to-speech alignment[END_REF].

To evaluate the accuracy of the phone segmentation, an expert annotator performed a manual validation using Praat [START_REF] Boersma | Praat: Doing phonetics by computer[END_REF].

The evaluation process of the forced alignment consists in first generating the automatic segmentation and phonetic labels of selected dataset based on this annotation. The annotator has to add a sequence corresponding to the correction.

Since there is only one speaker, half an hour of the SynPaFlex corpus has been taken into account to evaluate the quality of phone labels and boundaries. The set of data used for the evaluation task has been selected respecting the proportions of the different literary genres in the corpus.

Results related to the validation are presented in Table 3.1. We can observe that the Phoneme Error Rate (PER) is low for every literary genres, and the average PER is 6.1%. Concerning the average alignment error, results are reported in the fourth column of Table 3.1. Globally, on average, the error is 11ms.

As far as errors on label assignment are concerned, they mostly occur on vocalic segments. Most of the deletion observed involve /@/ (83.31%), this phoneme being generally optional in French. The majority of substitutions concerns mid vowels (37.04% for the substitution of /E/ by /e/, and 31.04% for /o/ by /O/), these realizations being the result of a specific pronunciation or simply phonetization errors. than 20ms. In poems, however, alignment errors are more important: for 35% of the vowels, boundaries have been shifted by more than 20ms. It could be explained by two distinct factors. First, the speech rate is relatively slow in poems (with an average of 5 syllables/s) in comparison to other literary genres where the speech rate is of 6 syllables/s on average. Secondly, the acoustic models used to achieve the automatic segmentation [START_REF] Cerisara | JTrans, an opensource software for semi-automatic text-to-speech alignment[END_REF] have been trained on the Ester2 corpus [START_REF] Galliano | The ESTER 2 evaluation campaign for the rich transcription of French radio broadcasts[END_REF] which is a French radio broadcasts corpus. The resulting models could thus be slightly not well-adapted for poem reading data.

To improve the segmentation performance we have tried two different ways:

• First way consists of adapting the default acoustic model to our speakers.

• For the second one, we have trained a new model using Montreal Forced Aligner [START_REF] Mcauliffe | Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi[END_REF] tool which is based on Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF]] Speech Recognition. Although, this tool is easy to set up, it is difficult to align long speech data like the chapters. To face this problem, we had first to segment the chapters into utterances using JTrans. The utterances thus obtained are then used to learn the acoustic model and then align the corresponding transcriptions at the phoneme level.

Furthermore, we plan to use the Train& Align [START_REF] Brognaux | Train&Align: A new online tool for automatic phonetic alignment[END_REF]] online tool which seems to be more appropriate to our data. This tool proposes to train an acoustic model and to align the data at the same time which corresponds better to SynPaFlex corpus structure.

Linguistic Information

Additional linguistic information has been added to the corpus, such as syllables and Part-Of-Speech tags using the Stanford parser [START_REF] Green | Multiword Expression Identification with Tree Substitution Grammars: A Parsing tour de force with French[END_REF]]. Table 3.3 sums up the content of the corpus in terms of linguistic units. However, we did not verify the precision of this annotation. We plan to include constituency-dependency parsing using BONSAI4 presented in [START_REF] Candito | Benchmarking of statistical dependency parsers for french[END_REF]

Acoustic and Prosodic Information

The speech signal is stored using a sampling frequency of 22.05 kHz. From the signal, we have extracted (i) the energy and 12 mel-frequency cepstral coefficients (MFCC 1-12) which we have added delta and delta-delta coefficients using [START_REF] Gravier | SPro: "Speech Signal Processing Toolkit[END_REF]], (ii) the instantaneous fundamental frequency (F 0 ) using the ESPS get_f0 method implementing the algorithm presented in [START_REF] Talkin | A robust algorithm for pitch tracking (RAPT)[END_REF], and (iii) pitchmarks using our own software.

Additionally, we have added some prosody related features as the articulation rate (in syllables/s), the speech rate (in syllables/s), and F0 mean/min/max/range (in Hz) at the syllable and word levels. Since the corpus contains several speakers, we suggest to compute Fundamental frequency in semi-tone5 scale to be able to compare among speaker voices.

MUFASA Corpus

MUFASA is an extension of the SynPaFlex Corpus. The database has been collected and processed in a similar way of the SynPaFlex Corpus. Unlike the SynPaFlex Corpus, this corpus was collected from two different libraries i.e, LibriVox.org (LV)6 and LitteratureAudio.com (LA)7 entirely dedicated to French audiobooks. LA is not in the public domain, unlike LV, authorization is required and we have asked the administrator for authorization to use certain voices exclusively for research purposes.

Motivation

We decide to build MUFASA corpus for:

-Analyzing inter-speaker variations to have a better understanding and a more comprehensive view of the strategy adopted when reading audiobooks.

-Distinguishing the speaker-related characteristics from those related to texts.

-Finding strategies common to the various speakers, which makes it possible to extract prosodic structures appropriate to the reading of audiobooks.

-Considering the speaker's prosodic identity by characterizing the inter-speaker variability.

The MUFASA corpus is intended to be close to the LibriTTS [START_REF] Zen | LibriTTS: A corpus derived from librispeech for text-to-speech[END_REF]] corpus, as both deal with the exponents of amateur audiobooks and contain several speakers. On the other hand, the two corpora differ in the fact that in MUFASA, each speaker is represented by at least two hours of speech, and the language of reference is French. Several recordings for the same text (parallel data) are provided, allowing an analysis of the difference between speakers without worrying about the linguistic characteristics.

The novelty of this work

This MUFASA Corpus offers the possibility to explore the expressivity in different way such as:

-Parallel data (Appendix 4): same text recorded by different speakers.

-Certain famous French authors are well represented in the MUFASA corpus. This can be exploited to study author style.

-Enough data to characterize the style of speakers: as the first voice we favored voices that recorded more works, of different genres (poem, fable, tale, short story, and novel).

-Enough data to characterize the genre: In order to study and analyze the characteristics of genres regardless of speakers. The genre may convey a rather special expressiveness depending on the speaker and the authors.

-To compare professional and amateur recording, we also collect certain passages read by amateurs and professionals 8 .

A summary of the contents of the MUFASA-Corpus is presented in the For more details about the contents of the MUFASA see the Appendix 3.

Gap between TTS designed corpora and amateur audiobook recording

Both SynPaFlex and MUFASA corpora were constructed to build and to improve TTS systems, whether it is concatenative, Statistical Parametric Speech Synthesis, or End-To-End (E2E) systems. Careful analysis of these databases that we called amateur audiobooks allows us to notice a difference between the quality of their recording and the recording quality of the databases made in a laboratory (ex. SIWIS French Speech Synthesis Database).

When we design a corpus for TTS (in a laboratory), we tend to be careful about the recording conditions, such as the microphone's position, acoustic properties of the recording room, and the reliability of recording materials. However, in amateur audiobooks, the lack of control over the recording conditions introduces an error on the measure of some prosodic parameters sensible to the noise. Beyond those signal processing gaps due to signal quality, there are differences between guided records data that we will consider as a professional recording and amateur ones at the suprasegmental level that we will try to highlight in this work.

Data and features extraction Data

To figure out the differences between professional and amateur recordings, we have investigated a sub-corpus that contains two chapters from two separate novels. Both chapters have been read by three different speakers, i.e., three recordings including a single professional record at each time and two amateur recordings. Table 3.5 summarizes the linguistic contents and the duration of the subset used for conducting the experiments aiming to measure the gap between the amateur and professional recording.

Features extraction

We choose two prosodic parameters to study the difference between the speakers: 1) the average length of pauses within utterances, and their distribution 2) Subharmonic-to-Harmonic Ratio (SHR) and the vowel trapezoid as voice quality features.

• We chose to study the pauses' duration and their distribution in an audiobook read by three different speakers to see if there is a difference between the so-called professional and amateur speakers. Because the pauses are good indicators of speech style and implicitly how the data are recorded and prepared, we hypothesize that in a professional recording, the pauses' position, frequency, and duration are controlled and regularized. In contrast, in the amateur record, the preparation level is lower, thus a broader range of variation, and this tends to influence the other prosodic parameters such as articulation rate and F0-range.

• The vowel trapezoid is an articulatory schema that represents all possible vocalic timbers of the human vocal tract. Figure 3.2a describes all the timbres of the oral vowels of the world's language. This space is divided according to language in functional units. In French, there are ten functional timbers of oral vowels (cf. Figure 3.2b). This schema space is formed along two first formants, F1/F2. In a prepared speech, such as read speech, the contrast between the three cardinal vowels (/i/, /a/, and /u/) [START_REF] Audibert | Distorsions de l'espace vocalique: quelles mesures? Application à la dysarthrie (Distortions of vocalic space: which measurements? An application to dysarthria[END_REF] is usually studied for characterizing articulatory behaviour. The cardinal vowels represent the boundaries of the vocalic area [START_REF] Lindau | Vowel features[END_REF]].

• According to [Sun 2002;[START_REF] Sun | Perceived pitch of synthesized voice with alternate cycles[END_REF], alternate cycles9 (alternating in amplitude or period, or both) in speech signal make the determination of pitch more difficult. [START_REF] Titze | Workshop on acoustic voice analysis: Summary statement[END_REF]] claims that the alternating cycles in the time domain are manifested by the presence of subharmonics in the frequency domain. Furthermore, the magnitude of subharmonics with respect to harmonics reflects the degree of deviation from modal voice. From that, a new parameter called Subharmonic-to-Harmonic Ratio (SHR) was introduced to describe the amplitude ratio between subharmonics and harmonics.

The [START_REF] Sun | Perceived pitch of synthesized voice with alternate cycles[END_REF], present a pitch perception study. This experiment consisted of asking participants (expert listeners) to determine the pitch of synthesized vowels with alternate cycles through amplitude and frequency modulation. The results show that pitch perception is closely related to SHR. This experiment aims to find the relationship between the perceived pitch and SHR. This parameter is used for describing voice quality as well as for classifying voice production mode for one speaker or comparing voice quality for different speakers reliable for perceptual quality of voice.

Results

Voice quality analysis

We select all the voiced frames and calculate the SHR frequency distribution (see Table 3.6 according to [Sun 2002], when SHR is in the medium range, especially (0.2, 0.4], perceived pitch becomes ambiguous. Correspondingly, in Table 3.6, MFR0019 and FFR0001 have higher SHR percentage in the range of (0.2, 0.4] among three speakers, whereas Siwis French Speech (SFS) female speaker has the lowest. Visual inspection and listening to the speech waveform confirm that MFR0019 has indeed more "irregular" speech cycles and appears to have low and rough voice, whereas SFS's speech seems to be much more "regular". Similarly FFR0001 has more creaky voice than SFS despite the average pitch of FFR0001 is much higher. Table 3.6 also show that the professional speaker have greater number of SHRs in the range of (-0.2, 0.0] compared with amateur speakers. This indicates that professional speaker (SFS) speech might have greater amount of small amplitude or period fluctuations, which, however, are not significant enough to affect pitch perception. Table 3.6: Subharmonic-to-Harmonic Ratio distribution of the subcorpus speakers . For each speaker, we select all the voiced frames and calculate the Subharmonic-to-Harmonic Ratio frequency distribution.

The table shows that the SFS female voice has a high percentage of SHR close to 0 and the very low percentage of medium values (0.2 -0.4] which means that the pitch of this voice is quite easy to perceive by annotators if we consider the study [1,25]. In comparison with other voices FF0019/FFR0001 (reading the same text), the percentage of medium values is high and the percentage of values of SHR close to 0.0 is lower. In the second example considered in this study, we did not find a significant result between the professional voice (Podalydes) and amateurs' voices (FFR0020, FFR0011). We have conducted an informal perceptual test, where we asked an expert annotator to determine the pitch of the vowels /i/ and /a/ preceded by /p/,/t/,/k/ present in the subcorpus ( Table 3 The analysis of the vocalic trapeze in Figure 3.2, shows that SFS voice makes an important contrast among the vowels with minor variation, whereas, for the other speakers the contrast is not clear. The subjective assessment of the samples present in the Table 3.7 has also confirmed that among the six speakers, SFS is the speaker that tend to overarticulate the cardinal vowels. For instance there is a strong variation of /u/ and /i/ along F2 which implies a significant overlap between considered vowels.

Pauses

There is no difference between the three speakers concerning the number of pauses, within utterances according to Figure 3.4a, but the Figure 3.4b shows that the professional speaker SFS makes short pauses (average of 250 ms) and in constant manner. Whereas the two other speakers seem to produce long pauses ( 500 ms for FFR0001 and 480 ms for MFR0019), with important variation.

Discussion

In this study, we compared extracts of MUFASA corpus considering amateurs recordings with professional recordings through three prosodic parameters: SHR, vocal trapezoid, and pauses. The results show that professional data have stable pause durations and good voice quality. In contrast, amateur recordings tend to have inconsistent pause durations with considerable variation, and low recording condition compared to professional.

From these results, we can see that data recorded for speech synthesis has a couple of properties that distinguish them from amateur audiobooks, and professional speaker recordings that are not dedicated to speech synthesis. Despite the quality of audiobook data, professional data dedicated to speech synthesis is better to build a model of good quality.

A Phonetic Comparison between Different French

Corpora Types

The main purpose of this section is to highlight two representative properties of speech style carried by the audiobook corpus, which are the duration of the vowels and the values of the two first formants (F1, F2) of the cardinal vowels (/a/, /i/ and /u/). This second parameter has been chosen as the structure of the vowel trapezoids follows a logical organization, linked to the contrast and the stability of articulation. To highlight these properties, we conducted a comparative graphical and statistical analysis between a representative subset of MUFASA corpus and four other different types of French corpora(BREF [START_REF] Larnel | BREF, a large vocabulary spoken corpus for French[END_REF], ESTER [START_REF] Galliano | Corpus description of the ESTER Evaluation Campaign for the Rich Transcription of French Broadcast News[END_REF], RHAPSODIE [START_REF] Lacheret | Rhapsodie: a prosodic-syntactic treebank for spoken french[END_REF], and NCCFr [START_REF] Torreira | The Nijmegen corpus of casual French[END_REF]). While each of the considered corpora have been recorded for a different purpose and in varying conditions, using them allows us to evaluate the MUFASA-Corpus.

Corpus design

To be able to compare the data of the MUFASA Corpus with that of the other corpora, the vowels studied must appear with similar proportions in each of the extracts from the studied corpus. Ideally, even the context has to be similar. So the targeted vowels were therefore placed in an open syllable of CV structure and preceded by the consonants /p/, /t/, or /k/. The choice of consonants /p/, /t/, and /k/ is justified by the fact that this type of consonants facilitates the segmentation of vowels since their limits do not merge with those of vowels. Thus, three syllabic contexts were chosen for the three vowels studied. Table 3.8: The set of extracts for conducting a comparative study.

We will briefly describe the used dataset :

-The MUFASA extract contains nine different speakers reading distinct novels. We have selected the speakers with varying strategies of narration based on two criteria vowel duration and the average F0 amplitude.

-Bref [START_REF] Larnel | BREF, a large vocabulary spoken corpus for French[END_REF]: This read speech corpus designed for speech recognition (speaker-dependent and independent case), and it consists of texts selected from French newspapers, Le Monde. The extract contains nine speakers.

-Ester [START_REF] Galliano | Corpus description of the ESTER Evaluation Campaign for the Rich Transcription of French Broadcast News[END_REF]]: The used data are made up of France Inter radio broadcast news recorded in 1998, covering ten speakers.

-RHAPSODIE [START_REF] Lacheret | Rhapsodie: a prosodic-syntactic treebank for spoken french[END_REF]] is a spoken french corpus annotated in terms of prosody and syntax. From this corpus, we extract only the monologues and the private domain. This subset contains short clips ( 5 min per clips). Each clip was spoken by a single and unique speaker. The samples of these extracts have been mainly derived from C-PROM [START_REF] Avanzi | C-prom: An annotated corpus for french prominence study[END_REF]] corpus, which is also a French spoken corpus containing seven speaking styles: radio broadcast news, aloud reading, political speech, university conference, radio interview, route prescription, narrativelife story. Unlike the other corpora chosen for conducting this study, RHAPSODIE clips cover diverse speaking styles and contain 30 speakers.

-NCCFr [Torreira, Adda-Decker, and Ernestus 2010](The Nijmegen Corpus of Casual French): French speakers conversing among friends.

Table 3.8, summarize the contents of the designed corpus.

Data processing

The forced alignment at the phone level was performed using JTrans, then annotated according to the procedure described in the Section 1. All the information was stored in the TextGrid format. A Praat10 software script was used to collect the formant values F1 and F2 for each of the three vowels. All these data were compiled in a CSV data file where they were sorted and manipulated with a script written in the R language. The outliers, identified following the analysis of the formed vowel trapezoids, were removed from the report. The formants (F1 and F2) values in Hertz were then converted to Bark11 to be able to compare the data of the different corpora. The graphical analysis of the vowel trapezoids was done using the phonR12 package.

Results and discussion

After a graphical analysis of our data, we were able to observe a different dispersion in the vowel trapezoid for each corpus. Indeed, we can also notice that there is certain similarity among the read speech corpora (MUFASA and BREF), where there is a strong overlap between the vowels /u/ and /a/. We can group RHAPSODIE corpora that represent the diverse speaking styles. We can notice a large contrast on F2 in the ESTER and NCCFr corpora, which is not the case in the other corpora, this could be explained by the fact that these two last corpora have been recorded in good conditions.

The Figure 3.5 illustrates the dispersion obtained for each corpus. As expected, for most of the corpora studied, the /i/ is articulated on average in the closed anterior position, the /u/ in the closed posterior position and the /a/ in the open middle position.

Except for the data from RHAPSODIE ( Figure 3.5e), this differs from the others, especially from the two vowels /i/ and /u/, however the vowel /a/ is in the same position (in the open middle position.)

According to [START_REF] Moon | Interaction between duration, context, and speaking style in English stressed vowels[END_REF][START_REF] Baker | Variability in word duration as a function of probability, speech style, and prosody[END_REF][START_REF] Burdin | Phonetic reduction, vowel duration, and prosodic structure[END_REF] there is an interaction between speaking style and the duration of the vowels. The analysis of the density distribution according to the duration of the three vowels preceded by an occlusive consonant /p/,/t/,/k/ (similar context over speakers/corpus) illustrated by Figure 3.6, which consists of comparing each of the excerpts from the different corpora to the MUFASA corpus. It can be observed that the duration of the vowels is quite long, which is quite logical since speakers tend to take their time when it comes to read loudly or even when it is a prepared or partially prepared speech such as a radio diary.

Remarks

This study intended to be exploratory and attempted to provide some elements for reflection. This work raises two main reflections, which are the level of formality of the audiobook corpus in comparison with other speaking styles, and a second element, the presence of particular prosodic behavior specific to audiobooks data. In this work we try to compare five corpora designed for a different task with different sized speech inventories. Certain factors, mainly acoustic factors, may have influenced our results.

Conclusion

In this chapter, we presented a new audiobook corpus, the MUFASA corpus, dedicated to expressive speech synthesis but that can be used for other purposes, such as automatic speech recognition, natural language processing, second language acquisition, entity recog- Furthermore, we analyzed some aspects of expressivity of speech covered by the MUFASA corpus. We have shown that the recording of audiobooks differs between professional and amateurs in terms of voice quality. Nevertheless, we did not treat two important aspects of expressivity in this chapter, which are the emotion and discourses. Emotional speech is the main topic of the coming chapter. The second aspect not treated in this chapter is the discourse. Audiobooks cover an extensive variety of discourse encoded in the text, like dialogues amongst characters in a given novel, which contribute a lot to the expressivity of the audiobooks. We addressed the discourse typology in audiobooks in two chapters. In Chapter 5, we present the automatic detection and classification of the discourses types present in the audiobooks. Then, in chapter 6, we discuss the prosodic characteristics of discourse in audiobooks.

Chapter 4

Annotation Protocol and Emotional Studies

The first of this work was performed in collaboration with Gaelle Vidal (IRISA) and Marie Tahon (LIUM). I would like to thank both of them: Gaëlle for collecting the initial version of SynPaFlex corpus, the annotation of the data, and providing an invaluable discussion, and Marie for her technical support in the machine learning task.

Introduction

In this chapter, we first report the specificity of the annotation protocol that has been realized on the SynPaFlex corpus. This annotation allows highlighting certain aspects related to intonation patterns, discourses mode properties through the fictional characters present in the narrated stories and emotion segments. At the same time, this annotation allows the exploration of the SynPaFlex corpus. Then, the focus is made on the annotation of emotions through a pattern classification experiment to evaluate the annotation process. Finally, a clustering experiment was conducted, aiming at estimating the possible correlation between text and acoustic signal properties. All these experiments were conducted using the first version of the SynPaFlex-Corpus described in [START_REF] Sini | Annotation automatique des types de discours dans des livres audio en vue d'une oralisation par un système de synthèse[END_REF]] ( Appendix 2 summarizes the proportion of manually annotated parts).

Speech annotation

In recent decades, many works on speech annotation protocols have been proposed [START_REF] Bird | Speech annotation and corpus tools[END_REF]. In [START_REF] Brognaux | A new prosody annotation protocol for live sports commentaries[END_REF], the authors proposed an intonation annotation protocol dedicated to living sports commentaries. The annotation is made according to two levels: local labels are assigned to all syllables and refer to accentual phenomena; and global labels allow classifying sequences of words into five distinct sub-genres, defined in terms of valence and arousal. This approach deals with both discrete and continuous annotation strategies. Whereas [START_REF] Montaño | The Role of Prosody and Voice Quality in Indirect Storytelling Speech: Annotation Methodology and Expressive Categories[END_REF] proposes an analysis methodology to annotate storytelling speech at the sentence level based on storytelling discourse modes (narrative, descriptive, and dialogue), besides introducing narrative sub-modes denoted as expressive categories. Furthermore, in [START_REF] Devillers | Real life emotions in French and English TV video clips: an integrated annotation protocol combining continuous and discrete approaches[END_REF]], the authors explore real-life emotions in French and English TV video clips aiming to a federative annotation protocol by combining continuous and discrete approaches.

In this work, we propose a simple manual annotation covering four different aspects of audiobooks such as intonation patterns, characters/dialog labeling, discrete emotional labeling, and a set of labels relative to precise events.

The major challenge of expressive voice in general, and in the case of audiobooks in particular, is the lack of annotated data, as the annotation is a time-consuming step. In addition to time, the inter-annotator agreement is mostly problematic, especially when it comes to annotating emotions.

We propose unilateral annotation (One Voice -One Annotator), as we believe that this strategy will provide us with consistent and uniform annotation across all annotated data.

Protocol

Audio tracks corresponding to chapters of different books have also been annotated manually according to a set of intonation patterns, characters, emotions, and other events. This was achieved by the annotator who was listening to the audio signal using WaveSurfer1 software. The annotation method had first been defined on a small subset of readings, and then tested on audiobook recordings completed by other readers. It was found to be generic enough to render a global perceptive description of the speech. As Table 4.2 shows, 38% of the whole corpus have been processed manually to provide characters annotation, and 15% -included in those 38% -to describe emotional and intonation patterns contents.

Intonation Patterns

Delattre's work is one the earliest work in French intonation modelling. In [START_REF] Delattre | Les dix intonations de base du français[END_REF]], the author defines ten fundamental intonation patterns (cf Figure 4.1) which are considered as the most frequent pitch contours in French. After considering the whole speech data, eight intonation patterns were defined, then encoded and assigned by an expert annotator2 to a large number of audio tracks corresponding to chapters of eight different books, and corresponding to a 13h25m sub-corpus.

Literary

As far as possible, labels were assigned according to the perceived prosody, without taking into account the linguistic content. They characterize units which could range in length from a word to several sentences. Seven of these labels correspond to speech showing the following types of intonation patterns: Question (interrogative), Note, Nuance, Suspense, Resolution (authority, or imperative), Singing, and Nopip(no particular intonation pattern, or declarative). The eighth label, Emotion, was used to report -but without describing it -the presence of any perceived emotional content.

Let's notice as of now that the tag Exclamation is not listed above. This is because this information can be simply deduced from another level of description: in this corpus, the Exclamation pattern was found strictly correlated with the emotional content of surprise, which is reported in the emotion labeling level (presented in Section 2.4). Manual annotation is costly in time and redundancy is not desirable in its process. In the following analysis of the intonation manual labeling, emotion labels surprise will therefore be assimilated to hidden intonation labels for Exclamation.

Another important point is that, when needed for a more precise description, labels were combined (e.g. Emotion+Question+Nuance illustrated in Figure 4.3).

Among the intonation parameters, the perceived pitch-curve during voice production takes an important role in assigning the labels. For instance, the nuance pattern, which is one of the reading strategy of the speaker, maintains listener's attention. This pattern is characterized melodically by a high pitch at the beginning, then a decrease with modulations, and finally a slight increase when it doesn't end the sentence (see Figure D.3).

Table 4.3 shows total duration for each manual intonation labels in the 13h25 sub-corpus. hidden Exclamation tag is very largely represented (more than 4h42), before the Idle one (4h21m). The first particular intonation pattern that comes after is Nuance (3h58m), then come all the other intonation patterns that are relatively well represented and evenly distributed (around 40m): Resolution, Suspense, Question and Note. singing was found to be exceptional and is not reported here.

More than half of the speech showing particular intonation figures is described with combined labels pointing out where prosody may be more complex (cf. Figure 4.3).

Most of all, it was found that the Exclamation pattern happens very frequently, especially in narration. In a way, it is an inherent part of the speaker's style.

The generic Emotion intonation indicator is assigned to 39% of the whole sub-corpus (5h18m), showing a large amount of emotional data. Its manual description is presented in Section 2.4.

Characters

The speaker, who is the same for the whole corpus, can personify the different characters of the book by changing her voice or her way of speaking. The character's tags were identified from the text and any turn of speech has been labeled according to the following annotation scheme:

• Character ID: indicates which character is talking according to the text, and refers to Meta-data where each character is summarily described (name, age, gender, prosody and timbre features). For instance, to personify a gloomy man, the speaker uses a low pitch, low energy and devoiced voice.

• vocal personality ID: indicates which character is talking according to the vocal personality. Indeed, even if the speaker is very talented and coherent along the books, she can for example forget to change her voice when a new character starts taking.

Therefore, for such speech intervals, voice quality remains the one of speaker or corresponds to another character. This may also be an intentional choice. Reading with incessant voice changes may become painful to listen to, or artificial.

The characters labeling was annotated on more than one third of the whole corpus (33h39m) extracted from 18 different books. Dialogue tags were reported as parts of the narrator's speech.

Rough estimates indicate that one third of the speech is in direct speech style. The average duration for speech turns being of 7s, against 29s for the narrator. In some chapters, direct speech segments can also be very long, typically when a character becomes a narrator who tells his own story. 370 characters were identified, and the full data of their vocal personality labeling indicates a not negligible amount of prosody and vocal tone personification. Covering a wide range of types, the speaker's voice is thus more or less radically far from her natural style (males, children and elderly people embodiments, psychological singularization, imaginary figures). These vocal personality changes often happen: around 20% of the speech is concerned and, for the half of them, the voice used contrasts with the speaker's natural voice.

Emotions

Different theoretical backgrounds are classically used to identify emotional states, principally based on either distinct emotion categories or affective dimensions [START_REF] Cowen | Self-report captures 27 distinct categories of emotion bridged by continuous gradients[END_REF]. Usually, choosing the emotion categories and their number, or the emotion dimensions is an issue.

In the present study, the basic scheme used to manually encode emotions has three items:

• Emotion category: Six categories selected by the Basic Emotions theory [START_REF] Ekman | Basic Emotions[END_REF]] are used: Sadness, Anger, Fear, Happiness, Surprise, Disgust. Two other categories were added to better represent the content of the different books: Irony and Threat.

• Intensity level: a scale from 1 to 3 was added to give a measurement of the experienced emotion intensity. For instance, one can interpret its values as follows: slightly angry (1), angry (2) , and strongly angry (3).

• Introversion/Extroversion: This binary feature reflects the way the emotion is rendered through the speech (discreetly, prudently / obtrusively, ostentatiously)

The second and third features may have strong correlations with some of the widely used affective dimensions, as activation and arousal. Furthermore, an important feature of the manual emotion annotation used for the corpus is that the three items labels can be mixed together to provide a more precise description of the perceived emotion. For instance, speech can continuously convey strong and very expressive sadness as well as fear through some words, which could be tagged as [sadness-3-E + fear-1-E].

Manual emotion labeling was done on sub-part of the already annotated corpus (13h25m). A large amount of emotional content was reported (39% of the speech, including 13% with combined tags). Duration of tagged speech for each category of emotion is given in Table 4.4, and the number and average duration of labels are indicated in Significant observations have emerged during the annotation. A challenging one is that two radically different types of Joy can be conveyed by the speech, whereas none of the three items could take over their differentiation: on the one hand suave joy, and on other hand elation or gladness. Also, it is suggested that labels should be interpreted in context, notably in conjunction with the discourse mode. In particular, the expressive strategy implemented in the corpus narration is very specific, conveying almost continuously positive valence but in a subtle way, through pitch modulation and with focus words. The Surprise label was widely assigned to those recurrent patterns showing (i) a sudden pitch shifting upwards (ii) at least one accentuation onto the first syllable of a focus word (iii) a phonetic elongation or a short silence before this first syllable. Thus, as introduced in Section 2.2 Surprise describes a recurrent emotional attitude of the reader, attracting the listener attention by regularly emphasizing the text.

Other types of variation occur when the speech conveys emotion, some examples are related in

Other Events

Besides acoustic indications of loud noises or music, different unexpected speech events were also:

• Linguistic events such as the use of foreign languages;

• Phonetic events which are not written in the text such as phoneme substitutions, elisions and insertions, high elongations, breaks and pauses, specific voice quality (e.g. whispered voice).

All these features can be of high interest for rendering a more human synthetic voice [START_REF] Campbell | Conversational speech synthesis and the need for some laughter[END_REF]].

The manual data-sets could provide valuable guidance for further analysis, especially by combining speech signal properties with linguistic information, acoustic measurements, and other descriptions. Examining how manual labels are distributed among literary genres could also be of great interest.

Evaluation of the emotion annotation

Among the manual annotations presented above, we decided to focus our effort on the emotion annotation in order to measure the reliably of the proposed annotation protocol and for comparing results to what was observed in previous studies.

To do this, binary emotion classification3 experiments [START_REF] Sugiyama | Introduction to statistical machine learning[END_REF]] were conducted on emotional labels of the SynPaFlex sub-corpus. Results are presented in this section.

The use of a state of the art methodology aims at positioning our mono-speaker read expressive speech corpus among existing multi-speaker acted or spontaneous emotional speech corpora.

Data analysis

The manual segmentation and labeling of emotion -which concerns 15% of the whole corpus -results in a total number of 8 751 segments as shown in Table 4.6. Among them, 5 387 convey an emotional content, while 3 364 do not. To get around the issue of a "neutral" emotion. We decided to label these segments as Idle, which consists of all non-negative states, according to [START_REF] Schuller | The interspeech 2009 emotion challenge[END_REF]. As mentioned previously, label combinations were used during the annotation phase to better characterize some expressive content. Consequently, these annotations are considered as new emotional labels which can not be merged with single labels easily. One possible solution is to analyze these samples and choose the dominant emotion. A more in-depth investigation of these label combinations is needed in order to manage them in a speech synthesis system. Interestingly, the Surprise label is highly represented among other single emotional labels. Actually, as described in Section 2, Surprise better corresponds to an emotional attitude of the reader to keep the listener's attention, than an emotion conveyed from the text.

Emotional segments are defined as segments consisting of an homogeneous emotion, be it characterized by single or combined labels. Therefore, there is no constraint on segments' duration. As a consequence, some segments can be very long. For example, one Idle segment lasts more than 43s. On average (cf

Methodology

The following experiments aim at classifying the manual annotations with binary emotional models. We know that for multi-speaker acted emotions, classification scores usually reach high performance (for example with corpora such as EMO-DB [START_REF] Burkhardt | A database of German emotional speech[END_REF] or JL-Corpus [START_REF] James | An Open Source Emotional Speech Corpus for Human Robot Interaction Applications[END_REF]). However, with multi-speaker spontaneous speech, the classification rates are much lower, thus reflecting the difficulty to discriminate emotions in such a context [START_REF] Schuller | The interspeech 2009 emotion challenge[END_REF]]. The present corpus gives the opportunity to bring a new benchmark of performances on mono-speaker read speech.

To do so, our experimental set up follows a standard classification methodology [START_REF] Schuller | The interspeech 2009 emotion challenge[END_REF]Schuller et al. 2013a]. By this way, our results are comparable with those obtained on other existing emotion corpora. In other words, emotional models are trained in cross-validation conditions (here 5 folds to keep enough data) on acoustic features. 384 acoustic features (16 Low-Level-Descriptors (LLD) + 16 ∆) × 12 functionals ( Table 4 An informal analysis of the manual emotion segments and other emotional French speech corpora [START_REF] Chateau | Analysis of emotional speech in voice mail messages: The influence of speakers' gender[END_REF][START_REF] Scherer | Vocal expression of emotion[END_REF][START_REF] Johnstone | The effects of emotions on voice quality[END_REF][START_REF] Chateau | Analysis of emotional speech in voice mail messages: The influence of speakers' gender[END_REF][START_REF] Abrilian | Emotv1: Annotation of real-life emotions for the specification of multimodal affective interfaces[END_REF][START_REF] Beller | Talkapillar: outil d'analyse de corpus oraux[END_REF][START_REF] Devillers | Real life emotions in French and English TV video clips: an integrated annotation protocol combining continuous and discrete approaches[END_REF]] , we have observed that in most of cases one second is enough to recognize the emotional sample identity. For that reason and to have homogeneous segment durations, we decided to chunk manual segments every 1 s. This operation helps in increasing the amount of data available for the experiment, as reported in Table 4.6.

As aforementioned, Combination labels are not taken into account because merging them with single labels is clearly not obvious. Also, Irony and Threat segments are discarded regarding to the small number of labels. To better identify the pairs of labels that can be easily discriminated from those which can not, only binary models are trained thus resulting in an emotion confusion matrix. The number of segments is equally balanced among the two classes.

Results

Models are trained with Random Forests and entropy criterion. Similar performances were obtained with optimized Support Vector Machines (polynomial kernel, C=1, γ = 0.01) and normalized features. The results are given as a confusion matrix between emotions as shown in Table 4.8. On average, performances obtained with the smaller set are the best: 59.9% with OS24, 59.5% with OS192 and 58.8% with ∆OS192. This first observation underlines the importance of selecting features when classifying emotions in such corpora in order to avoid over fitting the data [START_REF] Tahon | Towards a small set of robust acoustic features for emotion recognition: challenges[END_REF]].

As we were expecting, the binary emotion classification UAR results range from 43.6% to 81.8%, a typical range for induced and spontaneous speech emotion recognition. These performances also reflect the high diversity of vocal personifications during direct speech as well as different recording conditions. The most impressive classification rates are reached with ∆OS192 for Idle/Anger (77.7%) and Anger/Disgust (81.8%) emotion pairs. Is seems that the acoustic dynamics captured by this feature subset is very relevant for these two emotion pairs. With ∆ features, classification rates drop compared to non-∆ features on other pairs of emotions.

Regarding the results obtained with the small OS24 feature subset, classification between non emotional (Idle) and emotional segments is over 60% (bold font in Table 4.8) for Anger, Sadness, Fear and Disgust. By analyzing the Table 4.8, we can distinguish two emotion groups:

• Idle/Joy (58.0%), Idle/Surprise (56.3%) and Joy/Surprise (56.7%)

• Sadness/Fear (53.0%), Sadness/Disgust (54.8%), Sadness/Anger (56.7%), Fear/Disgust (58.0%), Fear/Anger (57.8%) and Anger/Disgust (58.0%) assume that the lexico-semantic properties and the syntax constraints have an essential impact on speech production. This assumption seems to be trivial because, beyond the speaker style and acoustic speech parameters, the lexicon-semantic and syntactic properties, which are mostly preponderant. This information is also independent of the speaker.

Therefore, the main objective of this exploratory work is to analyze if there are correlations between lexicon-semantic properties of the read text and its acoustic parameters.

To achieve this objective, we propose to rely on natural language processing and sentiment analysis techniques [START_REF] Mohammad | From once upon a time to happily ever after: Tracking emotions in novels and fairy tales[END_REF][START_REF] Medhat | Sentiment analysis algorithms and applications: A survey[END_REF][START_REF] Mohammad | Sentiment Analysis of Mail and Books[END_REF][START_REF] Chaffar | Using a heterogeneous dataset for emotion analysis in text[END_REF] to characterize and study the lexico-semantic and syntactic properties of texts corresponding to the dataset presented previously (cf. Table 4.2).

Proposed Method

In this work, we suggest to use unsupervised learning methods in order to avoid the manual annotation process of the written text. This study is articulated in three stages, as illustrated in Figure 4.4. In the pre-processing stage, We present the process that we used to make the raw text usable afterward. Then, in the clustering stage, we combine a dimension reduction technique and clustering technique in order to find the best number of clusters that meet a couple of predefined criteria, and the final stage is dedicated to acoustic speech features visualization and interpretation that correspond to the clustered text.

Pre-processing stage

Data cleaning and representation

Each sentence is segmented into substrings (or Tokens which do not necessarily correspond to words but rather to a sequence of characters.) using whitespaces as a delimiter. This process called tokenization is provided by Natural Language Toolkit (NLTK) [START_REF] Loper | NLTK: the natural language toolkit[END_REF]. This toolkit also furnishes text segmentation into sentences based on appropriate rules of targeted language, for instance, French. These rules are most of the time based on punctuation. Each sentence is in the form of a series of tokens (or terms). This form cannot be computed, so it must be converted to computational form using the doc2vec model. Each sentence will be represented as a set of numerical features that will be later selected and extracted.

Sentence representation

The numeric representation of the text documents, paragraph or sentence is a challenging task in Natural Language Processing (NLP). For this task, we use document embedding, which are high-dimensional continuous vector representation of document, in our case we consider sentence as document. In this vector space, sentence that have similar distributions are closer together than sentence with different distributions, given some distance measure. This type of setup has been shown to capture relevant syntactic and semantic properties of sentences, and they have been successfully applied to various tasks [START_REF] Collobert | A unified architecture for natural language processing: Deep neural networks with multitask learning[END_REF][START_REF] Socher | Semi-supervised recursive autoencoders for predicting sentiment distributions[END_REF][START_REF] Mikolov | Exploiting similarities among languages for machine translation[END_REF]. For converting sentence to numerical vector, among the existing methods for sentence numerical representation, we can mention sparse word features, word2vec word vector averaging, and doc2vec [START_REF] Mikolov | Exploiting similarities among languages for machine translation[END_REF][START_REF] Le | Distributed representations of sentences and documents[END_REF]. We choose to use doc2vec, according to [START_REF] Lau | An empirical evaluation of doc2vec with practical insights into document embedding generation[END_REF]. We use the software [START_REF] Rehurek | Software framework for topic modelling with large corpora[END_REF] because it is easy to implement.

To build the doc2vec model required for generating the embedding sentence vectors, we use the entire text transcription of the SynPaFlex-Corpus. This data has been preprocessed and cleaned, and we have kept the first 5 million words. We trained the model on this dataset using an embedding size of 300. The systems use the publicly available doc2vec5 implementation of the skip-gram model with negative sampling, and they were trained for 15 epochs with a window of 5 words.

Emotional Scoring Vector

For now, each sentence is represented by numerical vector of size of 300. In order to estimate the impact of emotion text labeling we propose to aggregate doc2vec (doc2vec) sentence vectors with a emotional score vector. To label each sentence in terms of emotions, we propose to use the FEEL [START_REF] Abdaoui | Feel: a french expanded emotion lexicon[END_REF][START_REF] Nzali | FrenchSentiClass: un Système Automatisé pour la Classification de Sentiments en Français[END_REF]] corpus to determine the word score (emotional score as well as polarity score) and then infer sentence vector emotional scoring. To do so we suggest to calculate the emotional scores of each sentence using the equation Equation (4.1).

E emo = n i=1 emo_score i n (4.1)
Where n is the number of words in sentence, and emo ∈ { joy, fear,sadness, anger, neutral}, by aggregating the scores E emo , we have an emotional score vector that represent a sentence.

Features Selection

Note that during the experiment, we investigate two configurations. For the first configuration, the feature vectors of given sentence s (i) are composed only with the embedded feature vector made using doc2vec. In the second configuration, we aggregate the embedded feature vector and the emotional scoring vector together.

The k-means clustering method is quit sensible to the dimension of the input data, for that reason, we choose to use Principal Component Analysis (PCA) as dimension reduction method. This configuration PCA+K-means [Ding and [START_REF] Cristo | Adaptive dimension reduction using discriminant analysis and k-means clustering[END_REF]] is widely used for sentiment and text clustering.

Clustering Stage

For clustering the embedding sentences vectors generated with doc2vec model, we investigate K-means [START_REF] Hartigan | Algorithm AS 136: A k-means clustering algorithm[END_REF][START_REF] Kanungo | An efficient k-means clustering algorithm: Analysis and implementation[END_REF] clustering technique because it is commonly used for text clustering [START_REF] Jing | Subspace clustering of text documents with feature weighting k-means algorithm[END_REF][START_REF] Spangler | Method for adapting a K-means text clustering to emerging data[END_REF]], data mining [START_REF] Riaz | Opinion mining on large scale data using sentiment analysis and k-means clustering[END_REF]] and sentiment analysis task [START_REF] Orkphol | Sentiment Analysis on Microblogging with K-Means Clustering and Artificial Bee Colony[END_REF]. The process of the K-means clustering algorithm is simple. It consists of first to randomly generating K centroids in the data points space, where K corresponds to the number of clusters.

These centroids represent the initial positions for each cluster, and their number has to be specified. The position of these K centroids are optimized (adjusted) iteratively following these steps:

1. Calculate the sum of squared distance between data points and centroids.

2. Reassign each data point to the cluster closer than other clusters (centroid).

Update the position of centroids of clusters by taking the average of all data points

of that cluster.

The optimization process of the centroids ends when:

• The centroids have reached their stable position; no changes in their position are possible anymore.

• Alternatively, the number of iterations is achieved.

To find the optimal K number clusters, we used silhouette analysis.

Select the number of clusters by using silhouette analysis

Silhouette analysis is a graphical tool for interpretation and validation of cluster analysis by measuring how close each data point is in a cluster compared to other data points in its neighboring clusters.

To select the number of clusters we have applied the same constraints as the one described in [START_REF] Li | Sentiment analysis based on clustering: a framework in improving accuracy and recognizing neutral opinions[END_REF], which are:

-First, the average silhouette coefficients should be close to one as much as possible.

-Second, the plot of each cluster must be above the average of the silhouette coefficients as much as possible.

-Third, the thicknesses of all the clusters must be uniform as much as possible.

-After K-means has been run many times with different numbers of clusters (K), the best number of clusters will be selected based on previous aspects. The result of K-means with the optimal number of clusters will be interpreted and acoustically analysed in the next subsection.

Acoustic Analysis

To estimate the correlation of text with corresponding acoustic speech features, we propose analyzing and visualizing the acoustic data of the corresponding text. For this task, we follow three steps. First, we extract the acoustic features of each utterance extracted on emotional segments with OpenSmile toolkit and Interspeech 2009 configuration[95], as described in Section 3. Then, we apply PCA to reduce the dimension of features, and finally, we use T-distributed Stochastic Neighbor Embedding (t-SNE) [START_REF] Maaten | Visualizing data using t-SNE[END_REF] for data visualization.

Experiments and Results

There are two experiments being conducted. The first experiment is to verify whether clustering k-means method is affected by the input features. The second experiment is to find the optimal number of clusters (K) using silhouette analysis. The third experiment is to visualize the acoustic data corresponding to the resulting clusters and interpret the result.

Data setup

For this work we choose to use the same audiobooks that we used in Section 3.1. Because we assume that this set of data has potential emotional contents. We split each text into sentences using NLTK [START_REF] Loper | NLTK: the natural language toolkit[END_REF]] toolkit, at the end of the process we obtained 13384 sentences.

Clustering results and Analysis

Table 4.9 shows two candidates (one for each configuration) produce good results. K = 18 in doc2vec configuration and K= 7 for the second feature vectors configuration doc2vec + emotional vector scoring were selected because they achieve the best trade-off in terms of Silhouette average coefficient and balanced number of samples per group.

According to this result, it seems that the emotional scoring vector reduces the number of groups needed to represent all the data. As these results are too preliminary, we cannot explain or analyze the results obtained. An in-depth analysis of the content of the groups will allow us to bring more precision and a solid explanation of the present findings. Table 4.9: The best K-clusters according to the silhouette average criteria and average samples per cluster

Visual analysis of acoustic data

Figure 4.7 shows the projection of the acoustic data corresponding to the seven groups from the K-means analysis.

The contrast between the groups is low. It can also be observed that there is a correlation with the results obtained at the textual level, which is quite encouraging but not enough and need further experiments to understand data behaviors both in terms of acoustic and linguistic.

Discussion and issues

For clustering sentences, we used PCA + K-means. There are many issues related to this method. The coverage of PCA components does not exceed 50%, which means that we loose a lot of information from the initial features representation. The second issue is with K-means algorithm initialisation, where the initial clusters position are randomly assigned. This kind of initialization method is problematic regarding to the distribution of the data points. As future work, we propose to investigate Adversarial autoencoders [START_REF] Makhzani | Adversarial autoencoders[END_REF] as dimensional reduction method instead of PCA and Artificial Bee Colony (ABC) [START_REF] Krishnamoorthi | ABK-means: an algorithm for data clustering using ABC and K-means algorithm[END_REF][START_REF] Armano | Clustering Analysis with Combination of Artificial Bee Colony Algorithm and k-Means Technique[END_REF] algorithm for initializing of the K-means.

Conclusion

In this chapter, we have tried to make a quantitative analysis of the emotions contained in audiobooks. We have proposed a rather rich emotional annotation protocol as well as a baseline for future work. Indeed, the experiments carried out do not take into account the variability between speakers; we focused on a single speaker. In this work, we consider emotions as a category, thus neglecting the possibility of continuous representation with parameters such as Valence, Arousal and Dominance.

In our analysis of the text, we also omitted to take into account the phonetic and phonological transcriptions of the text, which would undoubtedly have brought possible explanations to the results. Through this preliminary work, we found that the analysis of the emotions and expressiveness carried by audiobooks is complex.

Introduction

In [START_REF] Durrer | Le modelage de la séquence dialoguée[END_REF][START_REF] Dominique | Analyser les textes de communication[END_REF][START_REF] Perret | L'énonciation en grammaire du texte[END_REF]] the structure of texts such as novels, short stories, and tales, are modeled with four types of discourse: the Direct Discourse (DD), the Indirect Discourse (ID), the free indirect discourse (FID), and the narrative discourse(ND). In [START_REF] Durrer | Le modelage de la séquence dialoguée[END_REF]], the author has illustrated this model in the case of french novels.

An example, which shows the combination of the four discourse types: all three as in (4.b)

The analysis and annotation of the discourse in literary works are of interest to characterize the expressiveness carried by audiobooks.

To synthesize audiobooks in satisfactory and expressive manners, it is essential to be able to indicate any modification in the enunciative perspectives(characters changes in dialogue sequence) by prosodic markings comparable to those observed in real speech [START_REF] Doukhan | Prosodic Analysis of a Corpus of Tales[END_REF][START_REF] Montaño | Prosodic analysis of storytelling discourse modes and narrative situations oriented to Text-to-Speech synthesis[END_REF].

For achieving a better expressivity, it is necessary to classify (paragraph means sequence separated by line breaks in the text) according to their type of discourse, and from there to have a precise idea of who speaks. All these reasons lead to classifying the paragraphs according to whether they correspond to Indirect Discourse (ID) (1), Direct Discourse (DD) (2), or mixed passages. In some paragraphs, reported discourse is inserted in the middle of narrative passages (3) or Incidental Clauses with reporting verbs (IC), which can be short (4a) or relatively long ( 4b). In these mixed cases, the task is to delimit precisely the types of speech present.

(1) On commença la récitation des leçons. Il les écouta de toutes ses oreilles, attentif comme au sermon, n'osant même croiser les cuisses, ni s'appuyer sur le coude, et, à deux heures, quand la cloche sonna, le maître d'études fut obligé de l'avertir, pour qu'il se mit avec nous dans les rangs.[We began reciting our lessons. He listened attentively, concentrating as though listening to a sermon, not daring even to cross his legs or lean on his elbow, and, at two o'clock, when the bell rang, the master had to tell him to line up with us all.] (Madame Bovary, chap. 1)

(2) -Soit, demain à une heure.

-A une heure.

-Dans la plaine Saint-Denis? -Dans la plaine Saint-Denis.

- While the detection of narrative passages (1), dialogues (2) and discourses related to the middle of narrative passages (3) in the mixed paragraphs may seem rather trivial, due in particular to typographical indications, the annotation of incidental citation is more complex, as evidenced by a simple comparison between cases (4a) and (4b). The presence of a comma after the citation is not enough.

The main objective of this work is to design an annotator for parsing french audiobook text in terms of discourse type. This chapter is organized in four sections. The Section 2 present the experimental data set and the annotation protocol. Then the Section 3, present the rule based annotator, followed by Section 4 which contains the machine learning approach, and finally a conclusion in Section 5.

Corpus and material

For this study, we worked on a subset from the SynPaFlex-Corpus presented in Section 1. This dataset corresponds to chapters of two French novels, Les Mystères de Paris by Eugène Sue and Madame Bovary by Gustave Flaubert. An expert chose these excerpts because they contained many changes in discursive and enunciative perspectives, and, at the same time allowed arriving at a relatively coherent set in terms of sequences with direct and indirect discourses, as shown in Table 5.1. For the whole corpus and the sub-part selected for this study, we have the orthographic transcription, phonetization, and alignment to the sound signal done automatically using JTrans [START_REF] Cerisara | JTrans, an opensource software for semi-automatic text-to-speech alignment[END_REF]. Other linguistic annotations of a phonological nature (syllable division, etc.) and morpho-syntactic (grammatical categorization of words, analysis, and indication of grammatical functions) are also available through the use of automatic annotation procedures [START_REF] Candito | Benchmarking of statistical dependency parsers for french[END_REF][START_REF] Candito | Analyse syntaxique du français : des constituants aux dépendances[END_REF]]. The entire annotation process was conducted using ROOTS [START_REF] Chevelu | ROOTS: a toolkit for easy, fast and consistent processing of large sequential annotated data collections[END_REF], which allows all annotations to be maintained consistently. The automatic annotation of discursive and changes for a given text (a chapter in our case) is done in two phases, illustrated in the following subsections:

Paragraph typology

-Classification of paragraphs according to the types of speech, in order to put aside all paragraphs with IC;

-Detection and delimitation of quotation marks (he says, and so on) and primers (he said: "...", and so on).

Classification of paragraphs according to types of speech:

From the text, the program classifies paragraphs, defined on a typographic basis (line break), into three distinct groups (see Figure 5.1, Phase 1). It is based on lexico-syntactic criteria (presence of speech verb, and other patterns), as well as on punctuation and typographic signs (presence of quotation marks or hyphens).

-the group DD gathers all the paragraphs which contain passages exclusively in direct speech as in the example (2);

-the group ID: contains paragraphs with only narration passages or descriptions as in example (1); -The Mixed Discourse group is composed of paragraphs that may contain both DD and indirect speech or narration. Will be present in this group both the narrative passages in which are inserted reported speeches as in example (3) and passages to the direct speech including Incidental Clauses with reporting verbs (IC) as in example (4)

In a second phase (see Figure 5.2, Phase 2), the paragraphs of the Mixed group are analyzed to determine the boundaries of discourse. This task is performed using expert rules. This step will make it possible to identify the Direct Discourse (DD), often in quotation marks and preceded by two points (3), Incidental Clauses with reporting verbs (IC) like ( 4), and the sequences introducing a dialogue like (5). Among these elements, IC are essential because they make it possible to delimit changes of characters and to provide indications on the characters present and their attitudes.

Detection and annotation of Incidental Clauses with reporting verbs (IC)

At the end of the first classification phase, the Mixed paragraphs are analyzed in detail to determine the boundaries of the different types of speech. After the phonetization and forced alignment of the chapter text with the corresponding audio file, the data are segmented into paragraphs/ pseudo-paragraphs and stored relying on roots toolkit. The segments follow two annotations process: (i) the manual annotation made by an expert (ii) the automatic annotation which has two phases, the first phase consists of labeling the segments according to typographic criteria as DD, ID, and mixed group. The mixed groups are processed in phase 2 ( Figure 5.2) in order to fine-tune the annotation and label the group according to DD, ID, and IC. The mixed groups annotated, and non-mixed groups form the automatic annotation sequence. The two annotations (manual and automatic) are fused to form the Annotated Corpus (AC). The method implemented to detect IC is initially based on the work of [START_REF] Mareüil | Traitement des incises en français: capture automatique et modèle prosodique[END_REF], which consists of a set of regular expressions.

Based on this work, we add a set of rules that aims to detect IC in a more detailed way, and to cover more complex cases by relying on the syntactic analysis of the IC described by [START_REF] Bonami | Syntaxe des incises de citation[END_REF] and [START_REF] Danlos | Analyse discursive des incises de citation[END_REF]. In our study, we can distinguish three configurations:

-Direct Discourse primers as in the example (5); -Incidental Clauses with reporting verbs located in the middle of the speech of a character (4a);

-Incidental Clauses with reporting verbs placed at the end of the words of a character (4b).

We add to these three configurations, the cases where: a DD is inserted into an indirect speech in a sudden manner that is to say without a primer or other indication of a change of discursive perspective (see the example (3) )

For analyzing these different configurations, it is necessary to look at other elements than just punctuation or dashes. In the proposed approach, we take into account both the result of the parser [START_REF] Candito | Analyse syntaxique du français : des constituants aux dépendances[END_REF]] and a lexicon of 327 reported verbs (affirm, repeat, exclaim, say, and so on). The entire list of reported verbs is in Appendix 1 .

Usually, when reporting verbs conjugated to the third person of the singular (in 97% of the cases), are detected, we have to have a look at their subject, in order to know its position compared to the verb. Two cases arise: if the subject is on the left of the verb (before), it is a primer; if, on the contrary, it is after the verb, we are dealing with an IC. In this case, it is crucial to establish its extension. The IC can be short (as (4a)) or relatively long (see (4b)). For doing this, we rely on punctuation, but also on parsing, especially for elements on the right of the verb that may depend on the subject as in the case of apposition or relatives (see example ( 4b)). The complete process is illustrated in Figure 5.2.

Rule-based results

The results obtained by this algorithm for detecting speech types are given in Table 5.2. Performance is estimated with three measurements: precision, recall, and F1-score. The algorithm allows proper classification of the paragraphs (92.19% of good detection or F1-score) at the end of phase 1. For the analysis of the incidental citation (801 annotated manually on the corpus), performance is calculated by distinguishing two levels of annotation. Simplified detection -which is based on taking into account speech verbs and punctuation (see Figure 5.1, phase 2) does not allow to delimit with precision the extension of the IC (F1-score: 86,3%), errors appearing when a relative or an apposition depends on the subject. Taking into account the syntax and dependencies as shown in Figure 5.2 allows, on the other hand, to refine the results and improve them, so that we reach, after this precise detection, a score of 89.09% (this last one including the type of speech and the extent of the IC).

A study of the errors makes it possible to isolate two main cases:

-Those where the verb takes the form of a participle, and not of a finite verb, as in the example ( 6). The results obtained are not as good as those presented in direct and indirect discourse classifications based on machine learning algorithms (see [START_REF] Schöch | Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels[END_REF]] who obtain an F1-score of 93.9% using a Random Forest (RF). However, this difference has to be taken with care because the objectives are not precisely the same. The procedure developed by [START_REF] Schöch | Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels[END_REF]] aims to say whether each sentence is direct or indirect discourse, but does not isolate IC insights, primers, or direct speech passages in a narrative sequence. A fundamental difference of objectives explains this: whereas [START_REF] Schöch | Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels[END_REF] wants to classify the works on literary bases by relying on the presence or not of direct speech, we wish to know who speaks and at what precise moment change occurs. Moreover, the differences of results can be explained by the chosen method: whereas [START_REF] Schöch | Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels[END_REF] takes as a basic unit the sentence, we take the paragraph to indicate any discursive change in the same paragraph.

Machine learning approach

We extend the above work to explore procedures similar to those retained by [START_REF] Schöch | Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels[END_REF]], but keeping the same objectives, namely determining precisely where discursive changes occur.

By observing more extensive data than those used Section 3, we observe that the identification of Direct Discourse (DD) , Indirect Discourse (ID),and the Incidental Clauses with reporting verbs (IC) less trivial using rule-based techniques than it seems to be since in French typographical DD is not marked with opening and closing quotation marks (example 8). By applying the above-described rule-based algorithm on new data, a different author, and written differently, we realized that rule-based algorithms have several disadvantages in other respects:

-Conflict between rules, some rules have to follow a specific order.

-When the size of the data increases, it is more likely to have additional cases not covered by the crafted rules. We need to increase the numbers of rules, which can be problematic because, at some point, the designed rules will become deprecated.

-It is difficult to generalize the procedure to other languages than the one for which the rules were designed.

To solve these problems, we propose to rely on machine learning to automatically identify the discourses in a larger collection of French-language fictional. We assume that there is enough data to build a robust model. We also hypothesize that there are enough linguistic markers that make distinguishable the three discourse types of Direct Discourse (DD),Indirect Discourse (ID), and Incidental Clauses with reporting verbs (IC).

General Methodology

We consider the rule-based approach proposed above as a baseline for discourse classification task. We have three distinct classes DD, ID, and IC. From the manuel annotated data, we separated the text related to each class.

To carry out this study, we have mainly relied on the work presented in [START_REF] Kowsari | Text classification algorithms: A survey[END_REF], which presents a survey of recent techniques dedicated to automatic text processing and classification.

Data used and feature extraction

For this work we have used the annotated extracts in Section 2 the text were preprocessed to perform supervised learning text.

In text classification paradigm, the word is the atomic element of an utterance (sentence, paragraph, and document). There are several methods for representing a word in sentence [START_REF] Kowsari | Text classification algorithms: A survey[END_REF]. For this task, we use word embedding, which are highdimensional continuous vector representation of words. In this vector space, words that have similar distributions are closer together than words with different distributions, given some distance measure. This type of setup has been shown to capture relevant syntactic and semantic properties of words, and they have been successfully applied to various tasks [START_REF] Collobert | A unified architecture for natural language processing: Deep neural networks with multitask learning[END_REF][START_REF] Socher | Semi-supervised recursive autoencoders for predicting sentiment distributions[END_REF][START_REF] Mikolov | Exploiting similarities among languages for machine translation[END_REF].

To learn these embeddings, we have used freely available text data of SynPaFlex-Corpus. This data has been preprocessed and cleaned, and we have kept the first 5 million words. We trained the model on this dataset using an embedding size of 100. The systems use the publicly available word2vec 2 implementation of the skip-gram model with negative sampling, and they were trained for 15 epochs with a window of 5 words. graphical plot of the performance is shown by ROC curve. This curve considers only true positive rate and false positive rate of the testing data. Here recall show the true positive rate, precision gives the positive predictive value, f-measure (Espindola and Ebecken, 2005) is the harmonic mean of precision and recall. The performance of the proposed methods is evaluated using confusion matrix, ROC curve, precision, recall, f1-score and accuracy. A graphical plot of the performance is shown by the ROC curves. This curve considers only true positive rate and false positive rate of the testing data. Here recall shows the true positive rate, precision gives the positive predictive value, f1-score [Espíndola and Ebecken 2005] is the harmonic mean of precision and recall. Table 5.3 presents the results of the discourse mode classification. RNN outperformed all the other models with an accuracy of 86.08%. DD mode classification is 91% because of more training data for this mode, and IC mode classification is 81%, and ID mode classification is 79% because for these classes we have fewer data to train our model. The results obtained with the RNN model outperforms other algorithms. The reported results indicate that almost all models show high-performance rates when it comes to predicting direct speech, incidental citation , and low rates when it comes to the narration. These results are probably due to the fact that the narration contains a large vocabulary as well as utterances with a different organization unlike the DD and the IC, which for most of them are short, even very short for some samples. In addition, the syntactic structure of the incidental citations and the direct speech are quite consistent, which is in line with the results found in the first part based on rules.

Conclusion

In this chapter, we have proposed an algorithm that automatically and accurately annotates discursive changes in audiobooks. The performance of the tool is relatively encouraging, but errors remain in syntactically complex cases. The rule-based approach still useful even though it does not generalize well, and it requires knowledge. We can use this approach to validate a given rule and extract robust and valuable features for machine learning approaches.

In the next chapter, we rely on the audio signal to have a better understanding of discourse changes in general and to better delineate the Incidental Clauses with reporting verbs (IC) in complex cases.

Chapter 6

Automatic prosodic analysis of discourse changes in audiobook

Emphasizing on discourse and character changes is very important to improve expressivity in text-to-speech synthesis (TTS) systems reading audiobooks. It makes stories easily understandable by listeners with a lower cognitive effort and it enables a better access to the exact content. Understanding which prosodic cues are used to this end is thus relevant as they could then be implemented to enhance speech quality and expressiveness in TTS systems.

This work aims to investigate how discourse and character changes occur and which robust prosodic patterns are able to encode them. For this exploration, focus was given to literary fictions which contain a considerable amount of dialogues and similar narrative schemes. Moreover, each selected story was recorded by two speakers allowing a comparison between speakers and styles.

For the prosodic analysis, the data were first segmented into breath groups, then five main prosodic features were automatically analysed: (i) F0 range (in semi-tone, because semitones are more suitable for measuring temporal events and to notify relative variations in a sequence.), (ii) articulation rate (syllables/sec), (iii) breath group duration, (iv) average log energy, and (v) inter-breath group pause duration.

The results obtained from a statistical analysis show that speakers mainly employ inter-breath-group pause duration and F0 range to encode discourse and character changes.

Introduction

To achieve a good synthetic voice in terms of expressiveness and naturalness, a deeper understanding of natural speech is required. In recent years, although the use of data-driven techniques, together with appropriate data, enhances Text-to-Speech systems, there is still a gap between natural speech and synthesized speech. It becomes even more obvious while reading long and coherent texts such as audiobooks aloud. However, many works have been done to tackle this problem either in traditional or end-to-end approaches. Traditional approaches explore prosodic and linguistic features and enrich feature vectors [START_REF] Székely | Evaluating expressive speech synthesis from audiobooks in conversational phrases[END_REF]Székely et al. 2012a;[START_REF] Mamiya | Lightly supervised GMM VAD to use audiobook for speech synthesiser[END_REF][START_REF] Charfuelan | Expressive speech synthesis in MARY TTS using audiobook data and emotionML[END_REF]Vít and Matou?ek 2016]. [START_REF] Sarkar | Analysis and modeling pauses for synthesis of storytelling speech based on discourse modes[END_REF] for instance explore pause prediction problems using linguistic information such as discourse types. On the other hand, end-toend paradigms [Wang et al. 2017a;[START_REF] Ping | Deep voice 3: Scaling text-to-speech with convolutional sequence learning[END_REF][START_REF] Tachibana | Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention[END_REF], which avoid complex feature engineering process and learn from raw data, are data sensitive and are affected by the quality and quantity of the data to be treated. They are also computationally expensive.

One of the main challenges of prosody modeling is that there is considerable inter-and intra-speaker variability. We make two different assumptions; however: (i) all speakers assign different prosodic properties to encode a specific style or one of the characters involved in the story; (ii) discourse and character changes are always encoded, but, because of variability, different strategies and acoustic features may be used. These variabilities are not always integrated into the speech, some studies have shown that there are somewhat similar styles (spoken newspaper, neutral reading) and others which are not (political speech, slam); therefore, this inter-speaker variability depends on text genre.

Certain literary genres, such as storytelling [START_REF] Sarkar | Analysis and modeling pauses for synthesis of storytelling speech based on discourse modes[END_REF][START_REF] Theune | Generating expressive speech for storytelling applications[END_REF][START_REF] Buurman | Virtual Storytelling: Emotions for the narrator[END_REF][START_REF] Montaño | Prosodic analysis of storytelling discourse modes and narrative situations oriented to Text-to-Speech synthesis[END_REF][START_REF] Harikrishna | Multi-stage children story speech synthesis for Hindi[END_REF][START_REF] Ramli | Prosody Analysis of Malay Language Storytelling Corpus[END_REF][START_REF] Montaño | The Role of Prosody and Voice Quality in Indirect Storytelling Speech: Annotation Methodology and Expressive Categories[END_REF], have received much attention, most of which have analyzed discourse at the sentence level. Indeed, this granularity is not informative enough in the case of long textbooks, such as novels or short stories. It is mainly due to the fact that those texts are more complex and subtle along various dimensions such as syntactic structure, lexis, discourse patterns, but also character psychology. In this work, we thus decided to study French fictional stories addressed to adults and analyze how the different discourse types and character changes were encoded prosodically.

The current work investigates natural speech with the use of different automatic prosodic annotation procedures which can be compared to the ones used in tools such as SLAM [START_REF] Obin | SLAM: Automatic Stylization and Labelling of Speech Melody[END_REF], ADoReVa & ADoTeVa [START_REF] Looze | Detecting changes in key and range for the automatic modelling and coding of intonation[END_REF], MOMEL & INTSINT [START_REF] Hirst | A Praat plugin for Momel and INTSINT with improved algorithms for modelling and coding intonation[END_REF]]. It allowed focusing on the way certain prosodic features change over time. In order to generate adequate prosodic patterns in a TTS system, it is important to know exactly which prosodic cues come into play for indicating any change in the discursive perspective.

To analyze discourse changes in fictions, it is necessary to clearly delineate different types of speech in the text. This allows getting a more precise idea of "who speaks'.' In addition, it is important to study how these changes are encoded by prosodic cues, despite inter and intra-speaker variability. We have thus investigated breath group modifications, in terms of articulation rate and pitch range, in the particular case of discursive perspectives (from direct to indirect speech), or character changes (in dialogs) based on the fact that breath group is a good unit to study continuous speech in both read and spontaneous speech according to [START_REF] Wang | Breath group analysis for reading and spontaneous speech in healthy adults[END_REF]].

One of the goals of this study is then to integrate the observed prosodic cues into a speech synthesis system to improve its quality and expressiveness. To reach this end, we did refer to work that has already been done on discourse properties in French. There has been a growing interest in studies focusing on the parsing of incidental clauses with reporting verbs [START_REF] Buvet | Traitement automatique du discours rapporté[END_REF]] or on the semantic and syntactic values of such sequences in discourse [START_REF] Beyssade | Le statut sémantique des incises et des incidentes du français[END_REF]]. According to [START_REF] Buvet | Traitement automatique du discours rapporté[END_REF]], incidental clauses are characterized, in French read speech, by syntactic features as well as by a specific prosodic behavior. This chapitre firstly describes the corpus and methods. It further explains how the different discourse perspectives are encoded and how the prosodic features were analyzed. Finally, the results of the experiment are presented and discussed.

Corpus Design

Experimental dataset

This present work is concerned with highlighting prosodic cues, and the prosodic unit used for making discourse change in fictional audiobooks. This investigation was conducted on audiobook samples recorded by two female speakers (FFR0012, FFR0001 of MUFASA corpus). This subset includes extracts selected from two French novels, les Mystères de Paris by Eugène Sue and Madame Bovary by Gustave Flaubert. The selected extracts involve many discursive perspective changes, which allows getting a relatively coherent set in terms of direct and indirect discourse sequences. The Madame Bovary extracts are read by both speakers, where les Mystères de Paris extracts, is read-only by the principal speaker (FFR0001). The table Table 6.1 shows the details of the studied dataset.

Preprocessing

All audio data used for this experience are in wav format, with a sampling rate of 44.1kHz, mono channel, 16 bits. To obtain aligned and consistent data, the following steps were taken during prepossessing phase:

• All the speaker dependent data such as introductions and conclusions were removed;

• For reducing the potential background noise, we considered long noisy silences as a typical profile of the noise;

• The DC offset has been removed; in some collected data, there is fixed voltage offset inserted during the recording process, this offset can affect the quality of recordings; for that reason, it is preferable to remove this offset.

• Amplitude has been normalized to avoid clipping.

Text annotation

Using a rule-based program, presented in Section 3, texts were automatically annotated in order to distinguish the paragraphs consisting of simple narration (1) and the sequences corresponding to direct speech dialogs (2). The annotation procedure also allows to delimit the extension of direct speech passages within narrative paragraphs, i.e reported speech as in (3), and also parentheticals or incidental clauses with reported verb in direct discourses, which can be short (4a) or relatively long (4b), and located in the middle of a direct speech sequence (4a) or at the end of it (4b).

(1) On commença la récitation des leçons. Il les écouta de toutes ses oreilles, attentif comme au sermon, n'osant même croiser les cuisses, ni s'appuyer sur le coude, et, à deux heures, quand la cloche sonna, le maître d'études fut obligé de l'avertir, pour qu'il se mit avec nous dans les rangs. [We began reciting our lessons. He listened attentively, concentrating as though listening to a sermon, not daring even to cross his legs or lean on his elbow, and, at two o'clock, when the bell rang, the master had to tell him to line up with us all.] (Madame Bovary, chap. 1)

(2) -Soit, demain à une heure.

-A une heure.

-Dans la plaine Saint-Denis? -Dans la plaine Saint-Denis.

-Entre Saint-Ouen et le chemin de la Révolte, au bout de la route? -C'est dit.

[-Be it so; tomorrow at one. This work allowed to delimit with precision (87%) the passages according to their discourse type: direct discourse (DD), indirect discourse (ID) and incidental clauses with reporting verbs (IC). This allowed to distinguish six cases of discursive perspective changes:

-from indirect discourse (or narration) to direct discourse, noted DI⇒ DD; -from direct discourse to indirect discourse (or return to narrative paragraphs), noted DD⇒ DI; -from direct discourse to incidental clause with reporting verb, as in the transition from levez-vous to dit le professeur in (4a), noted DD⇒ IC;

-from incidental clause to direct discourse, as in dit le professeur to et dites-moi votre nom in (4a), noted IC ⇒ DD;

-from incidental clause with reporting verb to the indirect discourse (or narration), as in (4b), noted IC⇒ ID.

-from direct discourse to direct discourse, with a character change (dialog sequence), as in (2), noted DD⇒ DD.

The distribution of discursive perspective changes taken into account for the analysis of prosodic parameters are given in details in Table 6.2. 3 Prosodic analysis

Features Extraction

As we mention in Section 1, many tools are available to analyze prosody and extract relative features from different perspectives. We decided to build our framework to have more control and easily interpret the results. Also, we considered that the new framework corresponds better to the nature of the data that we are analyzing. The prosodic analysis is based on the breath group granularity (as a unit function) and focused on five cues: F 0 -range (in semi-tone), articulation rate (syllables/sec), average vowel lengthening rate, average vowel log energy, and pause duration at the juncture between breath groups. In addition, the analysis of these prosodic cues was first carried out on a subpart of our corpus, and then validated on the whole data set. To analyze the various prosodic features, the last breath group of a given discourse type (DD, ID) and the first breath group of the targeted discourse in case of change were taken into consideration. Thus, to analyze changes from direct discourse to incidental clause as in (4a), the values for F0-range were calculated for the last breath group of the direct discourse sequence, i.e. Levez-vous in (4a), and the first breath group of the clause, i.e. dit le professeur in (4a), according to the equation (6.1), where M represents the number of vowels within the breath group and V F 0median stands for the median vowel F 0 value within the breath group.

           F 0min = arg min( M i=0 V F 0median ) F 0max = arg max( M i=0 V F 0median ) F 0range = 12 × log 2 ( F 0min F 0max ) (6.1)
The same procedure was followed to study articulation rate AR, in syll/s, the latter being computed for the last breath group (BG) of the first discourse sequence and the first one in the second discourse sequence. The articulation rate is computed as follows:

AR = N N i=0 Syllable Duration[i] (6.2)
where N is the number of syllables in a given breath group.

The analysis of inter-breath groups pause duration in the studied data has shown that the speakers tend to insert breaks upper to 200 ms (0.2s) to mark the transition from discourse to another. These phenomena appear to be independent of the articulation rate of the surrounding speech segments.

For each breath group, the average log energy is computed over the set of extracted log energy of its vowels.

logEnergy = M i=0 V logEnergy M
Where M refer to the number of vowels in the breath group.

Since the duration of the breath group has a relative variability, we have also measured duration of each breath group.

A statistical analysis of significance has been done using a χ 2 test to analyze discourse changes impact. The six configurations have been tested for the different features, namely F 0 -range, articulation rate and pause duration, at α=0.01 level. 88

Hypothesis

In this work, we investigate two hypotheses:

-The first hypothesis is that both speakers mark discourse and character changes by using a particular prosodic properties, which can differ from one speaker to another.

-The second hypothesis is that discourse and character changes are local phenomena ; as a consequence, the way changes are encoded may differ within an entire novel, but differences between two consecutive breath groups should occur in a clear dynamic way.

Results and discussion

In this section, the results of the prosodic analysis outlined in Section 2.3 are presented. Table 6.3 presents the results obtained for one speaker. Among the five cues investigated, F 0 -range and pause duration play a role. By contrast, articulation rate, average log-Energy and breath group duration differences between two breath groups surrounding a discourse change are not significant for both speakers. Results even tend to show that articulation rate is quite stable among breath groups According to the results in Table 6.3, discourse changes have a significant impact on F 0 -range. Among the different cases investigated, a pitch range compression occurs on the first breath group after discourse change, except after an incidental clause (IC ⇒ DD or IC ⇒ ID). This could be related to the fact that incidental clauses are shorter and treated as embedded and autonomous at the syntactic and prosodic level. Concerning changes from DD to ID, F 0 -range difference is less significant (statistical significance at α = 0.05 level), but other parameters such as pause duration enter into play, as we will see later. Thus, the combination of prosodic features allows the correct encoding of discourse changes.

Average duration of inter-breath group pauses according to discourse change types are reported in Table 6.4. We can thus observe that pause duration is significantly lower when changing from an incidental clause (IC) to direct discourse (DD), and the other way around. This could result from the fact that incidental clauses are embedded in a larger group (e.g. [levez-vous (dit le professeur) et prenez. . . ]). In addition, one can notice that longer pauses are realized when returning to indirect discourse. When changing, for instance, from direct discourse (DD) to indirect discourse (ID) the average length of a pause is 1.26s, whereas the average pause duration is only 1.02s when introducing a direct discourse (ID to DD). Note however that this type of pauses, which indicates a move from Indirect Discourse to Direct one, is also among the longer ones (see Table 6.4).

Table 6.5 reports the significance level of the inter-breath group pause length difference when comparing the discourse changes two by two. It can be seen that the p-value < 0.001 is statistically significant in nearly all cases. Moreover, one can point out that if we compare IC⇒ ID to DD⇒ ID and IC⇒ ID to ID⇒ DD, the difference is less significant than in other cases. Furthermore, we have analyzed the behavior of the same prosodic cues for consecutive breath groups when the discourse type remains the same. Concretely, it corresponds to the following transitions: IC⇒IC, ID⇒ID and DD⇒DD with no character change. The results of this analysis show that there is no significant difference for the three features in that case. This finding shows that the reader adopts a specific strategy to mark discourse changes.

The whole analysis has been done for both speakers and the results are similar. This

Conclusion and perspectives

This study investigated how discourse changes, in a French audiobook corpus, are prosodically characterized. This study relies on five basic prosodic cues at the breath group level: F 0 -range, articulation rate, breath group duration and inter-breath groups pause durations. The results show that F 0 -range and pause durations are relevant features to differentiate two distinct and consecutive discourse types. This work has also confirmed that the breath group is an interesting functional unit for studying long and expressive speech in audiobooks.

A deeper investigation on the breath group structure and its relations to the prosodic cues needs to be done. To do so, a large set of features could be studied such as linguistic, phonetic, phonological and paralinguistic features.

We also plan to redo the experiment with the rest of the parallel data of the MUFASA-Corpus . For these future experients, we are considering other tracks for better understanding the discursive change phenomena and the prosodic properties of breath group:

• Investigating the direction, as well as the amplitude of the pitch curve on the last word during the discursive changes. These phenomena seem to be an essential and necessary measure to have more control over a micro-prosodic manifest of discourse changes.

• Analyzing the possible prosodic reset (F0 reset, for example) between the last syllable of the breath group and the first syllable of the next breath group.

• Studying the declination line, which seems to appear in the F0-range as a marker of the central prosodic unit, and then the analysis of possible declination lines within specific sequences.

Furthermore, integrating the observed results in a TTS system should allow designing perceptual tests and collecting subjective evaluations. Then studying the relation between the features and their impact on synthesized speech could lead us to build a solid knowledge on the prosodic behavior occurring in audiobooks. In addition, tested prosodic features could bring more expressiveness to speech synthesis systems and thus enable new applications.

almost like humans. In the meantime, however, the control over models remains challenging because of their complexity.

Expressive speech synthesis relies on adequate control on the prosodic parameters. These parameters depend on the linguistic features of the text to read as well as information related to the voice used for synthesizing the speech.

Therefore, disentangling the speaker characteristics from the linguistic content is a key feature to control the rendering of the synthesis.

Disentangling speaker characteristics from linguistic content is even more crucial to have proper control in multi-speaker Statistical Parametric Speech Synthesis as, by definition, the model should produce a speech corresponding to one consistent speaker. Counting on the robustness of multi-speaker modelling, studies show that expressive speech synthesis systems can benefit from such an environment [START_REF] Fan | Multi-speaker modeling and speaker adaptation for DNN-based TTS synthesis[END_REF], although it raises other challenges related to recording conditions [Hsu et al. 2019], speaker coding [START_REF] Hojo | DNN-based speech synthesis using speaker codes[END_REF] and controllability [START_REF] Henter | Deep encoder-decoder models for unsupervised learning of controllable speech synthesis[END_REF][START_REF] Hsu | Hierarchical generative modeling for controllable speech synthesis[END_REF][START_REF] Lazaridis | DNN-based speech synthesis: Importance of input features and training data[END_REF][START_REF] Bian | Multi-reference Tacotron by Intercross Training for Style Disentangling, Transfer and Control in Speech Synthesis[END_REF].

Therefore, we propose here to investigate whether a model can seperate speaker characteristics from linguistic features in a standrad DNN TTS multispeaker environnement by using a naive but fully controllable representation of prosody. This chapter is structured as follows. The different speaker coding configurations are presented in Section 3. Section 4 gives an overview of the methodology and Section 5 details the experiments we conducted to analyze the influence of these configurations on the model. Finally, in Section 6, we go through the results of the experiments using complementary objective analysis methodologies and subjective assessment.

Speaker Coding

To encode the speaker voice characteristics, we are using three different configurations from the most opaque (OneHot-Vector) to the most controllable one (P-Vector). The intermediate representation (X-Vector) has been added as it is a state of the art representation for the speaker identification domain.

OneHot-Vector

This configuration to encode the speaker information for DNN based speech synthesis has been explored in [START_REF] Hojo | DNN-based speech synthesis using speaker codes[END_REF]. As a first and intuitive choice for speaker encoding, we propose a simple one-hot vector of two parts: (1) gender (female/male) as this is the highest level distinction we can do, (2) identifier of the speaker to distinguish speakers intra-gender. This approach makes the control of the synthesis most complicated as we just have a discrete choice; thus it does not take into account the acoustic proximity between speakers.

X-Vector

X-Vector [START_REF] Snyder | X-vectors: Robust DNN embeddings for speaker recognition[END_REF] are the state of the art represention used in the speaker identification field. To get the X-Vectors, we extract embedded vectors independently on the text using a pre-trained model1 . As stated before, this model was initially trained for a speaker verification task [START_REF] Snyder | X-vectors: Robust DNN embeddings for speaker recognition[END_REF][START_REF] Snyder | Deep Neural Network Embeddings for Text-Independent Speaker Verification[END_REF][START_REF] Xu | Generative x-vectors for text-independent speaker verification[END_REF] using NIST SRE recipe supported in the Kaldi toolkit. The details about the recipe and the pretained model are available in author's github2 . This configuration is more detailed than the OneHot-Vector but still remains difficult to control as the dimensions of the X-Vectors are difficult to interpret.

P-Vector

The last configuration we are proposing is the P-Vector. To characterize the speaker style and the specificity of an expressive voice, we propose to use the breath group as the functional unit to build a vector able to cover high-level prosodic information which are difficult to predict from the text. A P-Vector is defined by the following features:

• F0-range: for each vowel of the breath group, we are computing the median values.

Then, considering F 0 min and F 0 max , respectively, the minimum and the maximum median values, we computed the scaled F 0 range the following way:

         F 0min = min(V 0 F 0median , . . . , V M F 0median ) F 0max = max(V 0 F 0median , . . . , V M F 0median ) F 0range = 12 × log 2 ( F 0min F 0max )
beyond are the speaker code. The size of this part vary according to the configuration used among the configurations under study (e.g. OneHot-Vector, X-Vector or P-Vector).

The output feature vector contains the standard coefficient vector composed by the Voiced/Unvoiced (VUV) flag, the logF 0 , the Mel-Generalized Cepstrum (MGC), the BAP, and their dynamic counterparts. This leads to a vector of 265 coefficients.

Finally, the input and output vectors are normalized using, respectively, Min-Max Normalization (MMN) and Mean Variance Normalization (MVN) methods.

Method

The main goal is to see if and how the content of the input vector influences the ability to separate speaker-related information in a DNN-based TTS system. To do so, we learn several systems differing by the structure of the input vectors provided. OOnce the different systems are learned, to analyze if the various configurations are guiding the models to capture speaker specificities, we propose to measure differences at the output of the first hidden layer as well as at the output of the model. Two types of analyses are then done:

• Standard objective measures: MCD, BAP distortion, F 0 Root Mean Square Error (RMSE), F 0 correlation, VUV error rate, RMSE on the duration and duration correlation;

• A visual analysis protocol illustrated in the Figure 7.1: a PCA (see Appendix 1 for a technical prosodure of PCA) on the first hidden layer output is computed. Then, we visualize the main dimensions and analyze the results in function of the speakers to see if speaker-dependent information is captured by the model. We perform PCA at the end of each epoch on the validation dataset. We choose to do the analysis at this stage of the network because it is easier to interpret and quantify the variation brought by the input.

We also compare different epochs to see how the models are evolving. This monitoring is interesting since it enables to check quickly if the structure of the input vectors has an impact on speaker separability.

Experimental setup

Dataset

We have selected nine speaker from the MUFASA corpus (4 Females/ 5 Males). This subset contains fictional french audiobooks published between the 18th-20th century. We follow the same procedure as the one described in the 1. The text is splited into pseudo-paragraphs and then force-aligned to corresponding speech using JTrans [START_REF] Cerisara | JTrans, an opensource software for semi-automatic text-to-speech alignment[END_REF]. The speech signals are sampled at 48 kHz. All the meta-data information related to describe the book (speaker identifier, library name, . . . ) were removed. From the designed corpus, two groups of data were defined:

• parallel data: this group contains 5 audiobooks ( for more information about the books, see the Appendix 4); each transcription have been read by at least 2 speakers.

In total, the data for 9 speakers has been collected including 4 females. Voices were selected by an informal listening test considering their recording conditions (non-audible difference), and the fact that the voice quality of the speakers are quite different.

• non-parallel data: for each speaker in the parallel data, 1h of extra speech has been collected with no overlap in the transcription. This set of data is used to evaluate robustness and performance of the speaker encoder input.

The procedure used to achieve the annotation process and to extract the linguistic features is described in [START_REF] Sini | Annotation automatique des types de discours dans des livres audio en vue d'une oralisation par un système de synthèse[END_REF].

Models configuration

To achieve training and synthesis, we used the Merlin toolkit [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF]. The architecture of the model is a FF-DNN with 4 hidden layers. During the experiments, we changed first layer size to be 128, 256 or 512 neurons without any significant change. The last three layers have a fixed number of 512 neurons. The hidden layers use the tanh activation function and the output layer uses a linear activation function. We applied batch-training paradigm with a batch size of 256. The maximum number of epochs is set to 25 including 10 warm-up epochs. The learning rate is initially set to 0.002 for warm-up epochs and after that reduced by 50% for each epoch. Similarly, the momentum is set to 0.3 for warm-up epochs and to 0.9 otherwise. Finally, we used L2-regularization with a weight set to 10 -5 . Models are learned considering speaker coding schemes with the following dimensions: 2 for OneHot-Vector (OHV), 32 for X-Vector and 9 for P-Vector.

Results

Standard measurements

In order to evaluate DNN-based TTS synthesis, the proposed method was applied to train models for each audiobook present in the parallel training set, and then on the non-parallel training set.

All the models have been evaluated using MCD, BAP distortion, RMSE on F 0 and duration, VUV rate and Correlation (CORR) on F 0 and duration, between the predicted and the original coefficients. In this work, only the objective results concerning the nonparallel training dataset are reported as similar results have been observed in the parallel training dataset.

As shown in Table 7.1, the system involving the P-Vector outperforms the baseline system in all kinds of objective measures. PCA3 has been applied on the output of the first hidden layer to reduce the number of dimensions down to the two main ones. The first explanation for this behavior is that with non-parallel data, the linguistic, prosodic and phonetic context variability are dominant and most of the variation is hold by those components. As the data are non parallel, the neural network has more difficulty to distinguish the speakers. The second possible explanation is that the size and complexity of X-Vector and P-Vector bring more sparsity in the latent space, which is not the case with OneHot-Vector. Finally, it seems that X-Vector and P-Vector can be used equally to bring speaker control to the system but due to the lower complexity of P-Vector, this representation might be preferable.

The visualization of the evolution of the latent space projection at different epochs is illustrated on Figure 7.4. It enables to monitor the learning process and check quickly the impact of the vector structure on the speaker separation. Here, we can notice that from epoch 10, the projected latent space is quite stable and the speakers well separated.

Subjective Evaluation

Evaluation protocol

In order to validate our proposition, we conducted a subjective evaluation based on the MUSHRA protocol [START_REF] Series | Method for the subjective assessment of intermediate quality level of audio systems[END_REF]]. The reference is the re-synthesis using world. We use a speaker dependent baseline (spkdep) as well as a speaker independent model available in4 (spkadapt). Then, we evaluated the isolated configurations (OneHot-Vector, X-Vector and P-Vector).

The duration of each of the 54 samples presented to the listeners is comprised between 4 and 6s. The ratio of speech breaks present in the selected samples does not exceed the quarter of the total duration of the sample.The list of stimulis used for the subjective assessment are in Appendix 2 One evaluation instance is composed by 9 steps including all the models presented before (an example of step is illustated by the Appendix 3). 30 listeners completed the evaluation. They were French native speakers aged between 24 and 45. The majority of them have experience with listening tests but are not necessarily experts in the annotation of audio files. All materials are available in the dedicated repository5 .

Discussion and results

The results of the evaluation are presented in Figure 7.5. From them, we can see that the reference is correctly identified which guarantees the validity of the evaluation. It seems that some annotators estimate that even the reference was not good enough for some samples which explains the fact that the reference did not achieve a score of 100. Then, considering the models evaluated, no system is outperforming the other ones. This leads us to conclude that listeners do not distinguish major differences. To verify that the listeners didn't perceive minor differences, we also compute the rank of each systems for each step based on its score. Results are presented in Figure 7.6 (whereas Figure 7.7 represent the result related to all speakers).

The reference is still considered in huge majority as system number one. Considering the others, the proportion are globally similar to each other with some variations. This is amplified by the fact that the other systems are often ranked in second position which indicates they are all graded ex-aequo after the reference.

Conclusion

In this chapter, we have evaluated different speaker coding scheme both objectively and subjectively in a DNN-based framework. All the evaluations conducted show no difference in the quality of the modeling of the three different speaker coding schemes. These results are valid in both studied cases, parallel or non-parallel data for multi-speaker modeling. Moreover, the speaker coding scheme we proposed, the P-Vector, provides better control of the modeling. This investigation confirms the relevance of the prosodic parameters that we choose to build the prosodic identity of speakers. However, a close look at this representation shows that the intra-speaker prosodic variation related to discourse changes (narration, dialog) are excluded.

These results are encouraging and suggest further research work. Furthermore, we plan the evaluation of the robustness of the proposed speaker coding on a dataset that contains more speakers and investigating other factors such as language, literary genre, discourse typography, and structure. discrete emotion labeling and discourse labeling. The annotation process ends with four complementary annotations: intonation patterns, discourse represented by characters pattern, emotions patterns, and others. With emotion patterns annotation, we build a binary classifier, the results of the experiments highlighted the subtility of emotions in such type of data. Based on this observation, we proposed to explore the questions relating to emotions through the analysis of the lexical and semantic properties of transcriptions of audiobooks. To do these experiments, we favored unsupervised approaches. This second experience is based on the techniques of sentiment analysis and natural language processing. The process mainly consists of finding an adequate numeric representation of the texts, we choose the doc2vec model, then clustering the embedded text automatically according to lexico-semantic affinities using the Kmeans algorithm. Once the clusters formed, the last phase consists of interpreting the clusters in the acoustic features space. The results show that there strong correlation between text representation and acoustic speech features. This contribution opens perspectives that we will discuss later.

For studying the discourse, we first built a tool for parsing and annotating audiobooks texts considering three discourses types, namely indirect discourse, direct discourse, and incidental clauses with speech verbs. This tool contains two approaches. The first approach is rule-based consists of a set of rules derived from data analysis and crafted by expert knowledge using morpho-syntactic and typographical properties of the text. The second approach relies on machine learning techniques; we obtained the best result with deep learning models, for highlighting the prosodic properties during discourses changes and how speakers address this phenomenon. We proposed analyzing this phenomenon through a set of prosodic cues derived from InterPausal Unit (IPU) that we consider as pertinent discourse. We experimented with two female speakers of the MUFASA corpus. The results confirmed that the IPU is an adequate speech unit for studying discourse changes; F0-range and inter-IPU pause duration are good indicators of discourse changes.

Concerning the last pilar, speaker voice properties, We explore three speakers configurations, OneHot vector, representing the speaker identity through two parameters speaker gender and identifier, X-vector, this embedded vector derived from the pre-trained speaker recognition model, P-Vector, a new vectorial prosodic representation of voices. We implemented these configurations for guiding a DNN based multi-speaker speech synthesis system. To evaluate these configurations, we conducted two objective evaluations standard objective evaluation described in chapter 3, and objective visual evaluation, consisting of projecting the first hidden layer representation. Furthermore, we investigate a subjec-tive evaluation to support objective assessment funding. Both objective and subjective assessment has shown that P-vector's prosodic identity is capable of guiding the DNN based multi-speaker speech synthesis system as good as of the well established X-vector and OneHot Vector.

Further Issues

In this section, we briefly present work already started and preliminary results that we obtained.

Does granularity matter in speech synthesis ?

If we consider the three pillars that represent the contribution of this thesis as well as the construction of the MUFASA corpus, we can see that there is a common thread to all of them. In this work, granularity is designated as a discursive unit when it is a textual segment or speech unit when it is a segment of a speech signal.

Most speech synthesis systems are trained to process sentences. In most cases, the sentence is considered as both the discourse unit for processing the text to be analyzed, and the speech unit for training acoustic and duration models. We can easily claim that this seems relevant because most of the databases built for speech synthesis have been recorded in isolated sentences(sentence by sentence). However, is this unit the best choice when dealing with audiobooks where the original speech records are chapters or paragraphs? Or does it matter? Some studies have looked at the optimal level of granularity to improve speech synthesis systems' expressiveness, especially when it comes to long and coherent texts such as audiobooks. This preliminary work aims to study the discourse/speech unit's effects on learning statistical models on speech synthesis.

We consider two types of units, graphical-based discourse/speech units, represented by sentence, which is the most privileged prosodic unit in speech synthesis systems and the pseudo-paragraph, which represents the largest, and prosodical-based discourse/speech unit, represented by InterPausal Unit (IPU) which is the prosodic unit between two long pauses (pause> = 200ms). To measure each of these units' impact, we rely on the standard objective measures described in chapter 1.

Data and features extraction

To evaluate the effect of the prosodic unit on parametric speech synthesis, we consider an audiobook of 90 minutes, read by an amateur female speaker available in the SynPaFlex-Corpus.

Here is an example of the considered prosodic units extracted from the short story for the past two months had been reconnoitering with the utmost caution in the neighboring woods, occasionally shooting their own sentinels, and making ready for fight whenever a rabbit rustled in the undergrowth, had now returned to their homes. Their arms, their uniforms, all the death-dealing paraphernalia with which they had terrified all the milestones along the highroad for eight miles round had suddenly and marvelously disappeared.]

Sentences

System training configuration

To achieve training and synthesis, we used the Merlin toolkit [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF].

The architecture of the model is a FF-DNN with 4 hidden layers. Each hidden layer have a fixed number of 512 neurons. The hidden layers use the tanh activation function and the output layer uses a linear activation function. We applied batch-training paradigm with a batch size of 256. The maximum number of epochs is set to 25 including 10 warm-up epochs. The learning rate is initially set to 0.002 for warm-up epochs and after that reduced by 50% for each epoch. Similarly, the momentum is set to 0.3 for warm-up epochs and to 0.9 otherwise. Finally, we used L2-regularization with a weight set to 10 -5 . Models are learned considering different granularity unit schemes.

The input feature vector contains 319 coefficients corresponding to the linguistic description of the utterance. This part is based on the standard feature set for English described in [START_REF] Tokuda | An HMM-based speech synthesis system applied to English[END_REF] that we have adapted for French. The main differences with the English feature set concerns the accentuation. Indeed, as the accentuation information in French is strongly correlated to the POS information, we therefore consider that the POS information, already present in the vector, is enough to encode the accentuation information.

The output feature vector contains the standard coefficient vector composed by the VUV flag, the logF 0 , the MGC, the BAP, and their dynamic counterparts extracted using WORLD [START_REF] Morise | WORLD: a vocoderbased high-quality speech synthesis system for real-time applications[END_REF] vocoder. This leads to a vector of 265 coefficients.

Finally, the input and output vectors are normalized using, respectively, MMN and MVN methods.

Objective evaluation and results

In order to evaluate DNN-based TTS synthesis, the proposed method was applied to train three models using three training set and three test set.

All the models have been evaluated using MCD, BAP distortion, RMSE on F 0 and duration, VUV rate and Correlation (CORR) on F 0 and duration, between the predicted and the original coefficients.

The preliminary results reported in Table 7.2 show that the granularity of data used to build a synthetic voice is essential. According to the present results, the sentence is not always the best choice to build a synthetic voice in a Statistical Parametric Speech Synthesis (SPSS) system. A more in-depth investigation needs to be done with different voices and different audiobooks.

Modern Speech Synthesis Framework (End-to-End (E2E) Paradigm)

During this thesis, we had the opportunity to train and to test advanced techniques based on neural networks such as the WaveNet [START_REF] Oord | Wavenet: A generative model for raw audio[END_REF] Vocoder for speech generation and Tacotron-2 [START_REF] Shen | Natural tts synthesis by conditioning wavenet on mel spectrogram predictions[END_REF] -Amateurs audiobooks are not dedicated to speech synthesis at the origin, so the recordings are not as good in term of quality as the one recorded for synthesis.

During the test that we made, we found that these architectures are sensitive to the quality of the data. It thus made difficult to build a robust model with such type of data and with the difficulty to find more data.

-The majority of neural architectures rely on an attention mechanism to align the encoder part with the decoder part. Tests have shown that these mechanisms are fragile and not robust when it comes to long sentences, often present in the audiobooks of SynPaFlex corpus.

-Parameterization: Training these models, many parameters are defined empirically, which makes the training phase tricky.

-The training of the model and the synthesis phase are both highly time consuming.

-Lack of reliable objective evaluation.

For these different reasons in this thesis, we found it more judicious to focus exclusively on the improvement of statistical parametric models based on modular architectures, including distinct Front-End and Back-End parts.

Perspectives Short-term perspective

Chapter 7 presents a whole process of integrating speaker prosodic identity from the stage of hypothesis to the stage of concrete integration in a realistic speech synthesis system and formal objective and subjective assessment. The two other works concerning emotion pattern and discourse related prosodic cues still at the statistical analytic and objective evaluation stage. So as a short term perspective, we aim to integrate these two variables in the MERLIN [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF] toolkit framework relying on the same procedure presented in [START_REF] Malisz | Controlling Prominence Realisation in Parametric DNN-Based Speech Synthesis[END_REF]]. Concretely, we would like to insert two new neural network-based modules, one for building a discourse embedded vector and the other one for emotion embedded vector (EEV). Both modules will be inserted after the front-end module. These two modules will be trained before the duration module and acoustic module.

To evaluate these two modules' effects, we are considering two distinct subjective assessment one for each module. To the perceptual discourse assessment, the stimuli are extracts of discourses changes mode (DD, ID, IC), two questions are planned: direct question "do you notice any changes speech sample?" (yes/no), to see if the subject has noticed any changes, then a second question "What kind of changes you perceive? a) speech rate "fast/slow" b) speech amplitude c) "pause duration shorter/longer." .A similar evaluation process will be conducted to evaluate the emotion module impact. The stimuli will be the same as those used for assessing the discourse module, but the questions will not be the same. As in this second subjective assessment, the questions will be "do you recognize emotion in this speech sample," if the subject answer yes, a list of considering emotions will be presented followed by the intensity of the perceived emotion or emotions because we suppose that the subject can assign for same sample several emotion labels with different intensity.

Beyond the analysis of the results of each module's respective effect, the combination of the results is also considered a perspective because it allows us to measure the correlation between discourse and emotion.

Long-term perspective

As a concrete long term perspective, we plan to migrate from the Merlin [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF] -SPSS framework to E2E Framework, more precisely Tacotran2 [Wang et al. 2017a;[START_REF] Shen | Natural tts synthesis by conditioning wavenet on mel spectrogram predictions[END_REF]] available in the ESPNET [START_REF] Hayashi | Espnet-tts: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit[END_REF]] toolkit, to gain a considerable quality. Then, we plan to build a module similar to the one developed in the short term perspectives. However, in this new configuration we will merge the two modules into a single module relying on deep multi-task neural networks [START_REF] Liu | Multi-task deep neural networks for natural language understanding[END_REF]. This module will be trained together with the acoustic models.

General Discussion

In this thesis, we addressed prosodic characterizations in case of the synthesis of audiobooks through three dimensions:

-Emotions acted by a speaker to set the story context and provided additional elements for entertaining the listener's attention.

-Discourse typography to highlight the structures of the texts and the correlation with prosodic indices.

-The speaker's identity with the long-term goal of highlighting the reading strategy. Speech has to respect syntactic, semantic, pragmatic constraints as well as related to the written-text discourse typology. In parallel, the speaker strategy and reading identity constraint emotion realization and acting.

The correlation between the three parameters explored in this thesis makes it difficult to build up a robust and reliable expressive speech synthesis system. Disentangling these "three pillars" using factorization techniques based on advanced deep learning algorithms seems to be interesting, according to [Hsu et al. 2019;[START_REF] Mathieu | Disentangling factors of variation in deep representation using adversarial training[END_REF].

Whereas [START_REF] Brognaux | Expressive speech synthesis : research and system design with hidden Markov models[END_REF]] explores the expressivity through spontaneous speech, we focus on reading written text. It will be interesting to make a comparison between spontaneous, in particular, sports comments and read-text, in particular, audiobooks, to find common representation to expressive speech. Résumé : Dans ces travaux de thèse nous abordons l'expressivité de la parole lue avec un type de données particulier qui sont les livres audio. Les livres audio sont des enregistrements audio d'oeuvres littéraires fait par des professionnels (des acteurs, des chanteurs, des narrateurs professionnels) ou par des amateurs. Ces enregistrements peuvent être destinés à un public particulier (aveugles ou personnes mal voyantes). La disponibilité de ce genre de données en grande quantité avec une assez bonne qualité a attiré l'attention de la communauté scientifique en traitement automatique du langage et de la parole en général, ainsi que des chercheurs spécialisés dans la synthèse de parole expressive. Pour explorer ce vaste champ d'investigation qui est l'expressivité, nous proposons dans cette thèse d'étudier trois entités élémen-taires de l'expressivité qui sont véhiculées par les livres audio: l'émotion, les variations liées aux changements discursifs et les propriétés du locuteur. Nous traitons ces patrons d'un point de vue prosodique. Les principales contributions de cette thèse sont la construction d'un corpus de livres audio comportant un nombre important d'enregistrements partiellement annotés par un expert, une étude quantitative caractérisant les émotions dans ce type de données, la construction de modèles basés sur des techniques d'apprentissage automatique pour l'annotation automatique de types de discours et enfin nous proposons une représentation vectorielle de l'identité prosodique d'un locuteur dans le cadre de la synthèse statistique paramétrique de la parole.

Historic novels

Les

Nuance pattern

Resolution pattern

Title: Characterisation and generation of expressivity in function of speaking styles for audiobook synthesis Keywords: Computer Science,Speech Prosody, Audiobook, Expressive Speech Synthesis,Machine Learning Abstract: In this thesis, we study the expressivity of read speech with a particular type of data, which are audiobooks. Audiobooks are audio recordings of literary works made by professionals (actors, singers, professional narrators) or by amateurs. These recordings may be intended for a particular audience (blind or visually impaired people). The availability of this kind of data in large quantities with a good enough quality has attracted the attention of the research community in automatic speech and language processing in general and of researchers specialized in expressive speech synthesis systems. We propose in this thesis to study three elementary entities of ex-pressivity that are conveyed by audiobooks: emotion, variations related to discursive changes, and speaker properties. We treat these patterns from a prosodic point of view. The main contributions of this thesis are: the construction of a corpus of audiobooks with a large number of recordings partially annotated by an expert, a quantitative study characterizing the emotions in this type of data, the construction of a model based on automatic learning techniques for the automatic annotation of discourse types and finally we propose a vector representation of the prosodic identity of a speaker in the framework of parametric statistical speech synthesis.

1. 1

 1 Text-to-Speech (TTS) system pipeline . . . . . . . . . . . . . . . . . . . . 5 3.1 Overview of the Speech Segmentation process . . . . . . . . . . . . . . . . 26 3.2 the vowel trapezoids of the three cardinal vowels /u/, /i/, and /a/ . . . . 37 3.3 Pauses distribution and average duration for "Mademoiselle Albertine est partie" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.4 Pauses distribution and average duration for "Vingt mille lieues sous les mers Chapter 3". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.5 The vowel trapezoids of the three cardinal vowel, in the context of occlusive /p/,/t/,/k/ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.6 The density distribution according to the duration of the three vowels preceded by an occlusive consonant /p/,/t/,/k/ . . . . . . . . . . . . . . . 43 4.1 The ten fundamental intonations defined in [Delattre 1966], illustrated by a dialogue: -Si ces oeufs étaient frais j'en prendrias. Qui les vend? C'est bien toi, ma jolie? -Évidemment, Monsieur. -Allons doc! Prouve-le-moi. [-If these eggs were fresh, I'd take some. Who sells them? Is it you, my pretty? -Of course it is, sir. -Come on, then! Prove it to me.] . . . . . . . . . . . . 48 4.2 Nuance Intonation Pattern Example : puis il me semblait avoir entendu sur l'escalier les pas légers de plusieurs femmes se dirigeant vers l'extrémité du corridor opposé à ma chambre. . . . . . . . . . . . . . . . . . . . . . . . . 49 4.3 A combination of three non exclusive intonation pattern. The nuance pattern is recognized with its particular pitch contour described in Figure D.3 Dans cette cruelle position, elle ne s'est donc pas adressée at begining of the utterance, followed by an emotional pattern characterized by the dynamic pitch (high F0-range) à la marquise d'Harville, sa parente, and finishing with an explicit question pattern sa meilleure amie ? . . . . . . . . . . . . 50 4.4 Scheme of proposed framework . . . . . . . . . . . . . . . . . . . . . . . . 58 vii 4.5 The data points scatter in k = 18 groups -doc2vec features. The right-hand side shows the result of K-means, i.e., the data points of each cluster. The left-hand side shows the silhouette coefficient of each cluster. The thickness of each cluster plot depends on the number of data points lying in the cluster. The red bar is the average of the silhouette coefficient of entire clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 The data points scatter in k = 7 groups -doc2vecs + emotional vector features. The right-hand side shows the result of K-means, i.e., the data points of each cluster. The left-hand side shows the silhouette coefficient of each cluster. The thickness of each cluster plot depends on the number of data points lying in the cluster. The red bar is the average of the silhouette coefficient of entire clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Principal Component Analysis (PCA) variation coverage of 73 % with 50 components, T-distributed Stochastic Neighbor Embedding (t-SNE) with perplexity of 45 and with iteration of 250 . . . . . . . . . . . . . . . . . . .

5. 2

 2 Detection and annotation of incidental clauses with reporting verbs (IC) . 5.3 Receiver Operating Characteristic (ROC) . . . . . . . . . . . . . . . . . . . viii 6.1 Illustration of an example of discourse passage from Direct Discourse to Incidental Clauses with reporting verbs ( Direct Discourse (DD) ⇒ Incidental Clauses with reporting verbs (IC)) corresponding to one modality and data structure. The tiers correspond (from the buttom to the upper one): Articulation Rate articulation rate measured with Equation (6.2) , Fundamental frequency (F 0 )-range with Equation (6.1), syllables, words, breath group and related discourse. . . . . . . . . . . . . . . . . . . . . . . 7.1 Top part represents the architecture of the proposed model, the bottom part illustrates the visualization process of the first hidden layer. . . . . . . 7.2 Principal Component Analysis (PCA) projection for the parallel data during the validation phase, the speaker identify is encoded as following (F/M: Female/Male, FR: French, ID:XXXX). . . . . . . . . . . . . . . . . . . . . 7.3 Principal Component Analysis (PCA) projection for the non parallel data during the validation phase. . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Visualization of the latent representation in case of P-Vector using parallel data. We can notice the separation of the speakers representation from epoch 5 to epoch 25. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Result of the MUSHRA of the listening test. . . . . . . . . . . . . . . . . . 7.6 Ranking score of two representative speakers female (ffr001) and male (mfr0008), the present results are similar for the other speakers. . . . . . . 7.7 Ranking score of all speakers . . . . . . . . . . . . . . . . . . . . . . . . . D.1 Avez-vous entendu ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 La voiture arrivait près de Saint-Denis, la haute flèche de l'église se voyait au loin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.3 Nuance Intonation Pattern Example puis il me semblait avoir entendu sur l'escalier les pas légers de plusieurs femmes se dirigeant vers l'extrémité du corridor opposé à ma chambre. . . . . . . . . . . . . . . . . . . . . . . . . . D.4 --Ma cravache, s'il vous plaît . . . . . . . . . . . . . . . . . . . . . . . . D.5 -Je ne les connais pas . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.6 [Note : me tendre un piège.] . . . . . . . . . . . . . . . . . . . . . . . . . . D.7 ...M'en allant promener, J'ai trouvé l'eau si belle Que je me suis baigné... . ix D.8 Screenshot of the platform PercEval (Recently renamed FlexEval

3. 8

 8 The set of extracts for conducting a comparative study. . . . . . . . . . . 39 4.2 Durations and amount of annotated data according to discourse mode in the first version of the SynPaFlex-Corpus . . . . . . . . . . . . . . . . . . . 47 4.3 Manual annotations -Total duration of intonation patterns (including combinations) in the 13h25 sub-corpus . . . . . . . . . . . . . . . . . . . . 48 4.4 Manual annotations -Total durations of emotion categories labels (including combinations) in the 13h25 sub-corpus . . . . . . . . . . . . . . . . . . . . 52 4.5 Examples of perceived impacts of emotion on the speech . . . . . . . . . . 52 xi 4.6 Number of manually annotated emotional segments and segments resulting from a 1 s. max chunking. The latest are used in the classification experiments. Other includes Irony and Threat labels. . . . . . . . . . . 4.7 Feature set of the INTERSPEECH 2009 Emotion Challenge 384 features, (16 LLD + 16 ∆)*12 functionals . . . . . . . . . . . . . . . . . . . . . . . . 4.8 Unweighted Average Recall (UAR) results for binary emotion classification using the three feature subsets. In bold, UAR > 60%, which we considered as a reasonable classification rate. . . . . . . . . . . . . . . . . . . . . . . . 4.9 The best K-clusters according to the silhouette average criteria and average samples per cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Composition of the corpus according to types of discourse, selected from the corpus SynPaFlex describe in Section 1 . . . . . . . . . . . . . . . . . 5.2 Results of detection and annotation of discursive changes . . . . . . . . . . 5.3 Result of classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Overview of the sub-corpus content. N-utt represent the number of utterances.s 6.2 Discursive changes distribution sub-corpus . . . . . . . . . . . . . . . . . . 6.3 Means and standard deviations for F 0 -range and articulation rate (AR) for the different types of discourse change. . . . . . . . . . . . . . . . . . . . . 6.4 Means and standard deviations for Inter-Breath Group Pause Duration (IBGP) according to different types of discourse change. . . . . . . . . . . . 6.5 Comparing IBGP across the different discourse changes modalities (** represents p-value<0.001). . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Objective results for multi-speaker modeling, considering five speaker code configurations. Mel-Cepstral Distortion (MCD), Band Aperiodicity Parameter (BAP), Root Mean Square Error (RMSE), Voiced/Unvoiced (VUV) and Correlation (CORR) between the predicted and the original coefficients. For the Fundamental frequency (F 0 ), Root Mean Square Error (RMSE) and Correlation (CORR) are computed on the voiced frames only. . . . . . . . xii 7.2 Objective results of the acoustic model, considering the three granularity. Mel-Cepstral Distortion (MCD), Band Aperiodicity Parameter (BAP), Root Mean Square Error (RMSE), Voiced/Unvoiced (VUV) and Correlation (CORR) between the predicted and the original coefficients. For the Fundamental frequency (F 0 ), Root Mean Square Error (RMSE) and Correlation (CORR) are computed on the voiced frames only. . . . . . . . . . . . . . . 113 A.2 MUltispeaker French Audiobooks corpus dedicated to expressive read Speech Analysis (MUFASA) corpus . . . . . . . . . . . . . . . . . . . . . . . . . . 119 A.3 MUltispeaker French Audiobooks corpus dedicated to expressive read Speech Analysis (MUFASA) Parallel Subcorpus . . . . . . . . . . . . . . . . . .

Figure

  Figure 1.1: TTS system pipeline

  Figure 3.1: Overview of the Speech Segmentation process

  (a) MUFASA Speakers duration distribution and gender labeling (b) MUFASA Speakers duration distribution and library belong to. (a) Book author's distribution in MUFASA corpus (b) Genre proportions in MUFASA corpus Narrative genre such as novel, short story, and tale are the most frequent in the corpus as shown in Figure 3.2b. Some author are well represented in the corpus Figure 3.2a.

  (a) A schematic of vowel triangle (b) A schematic of vowel triangle (French)

F2

F2Figure 3

 3 Figure3.2: the vowel trapezoids of the three cardinal vowels /u/, /i/, and /a/

F2Figure 3 . 5 :

 35 Figure 3.5: The vowel trapezoids of the three cardinal vowel, in the context of occlusive /p/,/t/,/k/

Figure 4 . 1 :

 41 Figure 4.1: The ten fundamental intonations defined in [Delattre 1966], illustrated by a dialogue: -Si ces oeufs étaient frais j'en prendrias. Qui les vend? C'est bien toi, ma jolie? -Évidemment, Monsieur. -Allons doc! Prouve-le-moi. [-If these eggs were fresh, I'd take some. Who sells them? Is it you, my pretty? -Of course it is, sir. -Come on, then! Prove it to me.]

Figure 4

 4 Figure 4.2: Nuance Intonation Pattern Example : puis il me semblait avoir entendu sur l'escalier les pas légers de plusieurs femmes se dirigeant vers l'extrémité du corridor opposé à ma chambre.

Figure 4

 4 Figure 4.3: A combination of three non exclusive intonation pattern. The nuance pattern is recognized with its particular pitch contour described in Figure D.3 Dans cette cruelle position, elle ne s'est donc pas adressée at begining of the utterance, followed by an emotional pattern characterized by the dynamic pitch (high F0-range) à la marquise d'Harville, sa parente, and finishing with an explicit question pattern sa meilleure amie ?

Figure 4 . 4 :

 44 Figure 4.4: Scheme of proposed framework

Figure 4 . 5 :

 45 Figure 4.5: The data points scatter in k = 18 groups -doc2vec features. The right-hand side shows the result of K-means, i.e., the data points of each cluster. The left-hand side shows the silhouette coefficient of each cluster. The thickness of each cluster plot depends on the number of data points lying in the cluster. The red bar is the average of the silhouette coefficient of entire clusters.

Figure 4

 4 Figure 4.6: The data points scatter in k = 7 groups -doc2vecs + emotional vector features. The right-hand side shows the result of K-means, i.e., the data points of each cluster. The left-hand side shows the silhouette coefficient of each cluster. The thickness of each cluster plot depends on the number of data points lying in the cluster. The red bar is the average of the silhouette coefficient of entire clusters.

Figure 4

 4 Figure 4.7: PCA variation coverage of 73 % with 50 components, t-SNE with perplexity of 45 and with iteration of 250

Figure 5 . 1 :

 51 Figure5.1: This figure illustrates the workflow guiding the rule-based approach. After the phonetization and forced alignment of the chapter text with the corresponding audio file, the data are segmented into paragraphs/ pseudo-paragraphs and stored relying on roots toolkit. The segments follow two annotations process: (i) the manual annotation made by an expert (ii) the automatic annotation which has two phases, the first phase consists of labeling the segments according to typographic criteria as DD, ID, and mixed group. The mixed groups are processed in phase 2 ( Figure5.2) in order to fine-tune the annotation and label the group according to DD, ID, and IC. The mixed groups annotated, and non-mixed groups form the automatic annotation sequence. The two annotations (manual and automatic) are fused to form the Annotated Corpus (AC).

+

  Figure 5.2: Detection and annotation of incidental clauses with reporting verbs (IC)

( 6 )

 6 -Cinq cents vers à toute la classe ! exclamé d'une voix furieuse, arrêta, comme le Quos ego, une bourrasque nouvelle. (Madame Bovary, Chapitre 1 ) -Those where the parsing performed is erroneous as in the extract (7). "the syntactic complexity" of the Incidental Clauses with reporting verbs (IC) makes its analysis difficult (7) -Oui... j'entends bien ; vous voulez que je vous mène à sa porte... et puis à son lit... et puis que je vous dise où frapper, et puis que je vous guide le bras, n'est-ce pas ? Vous voulez enfin me faire servir de manche à votre couteau !... vieux monstre! reprit Tortillard avec une expression de mépris, de colère et d'horreur qui, pour la première fois de la journée, rendit sérieuse sa figure de fouine, jusqu'alors railleuse et effrontée. On me tuerait plutôt... entendez-vous... que de me forcer à vous conduire chez votre femme. (Les Mystères de Paris,Chapitre 7, Partie 2 )

  Figure 5.3 represents the ROC for tested models where class 0, class 1 and class 2 accounts for the DD, ID, and IC mode respectively. Class 0 (DD mode) has larger true positive rate than other two classes.

  Figure 5.3: Receiver Operating Characteristic (ROC)

-

  Figure 6.1: Illustration of an example of discourse passage from Direct Discourse to Incidental Clauses with reporting verbs ( DD ⇒ IC) corresponding to one modality and data structure. The tiers correspond (from the buttom to the upper one): Articulation Rate articulation rate measured with Equation (6.2) , F 0 -range with Equation (6.1), syllables, words, breath group and related discourse.

Figure 7 . 1 :

 71 Figure 7.1: Top part represents the architecture of the proposed model, the bottom part illustrates the visualization process of the first hidden layer.

Figure 7

 7 Figure 7.2: PCA projection for the parallel data during the validation phase, the speaker identify is encoded as following (F/M: Female/Male, FR: French, ID:XXXX).

Figure 7

 7 Figure 7.3: PCA projection for the non parallel data during the validation phase.

Figure 7 . 4 :

 74 Figure 7.4: Visualization of the latent representation in case of P-Vector using parallel data. We can notice the separation of the speakers representation from epoch 5 to epoch 25.

Figure 7 .

 7 Figure 7.2 and Figure 7.3 illustrate respectively the parallel data and non-parallel data projections for the different configurations. While with the parallel data, it seems that the configuration involving the OneHot-Vector fails to separate the speakers, P-Vector and X-Vector achieve almost the same result and succeed to separate speakers representation. With non-parallel data, both X-Vector and P-Vector do not show a clear separation between speakers compared to OneHot-Vector.

Figure 7 . 5 :

 75 Figure 7.5: Result of the MUSHRA of the listening test.

Figure 7 . 6 :

 76 Figure 7.6: Ranking score of two representative speakers female (ffr001) and male (mfr0008), the present results are similar for the other speakers.

Figure 7 . 7 :

 77 Figure 7.7: Ranking score of all speakers

1. 2

 2 Figure D.2: La voiture arrivait près de Saint-Denis, la haute flèche de l'église se voyait au loin.

Figure

  Figure D.3: Nuance Intonation Pattern Example puis il me semblait avoir entendu sur l'escalier les pas légers de plusieurs femmes se dirigeant vers l'extrémité du corridor opposé à ma chambre.

Figure

  Figure D.4: --Ma cravache, s'il vous plaît

Figure D. 6 :

 6 Figure D.6: [Note : me tendre un piège.]

Introduction 1 1 Text-to-Speech Synthesis 5

			. 10
	3	Expressive Speech Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . 14
		3.1	What do we mean by "expressive speech synthesis"? . . . . . . . . . 14
		3.2	Transversal questions . . . . . . . . . . . . . . . . . . . . . . . . . . 15
	2 Speech Prosody	17
	1	What is prosody? .

1 Text-To-Speech Synthesis System . . . . . . . . . . . . . . . . . . . . . . . 5 1.1 Front-End . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Back-End . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2 Statistical Parametric Speech Synthesis . . . . . . . . . . . . . . . . . . . . 8 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 i

  Validation results for the segmentation step per literary genre : lengths of the validation subsets, Phoneme Error Rate (PER), and average alignment error. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3 Amounts of linguistic units in the SynPaFlex corpus . . . . . . . . . . . . . 28 3.4 The main linguistic content of MUltispeaker French Audiobooks corpus dedicated to expressive read Speech Analysis (MUFASA) Corpus . . . . . 30 3.5 Subcorpus contents. The first column corresponds to the title of the novel, and author's name. Nbr. Utts is the number of utterances(sentences), Nbr.

	List of Tables
	architecture 155
	3.1
	x

  As for boundary alignment, in 77.17% of cases, boundaries are misplaced from less Table3.1: Validation results for the segmentation step per literary genre : lengths of the validation subsets, Phoneme Error Rate (PER), and average alignment error.

		Validation PER	average
		subset	(%) alignment
				error (ms)
	Novels	25m36s	5.8	11.5
	Short stories	3m49s	7.1	9.4
	Tales	2m47s	0.8	14.3
	Fables	1m47s	6.5	12.1
	Poems	1m07s	6.3	28.3
	Total	35m52s	6.1	11.4

  ], which is more appropriate for long text and for French language syntactic parsing in a near future.

	Unit type	Number
	Paragraphs	23 671
	Sentences	54 393
	Words	799 773
	Orthographically distinct words	38 651
	Phonemically distinct words	28 734
	Non Stop Words	411 210
	Syllables	1 154 714
	Distinct syllables	8 910
	Open	808 503
	Closed	346 211
	Phonemes	2 613 496
	Distinct phonemes	33
	Table 3.3: Amounts of linguistic units in the SynPaFlex corpus

Table 3

 3

	.4.

2b illustrate each speaker's duration proportion in the corpus. The speaker name is encoded as following (F/M: Female/Male, FR: French, ID:XXXX).

Table 3 .

 3 5: Subcorpus contents. The first column corresponds to the title of the novel, and author's name. Nbr. Utts is the number of utterances(sentences), Nbr. Wrd is the number of words in the chapter and Nbr. Syl the number of syllables. The recording type (P) refers to a professional recording, whereas (A) refers to an amateur record.The Siwis French Speech (SFS) voice is the female voice of The SIWIS French Speech Synthesis Database. PODALYDES is a male voice. The speakers FFR0001, FFR0011, FFR0020, and MFR0019 are included in the MUFASA corpus.

Table 3 .

 3 .7). This perceptual test has confirmed that among the six speakers, SFS voice is the easiest to determine the pitch. 7: The frequency of the {/ka/,/ta/,/pa/,/ti/,/ti/,/pi/} in the considered dataset, that have been manually annotated in terms of pitch amplitude.

	book title, author	/ta/ /ka/ /pa/ /ti/ /ki/ /pi/
	Vingt mille lieues sous les mers Chapter 3, Jules Vern	21	33	51	28	27	9
	Mademoiselle Albertine est partie Marcel Proust	30	22	87	58	34	2

  Durations and amount of annotated data according to discourse mode in the first version of the SynPaFlex-Corpus

		Duration Discourses Expressivity
	genre		annotation	annotation
	Novels	80h12m	27h21m	10h59m
	Short stories	5h01m	4h08m	2h26m
	Tales	1h22m	1h22m	10m
	Fables	18m	18m	/
	Poems	29m	29m	/
	Total	87h23m	33h39m	13h25m
	Table 4.2:

Table 4

 4

	.3: Manual annotations -Total duration of intonation patterns (including combina-
	tions) in the 13h25 sub-corpus

A non-nopip tag has been assigned to 68% of the speech. As shown in Table

4

.3, the

  Table 4.6.

	Emotion	Idle	Surprise Sadness Joy	Anger Disgust Fear	Irony Threat
	Duration	8h11m	4h42m	44m	32m	31m	15m	11m	10m	3m
	Sub-corpus %	61.0%	34.8%	5.4 %	3.9 %	3.9%	1.9%	1.3%	1.2%	0.4%

Table 4

 4

	.4: Manual annotations -Total durations of emotion categories labels (including
	combinations) in the 13h25 sub-corpus

  Table 4.5.

	Emotion	Surprise	Sadness	Joy	Anger	Disgust	Fear
	Effects on
	the first syllable accentuation disappearance		accentuation accentuation
	of focus word(s)
	Pitch median	high	low	according to	low	low	low
				joy type
	Pitch curve		flat	flat (suave joy)	flat or	flat or	flat
					top-down	top-down
	Rate		slow	according to	fast	fast on	varying with
				joy type		focus words fear intensity
	Loudness		low	loud		low
				(intense joy)
	Timbre changes		breath during	breath during		yes	yes
			the speech	the speech
				(suave joy)

Table 4 .

 4 5: Examples of perceived impacts of emotion on the speech

  Table 4.6), Idle segments have the durations (8.76s), then comes Surprise segments (3.83s.) and Combination labels (3.45s.).

	Emotion	Idle Anger	Joy Sadness	Fear Surprise Disgust Other Comb.	Total
	# Seg. manual	3 364	147	115	295	76	2 895	47	23	1 699	8751
	Avg. dur (s)	8.76	2.62 2.99	2.67	2.20	3.83	2.26	2.30	3.45	5.55
	# Seg. 1 s. max 30 989	447	397	929	199	12 794	125	0	0 45 880

Table 4.6: Number of manually annotated emotional segments and segments resulting from a 1 s. max chunking. The latest are used in the classification experiments. Other includes Irony and Threat labels.

  .7 represent the description of the acoustic features) are extracted on emotional segments with OpenSmile toolkit and Interspeech 2009 configuration[START_REF] Schuller | The interspeech 2009 emotion challenge[END_REF].

	Low-Level-Descriptors (LLD)	Functionals
	zero-crossing-rate (ZCR) +∆	mean
	root mean square (RMS) Energy +∆	standard deviation
	Fundamental Frequency (F0)+∆	kurtosis, skewness
	Harmonics-to-noise ration (HNR) +∆	min. and max. value, relative position, range
	Mel-Frequency Cepstral Coefficients (MFCC) 1-12 + ∆ linear regression: offset, slope, mean square error (MSE)

Table 4

 4

	.7: Feature set of the INTERSPEECH 2009 Emotion Challenge 384 features, (16
	LLD + 16 ∆)*12 functionals
	To avoid over fitting the data 4 , different subsets of features are tested:
	• OS192: 16 LLD × 12 functionals without ∆
	• ∆ OS192: 16 LLD × 12 functionals with ∆ only
	• OS24: 2 LLD (range + mean) × 12 functionals without ∆

  Entre Saint-Ouen et le chemin de la Révolte, au bout de la route?

	-In the plain of St. Denis.
	-Between St. Ouen and the road of La Revolte, at the end of the road?
	-Agreed.] (Les Mystères de Paris, chap. 7, Tome 1)
	(3) D'autre part, la mort de sa femme ne l'avait pas mal servi dans son métier, car
	on avait répété durant un mois : « Ce pauvre jeune homme ! quel malheur
	! » [In any case, the death of his wife had done him no harm professionally; for a
	whole month people kept saying: «That poor young man! What a terrible
	thing!»] (Madame Bovary, chap. 3)
	-C'est dit.
	[-Be it so; tomorrow at one.
	-At one o'clock.
	-In the plain of St. Denis?

(4) a. -Levez-vous, reprit le professeur, et dites-moi votre nom. [-Stand up, repeated the master, and tell me your name.] (Madame Bovary, chap. 1) b. -Débarrassez-vous donc de votre casque, dit le professeur, qui était un homme d'esprit. Il y eut un rire éclatant des écoliers qui décontenança le pauvre garçon, ... [-I suggest you disencumber yourself of your helmet, said the master, a man of wit. A roar of laughter came from the class and disconcerted the poor lad,...] (Madame Bovary, chap. 1) (5) Puis, l'ayant considéré quelques minutes d'un oeil amoureux et tout humide, elle dit vivement: (Madame Bovary, chap. 18)

Table 6 .

 6 1: Overview of the sub-corpus content. N-utt represent the number of utterances.s

	Book	Nutt *	Direct Discourse (%)	Indirect Discourse (%)	Mixed Discours (%)	FFR0001 FFR0012	Duration (hours)
	Madam Bovary (9 Chapters)	579	25	50	25		∼3h20 (X2)
	Les Mystère de Paris (6 Chapters)	690	43	36	21		∼2h19

Table 6

 6

	.2: Discursive changes distribution sub-corpus
	Audio books Madame Bovary	Les Mystères de Paris
	Discourse	items	syl./item	items	syl./item
	changes	number	number	number	number
	IC ⇒ DD	88	34	554	46
	IC ⇒ DI	20	49	56	55
	DD ⇒ DI	73	73	172	88
	DI ⇒ DD	61	61	129	54
	DD ⇒ IC	107	15	528	26
	DD ⇒ DD	39	29	1195	78

Table 6 .

 6 3: Means and standard deviations for F 0 -range and articulation rate (AR) for the different types of discourse change.

	Parameters First Disc.	Second Disc.
		Last BG	First BG
		IC	DD	p-value
	F0range,st	5.91 (5.76)	7.47 (5.91)	2.0
	AR,syl./s	5.13 (0.77)	5.25 (3.17)	1.64
		DD	IC	p-value
	F0range,st 7.08 (5.91)	4.43 (4.94) <0.001
	AR,syl./s	4.78 (1.22)	5.11 (0.80)	2.0
		DD	DD	p-value
	F0range,st 8.72 (5.64)	7.42 (5.78) <0.001
	AR,syl./s	5.15 (0.91)	5.13 (2.70)	0.788
		DD	ID	p-value
	F0range,st 8.78 (5.66)	7.68 (5.25) 0.0235
	AR,syl./s	4.98 (0.96)	5.23 (3.06)	1.88
		ID	DD	p-value
	F0range,st 11.23 (5.35) 6.94 (5.67) <0.001
	AR,syl./s	4.95 (0.96)	5.04 (1.99)	1.44
		IC	ID	p-value
	F0range,st	6.49 (5.88)	8.11 (5.62)	1.91
	AR,syl./s	4.95 (0.68)	4.98 (0.74)	1.23

Table 6 .

 6 4: Means and standard deviations for Inter-Breath Group Pause Duration (IBGP) according to different types of discourse change.

	First Disc. ⇒ Second Disc. IBGP (in s.)
	IC ⇒ DD	0.67 (0.27)
	DD ⇒ IC	0.43 (0.17)
	DD ⇒ DD	0.95 (0.23)
	DD ⇒ ID	1.26 (0.39)
	ID ⇒ DD	1.02 (0.24)
	IC ⇒ ID	1.11 (0.32)
	may be related to the fact that the speakers are of the same sex. Further investigations
	are needed to assess how other speakers behave.
	Table 6.5: Comparing IBGP across the different discourse changes modalities (** represents
	p-value<0.001).

Table 7 .

 7 1: Objective results for multi-speaker modeling, considering five speaker code configurations. Mel-Cepstral Distortion (MCD), Band Aperiodicity Parameter (BAP), Root Mean Square Error (RMSE), Voiced/Unvoiced (VUV) and Correlation (CORR) between the predicted and the original coefficients. For the F 0 , RMSE and CORR are computed on the voiced frames only.

	OHV	X-Vector	P-Vector	MCD (dB) BAP (dB)	F0 RMSE (Hz) CORR	VUV	Duration RMSE (ms) CORR
				5.833	0.301	32.597	0.807 8.950	9.232	0.558
				5.935	0.303	33.018	0.801 8.971	8.889	0.601
				5.748	0.296	32.203	0.811 8.851	8.883	0.604
				5.756	0.297	32.169	0.810 8.944	8.860	0.607
				5.755	0.297	32.043	0.812 8.915	8.836	0.609
	6.2

Visualizing the first hidden-layer output

  La Garde nationale qui, depuis deux mois, faisait des reconnaissances très prudentes dans les bois voisins, fusillant parfois ses propres sentinelles, et se préparant au combat quand un petit lapin remuait sous des broussailles, était rentrée dans ses foyers. Ses armes, ses uniformes, tout son attirail meurtrier, dont elle épouvantait naguère les bornes des routes nationales à trois lieues à la ronde, avaient subitement disparu. [ The members of the National Guard, who

	Boule de Suif :
	Paragraph
	(P.1)

  (S.1) La Garde nationale qui, depuis deux mois, faisait des reconnaissances très prudentes dans les bois voisins, fusillant parfois ses propres sentinelles, et se préparant au combat quand un petit lapin remuait sous des broussailles, était rentrée dans ses foyers. [ The members of the National Guard, who for the past two

	(S.2) Ses armes, ses uniformes, tout son attirail meurtrier, dont elle épouvantait
	naguère les bornes des routes nationales à trois lieues à la ronde, avaient subitement
	disparu. [ Their arms, their uniforms, all the death-dealing paraphernalia with
	which they had terrified all the milestones along the highroad for eight miles round,
	had suddenly and marvelously disappeared.]
	IPUs
	(IUP.1) La Garde nationale qui, [The members of the National Guard, who]
	(IUP.2) depuis deux mois, faisait des reconnaissances très prudentes dans les bois
	voisins, [for the past two months had been reconnoitering with the utmost caution
	in the neighboring woods,]
	(IUP.3) fusillant parfois ses propres sentinelles, [occasionally shooting their own
	sentinels,]
	months had been reconnoitering with the utmost caution in the neighboring woods,
	occasionally shooting their own sentinels, and making ready for fight whenever a
	rabbit rustled in the undergrowth, had now returned to their homes.]
	110

(IUP.4

) et se préparant au combat quand un petit lapin remuait sous des broussailles, [and making ready for fight whenever a rabbit rustled in the undergrowth, ] (IUP.5) était rentrée dans ses foyers. [had now returned to their homes. ] (IUP.6) Ses armes, ses uniformes, [Their arms, their uniforms,] (IUP.7) tout son attirail meurtrier, dont elle épouvantait naguère les bornes des routes nationales à trois lieues à la ronde [all the death-dealing paraphernalia with which they had terrified all the milestones along the highroad for eight miles round, ] (IUP.8) avaient subitement disparu.

[had suddenly and marvelously disappeared.]

Table 7 .

 7 ] End-to-End framework (the tacotron network architecture is illustrated in Figure E.1). The results are better in terms of quality compared to the architectures presented and used during the thesis work. Nevertheless, these techniques 2: Objective results of the acoustic model, considering the three granularity. Mel-Cepstral Distortion (MCD), Band Aperiodicity Parameter (BAP), Root Mean Square Error (RMSE), Voiced/Unvoiced (VUV) and Correlation (CORR) between the predicted and the original coefficients. For the F 0 , RMSE and CORR are computed on the voiced frames only.

	test/train set granularity	IPU
	measures	MCD BAP F0-RMS F0-CORR UV
	IPU	5.125 0.193 35.040	0.404	6.954
	Sentence	5.155 0.193 35.033	0.400	7.188
	Paragraph	5.165 0.195 35.233	0.404	7.261
	test/train set granularity	Sentence
	IPU	5.102 0.194 33.358	0.511	7.348
	Sentence	5.179 0.195 34.253	0.492	7.247
	Paragraph	5.126 0.193 33.431	0.510	6.983
	test/train set granularity	Paragraph
	IPU	5.358 0.187 33.790	0.496	6.648
	Sentence	5.255 0.180 37.093	0.411	6.349
	Paragraph	5.128 0.177 34.108	0.472	6.486

have a couple of constraints:

  Mystères de Paris vol.1 and 2"

			Appendix C
	# Echange entamer manquer de ..	L'histoire de Koli, Franz de Zeltner Table A.2: (continued) Table A.2: (continued) Table A.2: (continued) Table A.2: (continued) * Table A.2: (continued) Table A.2: (continued) Table A.2: (continued) Table A.2: (continued) Table A.3: MUFASA Parallel Subcorpus Table A.3: (continued) insister avancer (s')incliner proposer rugir haleter	4m		-TAzy
	Eugène Sue Les misérables, Victor Hugo Madame Bovary, Gustave Flaubert Le Novel de la momie, Théophile Gautier Germinal, Émile Zola La vampire, Paul Féval Voyage au centre de la terre, Jules Verne La Vénus d'Ille, Prosper Mérimée La fille du pirate, Maurice Chevalier Carmen, Prosper Mérimée Tales cruels, Auguste Villiers de l'Isle-Adam La malle volante, Andersen Le monstre Yatama, Claudius Ferrand Les sept chevreaux, Claudius Ferrand Lettres persanes, Montesquieu Fantastic novels and short stories Adventure novel and short stories Symbolism short stories Epistolary Novel Fables Fables de La Fontaine Fantastic epic Les chants de Maldoror, Comte de Lautréamont Fantastic dramatic Infernaliana, Charles Nodier Poems L'albatros, Charles Baudelaire * Chanson d'automne, Paul Verlaine * Le Dormeur du val, Arthur Rimbaud * Fiez vous y !, Charles d'Orléans * Gaudriole en six couplets, unknown * Un matin, Emile Verhaeren * Perles, Jean Courdil * La veuve indienne, Eugène Fouques * Pamphlet Le cerf-volant aux six têtes, Guillaume Taillerand-Perigord * MUFASA Corpus Genre Title, author, date Spkid () Tale La Reine Ysabeau, Auguste de Villiers de L'Isle-Adam,1893 FFR0017* Cocotte * 25h 32m 14h 16m 2h 18m HNm 22m HNm * 13h 12m 26m HNm * 6h 33m -HNro 1h 46m 50m HNge * 10h 02m 2h 26m FNva 1h 52m 22m FNvo * 1h 02m -FSve * 6h 43m 4h 31m ANfi * 2h 18m 53m ASca 1h 42m 24m SYco * 12m -TAan * 8m -TAfx * 16m -TAfy 20m -ENle 18m 5m FAfo 18m 7m FEch 11m -FDin 1m -POal 1m -POch 1m -POdo 1m 1m POfi 3m -POga 1m -POma 1m -POpe 4m -POve 6m 6m PAce Spk Dur 15,34 Tale Le tailleur de Catanzaro, Alexandre Dumas,1870 FFR0017* Cocotte 24,77 Poem Les cailloux , Gaston Coute,1978 MFR0005* Alain 1,12 Tale Roland, de retour de Roncevaux, Alexandre Dumas,1870 FFR0017* Cocotte 8,55 Novel Voyage fait en la terre du Brésil, Jean de LÉRY,1578 MFR0019* Damien Genevois 942,29 correspondance la Grande Guerre, Ernst Wittefeld,1914 FFR0012* Victoria 26,64 Short story A quoi rêvent les pauvres filles, Emile Zola,1870 MFR0003* Dousset 5,81 Poem APRES VENDANGES, Gaston Coute,1978 MFR0005* Alain 3,05 theatre AUTREFOIS, Charle Cros,1881 MFR0003* Dousset 6,14 Poem Alouettes, Saint-Pol-Roux,1901 MFR0003* Dousset 2,69 Short story Aventure sans pareille d'un certain Hans Pfaall, Edgar Allan Poe,1835 FFR0007* Cecile 123,68 Short story Berthe aux grands pieds, André Rivoire,1899 FFR0011* Pomme 52,53 Tale Blanche-Neige, Grimm,1812 MFR0013* DanielLuttringer 18,26 MUFASA Corpus Genre Title, author, date Spkid () Spk Dur Novel David Copperfield, Charles Dickens,1850 FFR0012* Victoria 49,72 Short story Dernier voeu, Théophile Gautier,1852 MFR0015* Jean-LucFischer 0,74 Short story Deux acteurs pour un rôle, Théophile Gautier,1841 MFR0013* DanielLuttringer 19,77 Short story En voyage, Guy de Maupassant,1882 MFR0013* DanielLuttringer 11,83 philosophie Euthyphron, Platon,399 av. J-C MFR0003* Dousset 57,68 Fable Fables de La Fontaine, De la Fontaine,1668 FFR0001 Nadine 18,09 Tale Fables et légendes du Japon, Claudius Ferrand ,1903 FFR0001 Nadine 39,37 Short story Facino Cane, Honore de Balzac,1836 MFR0013* DanielLuttringer 31,48 Poem Fiez-Vous-Y, Charles d'Orléan,1450 FFR0001 Nadine 0,6 Novel Filles, lorettes et courtisanes, Alexandre Dumas,1843 MFR0005* Alain 163,69 Short story Gustave Flaubert, Guy de Maupassant,1884 MFR0013* DanielLuttringer 13,81 Poem Géorgiques, Virgile,30 av. J.-C. MFR0005* Alain 152,5 Short story Infernaliana, Charles Nodier,1822 MFR0014* ReneDepasse 8,84 MUFASA Corpus Genre Title, author, date Spkid () Spk Dur Novel L'art De Payer Ses Dettes, Émile Marco de Saint-Hilaire,1911 FFR0009 Ezwa 143,43 Tale L'expiation du roi Rodrigue, Alexandre Dumas,1870 FFR0017* Cocotte 17,72 Tale LA PORTE DES CENT MILLE PEINES, Anonyme,1918 MFR0013* DanielLuttringer 14,96 Tale La Belle au bois dormant, Grimm,1812 FFR0017* Cocotte 8,37 Short story La Chambre 11, Guy de Maupassant,1884 FFR0012* Victoria 15,61 Tale La Chèvre de Monsieur Seguin, Alphonse Daudet,1887 FFR0017* Cocotte 14,87 Novel La Comtesse de Cagliostro, Maurice Leblanc,1924 MFR0002* Menager 451,03 Short story La Confession, Guy de Maupassant,1883 FFR0012* Victoria 14,43 Novel La Cousine Bette, Honore de Balzac,1846 FFR0007* Cecile 1006,16 Novel La Demoiselle aux yeux verts, Maurice Leblanc,1927 MFR0013* DanielLuttringer 410,27 Novel La Fille Du Pirate, ÉMILE Chevalier,1878 FFR0001 Nadine 4,64 Short story La Fille aux yeux d'or, Honore de Balzac,1833 MFR0010 Graigolin 118,92 Short story La Folie de John Harned, Jack London,1912 MFR0005* Alain 47,56 MUFASA Corpus Genre Title, author, date Spkid () Spk Dur Short story La Petite Roque, Guy de Maupassant,1885 FFR0012* Victoria 88,96 Novel La Princesse de Montpensier, Madame De Lafayette,1662 FFR0012* Victoria 72,76 Poem La Revanche du Passé, Eugénie Pradez,1900 FFR0011* Pomme 321,69 Tale La Tarasque, Alexandre Dumas,1870 FFR0017* Cocotte 21,13 Novel La Tulipe noire, Alexandre Dumas,1850 FFR0009 Ezwa 21,94 Tale La Vision du Juge de Colmar, Alphonse Daudet,1880 MFR0013* DanielLuttringer 9,21 Tale La chèvre, le tailleur et ses trois fils, Alexandre Dumas,1838 FFR0017* Cocotte 36,28 Novel La fille du pirate, ÉMILE Chevalier,1878 FFR0001 Nadine 398,68 Tale La fée des eaux, Alexandre Dumas,1870 FFR0017* Cocotte 25,11 Fable La jeune veuve, De la Fontaine,1668 FFR0017* Cocotte 3,1 Tale La jeunesse de pierrot, Alexandre Dumas,1854 FFR0017* Cocotte 143,93 Short story La jeunesse de pierrot, Alexandre Dumas,1854 FFR0017* Cocotte 7,78 theatre La jeunesse de pierrot, Alexandre Dumas,1854 FFR0017* Cocotte 4,51 MUFASA Corpus Genre Title, author, date Spkid () Spk Dur Short story La moustache, Guy de Maupassant,1883 FFR0017* Cocotte 12,29 Tale La petite sirene, Alexandre Dumas,1860 FFR0017* Cocotte 83,81 Tale La reine des neiges, Alexandre Dumas,1860 FFR0017* Cocotte 98,22 Tale La reine des poissons, Gerard de Nerval,1850 FFR0016* Corinne 7,71 Tale La sirène du Rhin, Alexandre Dumas,1870 FFR0017* Cocotte 29,88 Novel La vampire, Paul Féval,1865 FFR0001 Nadine 602,38 Short story La vengeance d'une femme, Jules Barbey d'Aurevilly,1883 MFR0002* Menager 88,03 Short story Le Bifteck, Jack London,1911 MFR0005* Alain 48,8 Novel Le Capitaine Fracasse, Théophile Gautier,1863 FFR0016* Corinne 1307,65 Novel Le Cauchemar d'Innsmouth, Lovecraft,1936 MFR0015* Jean-LucFischer 198,07 Short story Le Chat noir, Edgar Allan Poe,1843 FFR0012* Victoria 28,39 Fable Le Chat, la Bellette, & le petit Lapin, De la Fontaine,1678 FFR0017* Cocotte 3,04 Novel Le Dernier des Mohicans, James Fenimore Cooper,1826 MFR0006 Bernard 1036,84 MUFASA Corpus Genre Title, author, date Spkid () Spk Dur Poem Le déraillement, Gaston Coute,1978 MFR0005* Alain 0,68 Short story Le fifre rouge, Paul_Arene,1887 FFR0017* Cocotte 10,12 Tale Le grand mouton noir, Abbe Baudiau,1854 FFR0016* Corinne 5,82 Novel Le nez d'un notaire, Edmond About,1862 MFR0008 Didier 166,8 Novel Le nez d'un notaire, Edmond About,1862 MFR0014* ReneDepasse 190,61 Short story Le pont du diable, Alexandre Dumas,1844 FFR0016* Corinne 14,25 Novel Le Novel de la momie, Théophile Gautier,1857 FFR0001 Nadine 388,89 Novel Le Novel de la momie, Théophile Gautier,1857 MFR0014* ReneDepasse 468,49 Tale Le vilain petit Canard, Hans Christian Andersen,1876 MFR0013* DanielLuttringer 20,85 Tale Le_Conseiller_Krespel, ETA Hoffmann,1967 MFR0005* Alain 68,85 Poem Les Amoueuses Trois jours de vendange, Alphonse Daudet,1908 FFR0017* Cocotte 1,36 Poem Les Amoureuses Le Rouge Gorge, Alphonse Daudet,1908 FFR0017* Cocotte 3,64 Poem Les Amoureuses Le croup, Alphonse Daudet,1908 FFR0017* Cocotte 3,07 MUFASA Corpus Genre Title, author, date Spkid () Spk Dur Novel Les Chouans, Honore de Balzac,1829 FFR0016* Corinne 919,58 Tale Les Deux Chemises, Alexandre Dumas,1870 FFR0017* Cocotte 11,22 societe Les Formes élémentaires de la vie religieuse, Émile Durkheim,1912 FFR0004* Julie 648,09 Poem Les Mangeux d'terre, Gaston Coute,1978 MFR0005* Alain 2,73 biographie Les Rustiques-Un Point d'Histoire, Louis Pergaud,1921 MFR0005* Alain 18,54 Novel Les Temps difficiles, Charles Dickens,1854 MFR0003* Dousset 858,9 Short story Les Trois Dames de la Kasbah, Pierre_Loti,1884 FFR0011* Pomme 61,3 Poem Les accroche-coeurs, Théophile Gautier,1852 MFR0015* Jean-LucFischer 0,73 Tale Les aventures du chardon, Hans Christian Andersen,1873 FFR0017* Cocotte 11,75 Tale Les deux bossus, Alexandre Dumas,1870 FFR0017* Cocotte 9,32 Tale Les onze mille vierges, Alexandre Dumas,1870 FFR0017* Cocotte 6,93 Short story Les présents des gnomes, Grimm,1864 MFR0015* Jean-LucFischer 4,51 Poem Les tâches, Gaston Coute,1978 MFR0005* Alain 2,21 MUFASA Corpus Genre Title, author, date Spkid () Spk Dur Novel Monsieur Lecoq, Emile Gaboriau,1869 FFR0001 Nadine 40,1 Novel Mémoires d'un jeune homme rangé, BERNARD Tristan,1899 FFR0011* Pomme 359,9 Short story Notre Dame de la Mort, Arthur Conan Doyle,1910 FFR0016* Corinne 102,13 Tale Nouveaux Tales de Fées Pour les Petits Enfants,Comtesse de Ségur,1857 FFR0009 Ezwa 328,39 Short story Novembre, Gustave Flaubert,1842 FFR0012* Victoria MUFASA Parallel Subset Genre Title, author, date Dur Spkid Spk Novel Albertine Disparue, Marcel Proust, 1925 325,19 FFR0020 Orangeno 393,26 FFR0011 Pomme Short story Boule de Suif, Guy de Maupassant, 1879 90,89 FFR0009 Ezwa 97,33 FFR0012 Victoria 82,32 MFR0015 Jean-Luc Fischer Short story Carmen, Prosper Mérimée, 1845 137,84 FFR0001 Nadine 150,57 MFR0014 Rene Depasse Novel Cinq Semaines en Ballon, Jules Verne, 1863 576,1 MFR0006 Bernard 564,57 FFR0020 Orangeno MUFASA Parallel Subset Genre Title, author, date Dur (min) Spk bavarder intercéder deviner entrer en matière louer (faire un compliment) # Interdiction #Colère chuchoter miauler Discourses Annotation confier inventorier énoncer finir obtempérer interdire Colère murmurer minauder Spkid Fiction La petite comtesse, Octave Feuillet, 1857 186,26 converser juger estimer interrompre opiner prohiber aboyer #Tentative postillonner MFR0013 Daniel Luttringer 205,81 deviser lister examiner intervenir préférer refuser apostropher essayer psalmodier FFR0011 Pomme 224,64 dialoguer mettre en garde imaginer poursuivre réaliser résister bougonner (se) lancer susurrer MFR0014 Rene Depasse Novel Le nez d'un Notaire, Edmond About, 1862 166,8 discourir minimiser hasarder répéter reconnaître #Exigence cracher risquer #Pédant MFR0008 Didier 190,61 MFR0014 Rene Depasse Novel Le Novel de la Momie, Théophile Gautier, 1857 388,89 papoter plaider jauger répondre renchérir ger (s')enflammer tenter annoter 1 Speech Verbs List parler présenter proposer reprendre la parole réviser (son opinion) intimer (quelqu'un de parler) (s')emporter vérifier commenter FFR0001 Nadine 468,49 saluer rajouter supputer terminer souscrire obliger (s')étrangler #Hésitation disserter MFR0014 Rene Depasse Poem Les chants de Maldoror, Comte de Lautréamont, 1869 23,13 #Neutre # Argumentation # Désaccord # Question rappeler # Réticence / regret # Moquerie #Humour ordonner enguirlander Hésitation fanfaronner FFR0018 Naf 17,61 affirmer alléguer accuser demander rapporter admettre ironiser Humour sommer exploser décider (se) gargariser FFR0001 Nadine affranchir (apprendre à qqun) apprendre (à quelqu'un) combattre interroger récapituler (se) décider (se) moquer badiner #Façon de parler grincer hésiter (se) glorifier 49,97 Short story Novembre, Gustave Flaubert,1842 FFR0012* Victoria Tale Tales Cruels, Auguste de villiers de L'isle Adam, 1883 102,11 FFR0001 Nadine 113,72 MFR0014 Rene Depasse Les Milles et une nuit, 398,96 MFR0015 Jean-Luc Fischer apprendre arguer contester questionner requérir lâcher narguer blaguer ânonner grogner risquer gloser Tale Anonyme, X siècle 153,8 FFR0007 Cecile assurer argumenter contredire (s')enquérir résumer regretter persifler éclater de rire articuler grommeler monologuer 146,12 Short story Noël, Théophile Gautier,1872 MFR0015* Jean-LucFischer 0,74 Poem Passage du poète, Charles-Ferdinand Ramuz,1923 FFR0011* Pomme 256,29 Poem Petit Poucet, Gaston_Coute,1978 MFR0005* Alain 1,54 Short story Petite discussion avec une momie, Edgar Allan Poe,1845 FFR0007* Cecile 38,22 autre Physiologie du goût, Jean Anthelme Brillat-Savarin,1825 MFR0003* Dousset Tale Tales de la Bécasse, Guy de Maupassant, 1883 188,64 MFR0006 Bernard 65,31 MFR0008 Didier Fables, Jean Pierre 180,72 FFR0017 Cocotte Novel 849,49 MFR0008 Didier aviser assener critiquer (s')instruire révéler # Demanander une faveur railler (s')esclaffer babiller gronder palabrer Les Misérables, 856,21 FFR0001 Nadine commenter assurer démentir # Réponse signaler adjurer # Honte glousser bafouiller (s')impatienter pérorer Victor Hugo, 1862 68,77 FFR0018 Naf considérer avancer dénoncer éluder souligner demander avouer gouailler balbutier injurier philosopher Fable Clarisde Florian, 1792 259,88 FFR0009 Ezwa Germinal, 105,83 FFR0001 Nadine Les mystères de Paris, 998,42 MFR0013 Daniel Luttringer conter (se) dédouaner discuter expliquer soutenir exhorter confesser plaisanter balbutier insulter plastronner Novel Eugène Sue, 1843 1531,55 FFR0001 Nadine déclarer (se) défendre douter indiquer tenter de convaincre implorer (s')excuser pouffer baragouiner piaffer (d'impatience) pontifier Novel Emile Zola, 1885 123,2 FFR0011 Pomme Infernalia, 48,98 MFR0010 Graigolin Madame Bovary, 791,69 FFR0001 Nadine décrire détailler huer répliquer # Accord négocier marmonner (se) réjouir bégayer proférer (des menaces) prophétiser Novel Gustave Flaubert, 1857 784,18 FFR0012 Victoria dire distinguer infirmer répondre accorder parlementer souffler rire bredouiller râler rabâcher Tale Charles Nodier, 1822 10,58 FFR0001 Nadine L'épouvante, 308,84 FFR0009 Ezwa Raison et sensibilité, 918,54 FFR0007 Cecile émettre (un son) égrener (s')insurger #Promesse acquiescer pleurer #Tristesse / douleur sourire cafouiller réprimander réciter Novel Jane Austen, 1857 849,85 MFR0013 Daniel Luttringer exprimer émettre(une opinion) nier jurer adhérer prier compatir #Volume chantonner ronchonner seriner Fiction Maurice Level, 1908 370,93 MFR0014 Rene Depasse Short Un coeur simple, 95,51 MFR0003 Dousset formuler énumérer (s')offusquer mentir admettre quémander geindre acclamer couiner siffler soliloquer 907 histoire Principes et motifs du plan de Constitution, Nicolas de Condorcet,1793 MFR0003* Dousset 41,28 Short story Promenade, Guy de Maupassant,1884 FFR0012* Victoria 16,07 La comtesse Cagliostro, 166,29 MFR0013 Daniel Luttringer story Gustave Flaubert, 1877 87,65 MFR0014 Rene Depasse narrer exagérer protester promettre approuver réclamer gémir appeler crachoter s'offusquer traduire Novel Maurice leblanc, 1924 159,91 MFR0003 Menager La Princesse de Clèves, 336,33 FFR0017 Cocotte Vingt mille lieues sous les mers, 11,71 FFR0001 Nadine observer exposer rectifier # Déroulement du dialogue capituler revendiquer (s')inquiéter beugler crépiter tempêter Novel Jules Verne, 1870 901,52 MFR0019 Damien Genevois parler faire miroiter remettre en question achever composer solliciter (se) plaindre brailler débiter tonner Novel Madame Lafayette, 1678 407,85 FFR0011 Pomme Short La vengeance d'une Femme, 32,56 MFR0003 Menager Voyage au centre de la terre, 111,64 FFR0001 Nadine penser tout haut faire remarquer renâcler (s')adresser concéder suggérer rassurer bramer déclamer vilipender Novel Jules Verne, 1864 1209,74 MFR0019 Damien Genevois préciser garantir reprendre (contredire) ajouter confirmer supplier # Surprise clamer dégoiser vitupérer
	Tales Short story theatre Short story Novel Poem Short story Tale Novel story Genre Jules Barbey d'Aurevilly, 1874 Ourashima Taro et la déesse de l'Océan, Claudius Ferrand Bombard, Guy de Maupassant,1884 FFR0018 Inscription, Charle Cros,1908 MFR0003* Dousset * Naf La Main , Guy de Maupassant,1883 FFR0012* Victoria La maison à vapeur, Jules Verne,1880 FFR0020* Orangeno 16m Le Dormeur du val, Arthur Rimbaud,1870 FFR0001 Nadine Les Bords du Sacramento, Jack London,1922 MFR0005* Alain Les voleurs et l'âne, Emile Zola,1864 FFR0017* Cocotte Robur le conquérant, Jules Verne,1886 FFR0020* Orangeno 104,6 MFR0014 Rene Depasse Title, author, date Dur Spkid Spk raconter glisser rétorquer arrêter croire # Permission Surprise crier entonner vociférer Short La Vénus D'Ille, 61,74 FFR0001 Nadine remarquer indiquer riposter compléter choisir accepter (s')etonner (s')égosiller épeler vomir des injures	-TAfz 11,37 2,78 15,73 841,35 1,15 22,77 39 422,7
	La Hyène, l'Hippopotame et l'Éléphant, Franz de Zeltner Title, author, date Spkid Title, author, date Spkid Title, author, date Spkid Title, author, date Spkid Title, author, date Spkid Title, author, date Spkid Title, author, date Spkid Title, author, date Spkid Posper Mérimée, 1837 Genre Genre Genre Genre Genre Genre Genre Genre story 74,76 MFR0014 Rene Depasse * Spk Spk Spk Spk Spk Spk Spk Spk rappeler innocenter tempérer conclure féliciter encourager (s')étouffer héler éternuer Genre Title, author, date Dur Spkid Spk (se) souvenir insinuer # Enigme couper flatter permettre (s')exclamer hurler faire	11m	Dur Dur Dur Dur Dur -TAzx Dur Dur Dur
		117

Les enregistrements amateurs sont des enregistrements audio dont la destination primaire n'était pas pour faire de la synthèse de parole et dont les conditions d'enregistrements ne sont pas connues.

Les enregistrements professionnels en revanche qu'on a eux ont été conçus pour développer des voix de synthèse. Les conditions d'enregistrement sont propres.

Le narrateur est aussi considéré comme un personnage

Le surapprentissage est notion du domain d'apprentissage automatique, qui fait référence aux modèles peu généralisable

This project is funded by the National French Research Agency (ANR)

The reader interested in an in-depth discussion of the concepts related to cognitive load, speech production, and perception can refer to[Christodoulides

2016]

http://freelcs.sourceforge.net/

https://www.gutenberg.org/wiki/Main_Page

https://fr.m.wikisource.org/wiki/Wikisource:Accueil

http://alpage.inria.fr/statgram/frdep/fr_stat_dep_bky.html

The logarithmic semitone scale seems to be the appropriate measure of the perceptual consequences of differences in fundamental frequency

https://librivox.org/

http://www.litteratureaudio.com/

recordings dedicated to the synthesis of speech, with favorable acoustic conditions and voice selection for this purpose unlike amateur recording.

"For normal speech, alternate cycles usually appear in creaky voice or voice with laryngealization, which are often characterized as perceptually rough voices. In pathological voice, alternate cycles can be found even in normal mode of production."[Sun 2002]

https://bigdataspeech.github.io/TP/tp/2018/07/10/TPPraat.html

Bark is a psycho-acoustical scale closer to subjective perceptual scale

http://drammock.github.io/phonR/

http://www.speech.kth.se/wavesurfer/

The expert in question is Gaëlle Vidal (gaelle.vidal@irisa.fr), who collected and annotated the first version of the SynPaFlex corpus.

The binary classification consists of assigning a given sample to one of two categories by relying on a set of attributes (features). In the case of a multiclass problem, as is the case in our case, it is possible to reduce and simplify the multiclass classification problem into a set of binary classification problems by considering two methods (i) one-versus-rest method, which makes a series of binary classifications where each model consider a class versus the others classes (ii) one-versus-one method, each binary model consider only samples from two classes at a time. In our experiments, we use the one-versus-one method because most studies have shown that this technique is more efficient than in one-versus-rest.

Over-fitting in statistic and machine learning is a model that is too close to the data is made from (training data) even the noisy ones, but can not be generalized to new coming data, for instance, test data. This model achieves very high performance in the training phase and mediocre in the phase test. Most of the time, this phenomenon is due to the size of the training data too small in comparison to the number of parameters of the model.

https://radimrehurek.com/gensim/models/doc2vec.html

https://code.google.com/p/word2vec/

https://kaldi-asr.org/models/m3

https://david-ryan-snyder.github.io/2017/10/04/model_sre16_v2.html

We choose PCA to find out the independent variables that hold the speaker's identity.

https://github.com/AghilasSini/merlin/tree/master/egs/speaker_adaptation

https://github.com/AghilasSini/SpeechProsody2020

Acknowledgement

Automatic Annotation of Discourses

This chapter is an extended version of the work described in "Automatic annotation of discourse types in audiobooks" [START_REF] Sini | Annotation automatique des types de discours dans des livres audio en vue d'une oralisation par un système de synthèse[END_REF] presented at Traitement Automatique de Langues Naturelles (TALN) 2018.

.580 .633 Dis.

.584

Table 4.8: Unweighted Average Recall (UAR) results for binary emotion classification using the three feature subsets. In bold, UAR > 60%, which we considered as a reasonable classification rate.

The second group clearly contains negative emotions with different arousal levels.

Further experiments are needed to deeper investigate these groups such as unsupervised clustering, feature selection, etc. For example, ∆ features are clearly relevant for Anger/Disgust classification. Moreover, emotions are likely to be strongly correlated with direct/indirect speech and also with characters. Additional analyses are required to confirm this observation. The addition of phonological and linguistic information could also help in understanding the emotional distribution of the SynPaFlex corpus.

Emotion Lexicon Study of Audiobooks

Given the difficulty encountered in classifying and recognizing emotional patterns at the acoustic analysis level, we suggest exploring the expressive properties conveyed by lexical and textual structures such as sentences in audiobooks.

In audiobooks, the written text holds a substantial portion of the expressivity. We

Chapter 5

[ People coming out of the theatre passed along the pavement, humming or shouting at the top of their voices, "O bel ange, ma Lucie!" [DD] Then Leon, playing the dilettante, began to talk music. [ND]He had seen Tambourini, Rubini, Persiani, Grisi, and, compared with them, Lagardy, despite his grand outbursts, was nowhere. [FID] Oh beautiful angel, my Lucie! "Yet," interrupted Charles, who was slowly sipping his rum-sherbet, "they say that he is quite admirable in the last act. I regret leaving before the end, because it was beginning to amuse me." [DD] "Why," said the clerk, "he will soon give another performance." [DD] But Charles replied that they were going back next day.[ID] "Unless," he added, turning to his wife, "you would like to stay alone, kitten?"

[DD]

And changing his tactics at this unexpected opportunity that presented itself to his hopes, the young man sang the praises of Lagardy in the last number. It was really superb, sublime. [FID]] (Madame Bovary, chap. 15, Part 2)

In our approach, we have decided to consider the indirect discourse, the free indirect discourse, and narrative discourse as a single entity that we call indirect discourse. This decision is motivated by the fact that these three discourse types represent words that the narrator says, even if they might express different points of view. By contrast, direct discourse corresponds to words that are said by a character involved in the story.

In [START_REF] Laferrière | Les Incises dans les genres narratifs[END_REF]], the author pointed out the importance of the so used Incidental Clauses with reporting verbs in narrative genres texts. This particular structure is often used within DD (dialogues or monologues) to describe the characters state or to initialize the DD.

In our annotation, we consider three main categories for modeling fictional texts (novels, short stories, and tales) :

• DD: containing the dialogues and monologues as specified, (see example (2)

• ID: covering indirect discourse, free indirect discourse, and narrative discourse.

Narrative discourse in (1),

• Mixed discourse involving DD, ID, and Incidental Clauses with reporting verbs pairwise : in example (3), illustrate the combination of direct discourse with indirect discourse; (4.a) and ( 5) contains DD and IC.

Experimental setup

To perform the classification task, we first consider the model algorithm proposed in [START_REF] Schöch | Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels[END_REF], where the authors used traditional machine learning models such as Support Vector Machine (SVM) [START_REF] Chang | LIBSVM: A library for support vector machines[END_REF], Maximum Entropy [START_REF] Nigam | Using maximum entropy for text classification[END_REF], Naïve Bayes [START_REF] John | Estimating continuous distributions in Bayesian classifiers[END_REF], Random Forest [START_REF] Liaw | Classification and regression by randomForest[END_REF] and JRip [START_REF] Cohen | Fast Effective Rule Induction[END_REF] to perform binary classification of direct speech vs narration. The best result were obtained with Random Forest. For this work we kept two algorithms Random Forest, Support Vector Machine, that we have fine-tuned with the grid search algorithms in order to find the optimal hyper-parameters.

We consider the deep neural approach proposed by [START_REF] Tripathi | Sentence Based Discourse Classification for Hindi Story Text-to-Speech (TTS) System[END_REF], which aims to classify hindi sentence storytelling according to three distinct discourses: descriptive, narration, and dialogue. The authors use the Convolutional Neural Network (CNN) for extracting a robust representation of sentences from word embedding, and SVM for classification.

In this work, in addition to the CNN-SVM implementation proposed in [START_REF] Tripathi | Sentence Based Discourse Classification for Hindi Story Text-to-Speech (TTS) System[END_REF], we propose to explore two other neural network-based architectures Recurrent Neural Network (RNN), Recurrent CNN (RCNN).

The experiment carried out on the SynPaFlex-corpus to analyze the accuracy of the discourse prediction models. To evaluate the performance of the implemented models we used a various parameters. Note that the effect of each parameter significantly alters the performance of the model. At the time of training and testing, input to the model is a sentence. Each of the sentences are labeled with one of the discourse mode( DD, ID, IC). In this work we use cross validation technique to avoid overfitting, in particular in case of models with a lot of parameters. This technique consists of splitting the data into K folds, where the blocks are sized equally. Each block turn as the validation set and the rest of (K-1) blocks is used for training the model. The final loss value is the average of K loss result. We fixed the number of blocks to K=6.

We used 2146 sentences in training and the remaining 920 sentences are used for testing.

Results

The performance of the proposed methods is evaluated using confusion matrix, Receiver Operating Characteristic (ROC) curve, precision, recall, f1-measure and accuracy. A Chapter 7

Speaker Prosodic Identity

This chapter is an extended version of the work described in "Introducing Prosodic Speaker Identity for a Better Expressive Speech Synthesis Control" presented at Speech Prosody 2020. This work was performed in collaboration with Sébastien Le Maguer (ADAPT Centre), who was an invaluable partner through-out.

General Context

To have more control over TTS synthesis and to improve expressivity, it is necessary to disentangle prosodic information carried by the speaker's voice identity from the one belonging to linguistic properties. In this work, we propose to analyze how information related to speaker voice identity affects a DNN based multi-speaker speech synthesis model. To do so, we feed the network with a vector encoding speaker information in addition to a set of basic linguistic features. We then compare three main speaker coding configurations: a) simple one-hot vector describing the speaker gender and identifier ; b) an embedding vector extracted from a speaker recognition pre-trained model ; c) a prosodic vector which summarizes information such as melody, intensity, and duration. To measure the impact of the input feature vector, we investigate the representation of the latent space at the output of the first layer of the network. The aim is to have an overview of our data representation and model behavior. Furthermore, we conducted a subjective assessment to validate the result. Results show that the prosodic identity of the speaker is captured by the model and therefore allows the user to control more precisely synthesis.

Introduction

The quality of speech synthesis systems has drastically increased during the last years. Thanks to the deep learning paradigm, it is now possible to generate speech, which sounds where M represents the number of vowels within the breath group and V i F 0median stands for the median F 0 value of the i th vowel within the breath group;

• Melodic pattern: for each vowel contained in a given breath group, the V F 0median has been extracted. The resulting sequence of values has been interpolated using a cubic spline. Then, a set of five equidistant values (at each 20% of the breath group duration starting from 10%) has been selected.

• Energy pattern: the same computation as the previous one is done on V logEnergy .

• Articulation Rate: it is the number of syllable per second computed at the breath group level ignoring pauses;

• Duration of breath group in second;

• Duration of pauses around the breath group in second.

Therefore, we obtain a fully controllable feature vector whose dimensions can be interpreted properly.

Analysis Methodology

The experiments and analyses presented in this work were carried out within the Merlin [START_REF] Wu | Merlin: An Open Source Neural Network Speech Synthesis System[END_REF] framework. We used the default configuration proposed in the toolkit, then we integrated the speaker coding vectors to achieve a multi-speaker TTS model.

Input and Output features

The input feature vector can be viewed as two concatenated vectors corresponding to two parts: a linguistic part and a speaker coding part.The first 319 coefficients correspond to the linguistic description of the utterance. This part is based on the standard feature set for English described in [START_REF] Tokuda | An HMM-based speech synthesis system applied to English[END_REF] that we have adapted for French. The main differences with the English feature set concerns the accentuation. Indeed, as the accentuation information in French is strongly correlated to the Part of Speech (POS) information, we therefore consider that the POS information, already present in the vector, is enough to encode the accentuation information.The coefficients from dimension 320 and

General Conclusion

Summary of the Contribution

In this thesis, we have explored speech's expressivity through a particular speech data type, which is audiobooks. We have proposed a complete process for gathering a sizeable French audiobook corpus and annotating it manually or automatically. MUFASA corpus includes twenty French speakers and contains about 600 hours of good continuous speech quality. We have shown in chapter 3 that this collection of amateur's reading is comparable to professional recording ones in terms of prosodic properties. Even though the voice quality of speech of MUFASA corpus is lower than comparable professional corpora due to the recording conditions, the quantity and diversity of data make it possible to explore new spoken speech horizons. We have compared extracts of MUFASA corpus with other well-known French corpora to measure the similarity. As we expected, MUFASA presents high similarity with the BREF corpus, which also a read speech corpus.

In this work, we have articulated the expressivity carried by audiobooks on three pillars: emotions, discourse, speaker. The emotions intervene at specific moments of speech to animate the discourse and bring depth. To structure and bring coherence in the story, the authors use different modes of discourse. In audiobooks, emotions and discourses depend on the text as much on the speech signal. The speech signal depends on the speaker's properties, which constitutes the third axis.

To explore these three pillars, we have investigated the text properties and the prosodic properties of a set of audiobooks read by nine speakers present in the MUFASA corpus. To study the emotional characteristics of data, we focus our effort on the SynPaFlex Corpus voice, representing the initial version of the MUFASA corpus. This database contains a single female speaker. For conducting the experiments, we used a representative extract proportional to the linguistic distribution of the database.

To study the emotional characteristics of data, we proposed to focus on the SynPaFlex corpus, containing a single female speaker, representing the initial version of the MUFASA corpus. For conducting the experiments, we have asked a speech expert annotator to select representative extract and to annotate the speech signal according to two parameters:

The major findings presented in this thesis are based on an acoustical perspective of speech. This level of representation of prosody is important but not enough to characterize the expressive speech carried by audiobooks. The perceptual representation and linguistic properties of prosody are crucial to have a complete and to validate the results presented in this thesis.

Appendix A

Audiobooks Corpora 1 SynPaFlex Corpus

MUFASA Corpus

MUFASA Parallel Subcorpus

Appendix B

Data visualization and high dimension reduction

1 Principal Component Analysis (PCA)

In this section, a brief procedural description of PCA is provided. More detailed theoretical description is directed to [bishop2006pattern; hastie2009elements; rogers2016first].

Assume that we are given by a m-by-n data matrix X consists of n number of m-dim vectors -→ x i ∈ R.

Step 1: Compute mean and covariance of data matrix

The covariance matrix of X is called S ∈ R m×m and defined by

where x ∈ R m is the mean of each row in X and defined by

Step 2: Singular Vector Decomposition (SVD)

SVD of S is implemented to extract principal components and directions:

where U ∈ R n×n , Σ ∈ R n×m , and V ∈ R m×m . In the implementation, we use the matrix

where a vector represents a principal component direction.

Step 3: Projection

The data matrix X can be projected into a new matrix Y ∈ R k×m by multiplying a matrix

Proper number of principal components k should be selected in prior to perform projection of data matrix.