SVM Support vector machines TDNN Time delay neural network UAR Unweighted average recall

Keywords:

En raison de la rareté des données non étiquetées, la tâche automatisée de détection du bégaiement est limitée dans son utilisation des modèles d'apprentissages profonds de grande taille lorsqu'il s'agit de capturer différentes variabilités. Nous avons introduit la toute première architecture d'apprentissage auto-supervisé «Self-supervised learning (SSL)», dans le domaine de la détection du bégaiement. L'architecture SSL entraîne d'abord un extracteur de caractéristiques pour une tâche de pré-texte en utilisant une grande quantité de données audio non étiquetées et ne contenant pas de bégaiement pour capturer ces différentes variabilités. Puis elle applique l'extracteur de caractéristiques appris à une tâche SD en aval, en utilisant des données audio étiquetées limitées et contenant bégaiement.

All glory to the ALLAH, Lord of all the worlds. Prayers & peace be upon our Prophet Muhammad, his family & all of his companions. A special thanks to my badminton mates Dr. Shoaib Khan, Dr. Mehdi & cricket mates Karamat, Nabeel, Ali, Talha Shahid, Aatiq Ovais, Khalil, Dr. Tareq Kaleemullah, Aadil Zargar, Shabir, Mohsin, Tatha, Yama, Asgar, etc. I would like to express my gratitude to Brij Mohan Lal & Seyed Arash Hossein with whom I explored some new cities, had lengthy, sometimes heated, tea-time technical conversations, & even prepared meals together. I am indebted to Louis Abel (the boss of our C137 gang) with whom I share a special bonding and to Mickaëlla Grondin-Verdon who corrected French summary & abstract of this thesis. Thanks to Hélène Cavallini, Elise Urban, Sylvie Hilbert, Delphine Hubert, Anne-Marie Messaoudi, Souad Boutaguermouchet, ix Aurore Tranchina, & Sabrina Ferry-Tritz for making my stay comfortable at Inria by providing an excellent administrative support.

I also offer my sincere gratitude to my European based Kashmiri brethren who made me feel home like Kashmir even though living thousands of miles away from our native place. Some of us traveled together to several places, had funny & serious political conversations over special Kashmiri meals such as Yakhni, Kahva, Noon chai, etc. These include but not limited to-Aamir Rashid Najar, Umar Bashir, Mohammad Mujtahid, Mir Junaid, , Saquib Lone, Dr. Anayat Bhat, Dr. Mudasir, Sahil Lone, Najmus Saqib, Zulkarnain Tantray, Furqaan Fayaz, Ashaq Hussain, Haroon Rashid, Masheer ul Haq, Moonis Ali, Mehran Jeelani, Aymen Mir, Jalees, Aadil Zargar, Shabir, Bilal, etc.

Penultimately, I would like to thank my Khaleels including Raashid ul Amin, Sheikh Mohsin, & Juanid Ahmad Dar, to my Sadeeqs Muneeb Ahmed, Kaiser Shabir, Imran Rather, Faisal Sofi, Abdul Haseeb Arif Mir, Sohaib Najat, Mustafa Rajab, Haseeb ullah, Zulfikar, Dr. Muzamil Wani, Dr. Arshad Hussain Zagoo, to Cette thèse tente de développer des systèmes de détection du bégaiement «Stuttering detection (SD)», basés sur l'audio qui réussissent à capturer les différentes variabilités des énoncés de bégaiement telles que les styles d'expression, l'âge, les accents, etc., et apprennent des représentations robustes du bégaiement dans le but de fournir une évaluation juste, cohérente et impartiale de la parole bégayée.

Alors que la plupart des systèmes SD existants utilisent plusieurs classificateurs binaires pour chaque type de bégaiement, nous présentons un système multi-classe unifié nommé StutterNet capable de détecter plusieurs types de bégaiement. En abordant le problème du déséquilibre des classes dans le domaine du bégaiement, nous avons étudié l'impact de l'application d'une fonction de perte pondérée et nous avons également présenté une version du modéle StutterNet qui est multi-contextuel et multti-branches pour améliorer la détection des classes minoritaires.

En exploitant les informations sur le locuteur et en supposant que les modèles de bégaiement sont invariants par rapport aux méta-données telles que les informations sur le locuteur, nous présentons un système de détection du bégaiement multi-tâches «Multi-task learning (MTL)» qui apprend des représentations robustes discriminant le bégaiement et les invariantes par rapport au locuteur. List of figures

Motivation

Over the past few decades, there has been a huge growth in interest in creating systems that mimic how humans comprehend and understand speech in terms of its linguistic content, personality, gender, age, health, emotional, and other paralinguistic signs such as stuttering 1 or disfluency [START_REF] Huang | Spoken Language Processing: A Guide to Theory, Algorithm, and System Development[END_REF][START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. The most practical and expressive methods of human communication are speaking and listening. During speech, the brain performs a series of cognitive processes in terms of selecting, activating, and organization of lexical units and sensorimotors that are required by the speech articulators to deliver the intended speech utterance [START_REF] Harrington | Speech Production: Models, Phonetic Processes, and Techniques[END_REF]. Normally, the speech utterance is governed by a number of nonlinear processes where the speakers continuously reshape the direction of their discourse. They could pause to consider what to say next, resume a sentence that was already spoken, add something new, or discard the ongoing utterance [START_REF] Hardcastle | The Handbook of Phonetic Sciences[END_REF]. As a result, spoken utterances are often regulated by cycles of fluent (lexicaly and grammatically accurate) vs disfluent speech (non-lexical items that disrupts the forward flow of speech) [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Notice that normal disfluencies are useful in speech production since they can be considered in time during which the speaker can correct or plan the upcoming discourse, however, in cases, like speech disorder/impairments, the normal disfluencies do not help the speaker to organize his/her discourse. Contrary to persons without any fluency disorder, persons who stutter (PWS) know what they want to pronounce but are momentarily unable to produce it because of the malfunctioning of their central nervous systems in generating the necessary patterns of motor commands for the production of fluent speech [START_REF] Smith | How stuttering develops: The multifactorial dynamic pathways theory[END_REF].

Speech impairments make it difficult for PWS to form correct and necessary speech sounds and impair a person's ability to produce these correct sounds that enable them to communicate with others [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF][START_REF] Ward | Stuttering and Cluttering: Frameworks for Understanding and Treatment[END_REF]. Speech impairment can affect all age groups and usually take the form of stuttering, apraxia (the brain is unable to send an exact message to speech muscles), dysarthria (caused by muscle weakness), cluttering (speech being too jerky or too rapid), and so on [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF][START_REF] Ward | Stuttering and Cluttering: Frameworks for Understanding and Treatment[END_REF][START_REF] Duffy | Motor Speech Disorders e-Book: Substrates, Differential Diagnosis, and Management[END_REF]. Among these speech impairments, stuttering is the most commonly experienced one [START_REF] Minnis | Speech disorders[END_REF].

The malfunctioning in sensorimotor regions which are responsible for speech production results in stuttering in the form of core behaviours: including repetitions, blocks, and prolongations, and its course is directed by lexical and psychological aspects [START_REF] Smith | How stuttering develops: The multifactorial dynamic pathways theory[END_REF][START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. In contrast to other speech impairments, stuttering can be treated if early intervention is provided to assist those PWS in achieving normal fluency [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF].

Driven by smart home devices an estimate of 8 billion digital voice assistants in use is to be expected by 2030 [START_REF] Smith | Digital voice assistants in use to triple to 8 billion by 2023, driven by smart home devices[END_REF]. On one extreme, a speech technology like this enhances the use of gadgets on a daily basis and make their life easier in certain minority groups such as blind and physically challenged who are unable to access textual mode of devices, and on the other end, for the people with speech impairment problems such an influential technology remains inaccessible.

Objectives

In conventional stuttering detection (SD) and therapy sessions, the speech therapists or speech-language pathologists manually analyze the PWS' speech [START_REF] Nöth | Automatic stuttering recognition using hidden Markov models[END_REF]. The speech therapists observe and monitor the speech patterns of PWS to rectify them [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. This convention of SD is very laborious and time-consuming and is also inclined toward the idiosyncratic belief of speech therapists. Moreover, speech therapists might not be available all the time. In addition, the automatic speech recognition systems (ASR) are working fine for normal fluent speech, however, they are unsuccessful in recognizing the stuttered speech [START_REF] Mitra | Analysis and tuning of a voice assistant system for dysfluent speech[END_REF], which makes it impractical for PWS to easily access virtual assistants such as Cortona, Alexa, Siri, etc. As a result, interactive automatic SD systems that provide an impartial objective, and consistent evaluation of stuttering speech are strongly encouraged. We believe that these automatic stuttering detection systems will also provide a fair, consistent, and unbiased assessment of stuttered speech which later on can also be adapted towards voice digital assistants for PWS.

Despite the fact of having numerous potential applications, very little research attention has been given to the domain of SD and measurement which is the main focus of this thesis. The detection and identification of stuttering events can be quite a difficult and complex problem due to several variable factors including language, gender, age, accent, speech rate, etc. Most of the earlier work in SD methods are based either on ASR systems [START_REF] Alharbi | A lightly supervised approach to detect stuttering in children's speech[END_REF][START_REF] Alharbi | Sequence labeling to detect stuttering events in read speech[END_REF] or language models (Zayats et al., 2016a;[START_REF] Chen | Controllable time-delay transformer for real-time punctuation prediction and disfluency detection[END_REF]. These methods are two-stage approaches that first convert the acoustic speech signals into their corresponding spoken textual modality, and then detect stuttering by the application of language models. Even though this ASR-based two-stage approach for identifying stuttering has shown promising results, the dependence on the ASR unit makes it computationally costly and prone to error. Moreover, the adaption towards the ASR task results in the possible loss of stuttering relevant information such as prosodic, emotional, and other paralinguistic information. In summary,

We try to develop audio based stuttering detection systems which successfully capture different variabilities from stuttering audio utterances and helps to distinguish between fluent, and various disfluent speech categories such as blocks, prolongations, repetitions, and interjections. Multi-contextual (MC) StutterNet: In addition, we also found previously, that the context optimized for one category are not good for another stutter class, so exploiting this fact, we propose a MC StutterNet, which is a multi-contextual multi-branched time delay neural network based architecture for SD. The multi-contextual framework is based on the way humans perceive speech. In the cochlea, the input acoustic speech signal is partitioned into several frequency bands so that the information in each band can be filtered independently and thus processed in parallel by the human brain.

Contribution summary

Multi-task and adversarial learning in stuttering detection: From recent years, the field of deep learning-based stuttering detection is evolving continuously. Yet all current systems focus only on the stuttering learning method without considering any auxiliary tasks. Unfortunately, this is not the way, how we humans perceive speech; instead, we simultaneously decode the main task with various meta contents like speaker traits, linguistic content, emotion, etc., in a multi-tasking fashion. Motivated by this, we exploited speaker characteristics in stuttering detection. The third contribution of this thesis is to learn and exploit these speaker traits in combination with the primary stuttering detection task in a multi-tasking learning paradigm. As the stuttering speech detection task should not be dependent on a specific set of speakers, an alternative approach, is to assume that a robust acoustic stuttering representation needs to be invariant to secondary tasks, particularly speaker recognition. We also contributed to this by proposing an adversarial multi-task (MTL) learning method that learns latent representations that are speaker invariant but at the same time are stutter discriminative.

Introducing self-supervised learning to stuttering detection: Due to a paucity of unlabeled data, the automated stuttering detection task is limited in its use of large deep models. As stated in Appendix A, several stuttering datasets, such as SEP-28k [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], UCLASS [START_REF] Howell | Automatic recognition of repetitions and prolongations in stuttered speech[END_REF], LibriStutter [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF], and FluencyBank [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], have been curated throughout the time to examine various SD models, however, the improvement is bounded most likely because of the small sized and unlabeled stuttering datasets, which are unable to encapsulate several variable factors such as linguistic content, speaking style, gender, age, accent, speech tempo, recording conditions, and others. Our contribution to this challenge is that we introduced the first-ever self-supervised learning framework to stuttering detection domain. The self-supervised learning framework first trains a feature extractor for a pre-text task using a large quantity of unlabeled non-stuttering audio data to capture different speaking styles, accents, age group, linguistic content, paralinguistic content, prosody, environmental conditions, etc, and then applies the learned feature extractor to a downstream stuttering detection task using limited labeled stuttering audio data.

Publications

The below mentioned papers serve as a foundation of this thesis, and are presented in order in which they occur in this thesis.

Thesis outline

This PhD thesis is situated at the intersection of deep learning applied to stuttering problem in the acoustic modality. The outline of this thesis is as follows: the first chapter discusses the speech disorder problem in general with a focus on stuttering from multiple views. The third chapter reviews the stat-of-the-art work in stuttering detection domain.

The next three chapters details about the StutterNet and its variants. Finally, the chapter seven introduces the SSL framework to stuttering detection with an extensive evaluation on SEP-28k and KSoF dataset. The final thoughts towards the future research developments are presented in the last chapter. A more detailed summary of the chapters is given below and are based on the peer-reviewed and under revision papers highlighted in the box after the outline of each chapter:

Chapter 2: In this chapter, we discuss the general speech disorder problem with a focus on stuttering. We first begin by discussing the stuttering problem and its categories, and its impact on the real life of PWS. Then, we provide a discussion on the characteristics of stuttered speech from neurogenic, acoustic and phonetic point of views. Lastly, we give a detailed motivation about the need of automatic stuttering detection systems after pointing out the limitations and drawbacks of the conventional stuttering assessment.

Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "Machine learning for stuttering identification: Review, challenges and future directions". Journal of Neurocomputing, 514 (2022), pp 385-402 (peer-reviewed).

Chapter 3 In this chapter, we walk through the foundational ideas and fundamental concepts of the fields including datasets, speech processing and deep learning, which are essential to our work. Firstly, we present a brief discussion on speech production followed by acoustic features employed mostly in stuttering detection domain. Next, we present some challenges, the stuttering detection domain is currently facing, and, in the subsequent chapters, we try to address some of these challenges. Following the discussion on feature extraction, we provide a discussion on various deep learning methods that we adopted towards stuttering detection. Lastly, we review the state-of-the-art SD systems with their possible limitations.

Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "Machine learning for stuttering identification: Review, challenges and future directions". Journal of Neurocomputing, 514 (2022), pp 385-402 (peer-reviewed).

Chapter 4: This chapter gives a detailed discussion about our proposed StutterNet model, a time delay neural network single multi-class classifier capable of detecting stuttering and its types efficiently. Firstly, we discuss its architecture, followed by the dataset details used to evaluate the effectiveness of the proposed method. Finally, we comment on the results achieved by the proposed StutterNet.

Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "StutterNet: Stuttering detection using time delay neural network". In Proc. 29th European Signal Processing Conference (EUSIPCO), (2021), pp 426-430 (peer-reviewed).

Chapter 5: This chapter addresses the class-imbalance problem in stuttering detection domain via weighted loss function and multi-branch training schemes. In addition, this also investigates the effectiveness of data augmentation when applied to stuttering detection problem. Moreover, we found previously in Chapter 4 that the contexts optimized for one stutter class are not good for other class. Driven by this fact, this chapter also discusses the proposed MC StutterNet, which exploits variable contexts from the stuttered speech utterance.

The recently released first-ever public labeled stutter dataset is used to assess the proposed framework's overall performance.

Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "Advancing stuttering detection via data dugmentation class-balanced loss and multi-contextual deep learning". Journal of IEEE Biomedical Health Informatics, (2022) (peer-reviewed).

Chapter 6: In this chapter, we introduce and discuss an adversarial multi-task learning framework applied to stuttering detection problems to learn robust latent stutter representations that are stutter discriminative while being speaker-invariant. The overall methodology of the proposed framework is provided and evaluated on the newly released first-ever publicly labeled stutter dataset.

Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "Robust stuttering detection via multi-task and adversarial learning". In Proc. 3oth European Signal Processing Conference (EUSIPCO), (2022) (peer-reviewed).

Chapter 7: Finally, the last chapter discusses the employment of first-ever self-supervised learning framework to stuttering detection problem. We first, describe the motivation of applying contextual embeddings to SD, learned in an SSL setup on some pre-text auxiliary task. We mainly focus on speaker and contextual embeddings where the SSL models are pre-trained on massive datasets with the hypothesis in capturing various speaking styles and prosodic information which are extremely important to stuttering detection. We also discuss which layers of SSL model captures more stutter-related information. In addition, this chapter also discusses the evaluation of pre-trained SSL models in cross-language stuttering detection platform. To this, we evaluated the English language pre-trained SSL model on the German language stutter dataset provided to us in the ComParE challenge organised during ACM workshop [START_REF] Schuller | The ACM multimedia 2022 computational paralinguistics challenge: Vocalisations, stuttering, activity, & mosquitos[END_REF]. Lastly, we discuss the results from the SSL based stuttering detection, and showed empirically that the contextual embeddings achieve a considerable gain and surpasses in all the disfluent classes over the state-of-the-art and proposed methods discussed in previous chapters.

Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "Introducing ECAPA-TDNN and Wav2Vec2.0 embeddings to stuttering detection". Under review in International Journal of Speech Technology, (2022) (under review). Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "End-to-end and self-supervised learning for ComParE 2022 stuttering sub-challenge". In Proc. 30th ACM International Conference on Multimedia, (2022), pp 7104-7108 (peer-reviewed).

Chapter 8: Finally in this chapter, we conclude our contributions and provide some of our reflections on the future directions that the studies conducted for this thesis has initiated.

Stuttering problem

"The earlier that the intervention starts for your child, the quicker that the disorder can be addressed and corrected." Unknown

Speech disorders

Voice disorders1 known as speech disorders or speech impairments occur when a person struggles to produce and articulate the typical spoken sounds needed to interact with others [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF][START_REF] Duffy | Motor Speech Disorders e-Book: Substrates, Differential Diagnosis, and Management[END_REF]. These voice disorders can take the form of dysarthria, apraxia, stuttering, cluttering, lisping, and so on [START_REF] Duffy | Motor Speech Disorders e-Book: Substrates, Differential Diagnosis, and Management[END_REF][START_REF] Ratner | Fluency bank: A new resource for fluency research and practice[END_REF][START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF][START_REF] Ward | Stuttering and Cluttering: Frameworks for Understanding and Treatment[END_REF][START_REF] Kehoe | Speech Language Pathology-Stuttering[END_REF]. Dysarthria is defined as a speech disorder caused by the muscle weakness (including face, lips, tongue, and throat) controlled by the nervous system. The patients with dysarthria produce slurred or mumbled sounds with aberrant speech patterns, such as flat intonation or very low or fast speech rate, which makes their speech very difficult to comprehend [START_REF] Duffy | Motor Speech Disorders e-Book: Substrates, Differential Diagnosis, and Management[END_REF]. Cluttering is characterized by a patient's speech being too jerky, too rapid, or both. Persons with cluttering usually exclude/collapse most of the syllables, or aberrant rhythms or syllable stresses, and also contain excessive amounts of interjections such as so, hmm, like, umm, etc [START_REF] Ward | Stuttering and Cluttering: Frameworks for Understanding and Treatment[END_REF]. Apraxia is defined as a speech disorder when the neural path between the nervous system and the muscles responsible for speech production is obscured or lost. The persons with apraxia knows what they want to speak, but can not speak because the brain is unable to send an exact message to the speech muscles which can articulate the intended sounds, despite the fact speech muscle movements are working fine [START_REF] Duffy | Motor Speech Disorders e-Book: Substrates, Differential Diagnosis, and Management[END_REF][START_REF] Kehoe | Speech Language Pathology-Stuttering[END_REF]. Lisping speech disorder is defined as the incapability of producing sibilant consonants (z or s) correctly. The sibilant sounds are usually substituted by th sounds. For example, the persons with lisping speech disorder would pronounce the word lisp as lithp [START_REF] Duffy | Motor Speech Disorders e-Book: Substrates, Differential Diagnosis, and Management[END_REF]. Stuttering speech impairment is different than the other speech disorders because it can be cured if early intervention is being made to help the PWS develop normal fluency [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Of all these speech impairments, stuttering is the most common and predominant one 2 .

Stuttering speech disorder

Stuttering is a neurodevelopmental speech disorder, caused by the failure of speech sensorimotors. Stuttering -also known by the name stammering/disfluency -commences when neural connections supporting language, speech, and emotional functions are quickly changing [START_REF] Duffy | Motor Speech Disorders e-Book: Substrates, Differential Diagnosis, and Management[END_REF][START_REF] Ward | Stuttering and Cluttering: Frameworks for Understanding and Treatment[END_REF][START_REF] Smith | How stuttering develops: The multifactorial dynamic pathways theory[END_REF], and is characterized by core behaviors that usually take the form of involuntary stoppages, repetition, and prolongation of sounds, syllables, words or phrases. Stuttering can be described as an abnormally and persistent duration of stoppages in the normal forward flow of speech [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. These speech abnormalities are generally accompanied by unusual behaviors like head nodding, lip tremors, quick eye blinks, unusual lip shapes, etc [START_REF] Riva-Posse | Phenomenology of abnormal movements in stuttering[END_REF]. Fluency can be defined as the capacity to produce speech without any effort, at a normal rate [START_REF] Starkweather | Fluency and Stuttering[END_REF]. A fluent speech requires linguistic knowledge of the spoken language and a mastery of the message content. Concerning physiological aspects, a precise respiratory, laryngeal and supraglottic control movement is necessary to maintain fluency [START_REF] Adams | A physiologic and aerodynamic interpretation of fluent and stuttered speech[END_REF].

When all these conditions are not met, speech disorder (stuttering) can emerge. They can take the form of silent or filled pauses, repetitions, interjections, revisions (content change or grammatical structure or pronunciation change), and incomplete phrases [START_REF] Roberts | Disfluencies in non-stuttering adults across sample lengths and topics[END_REF]. Generally, normal speech is made up of mostly the fluent and some disfluent parts.

Notice that normal disfluencies are useful in speech production since they can be considered in time during which the speaker can correct or plan the upcoming discourse. In some cases, like stuttering, disfluencies do not help the speaker to organize his/her discourse. Indeed, contrary to persons without any fluency disorder, PWS know what they want to pronounce but are momentarily unable to produce it [START_REF] Organization | Manual of the international statistical classification of diseases, injuries, and causes of death: based on the recommendations of the ninth revision conference[END_REF][START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF].

Stuttering categories

Stuttering can broadly be classified into two types (NIDCD, 2015):

• Developmental Stuttering: This stuttering is the most common one and it usually occurs in the learning phase of the language, i.e. between two and seven years of language development. Recent researches conclude that developmental stuttering is multifactorial trouble including neurological and genetic aspects [START_REF] Etchell | A systematic literature review of neuroimaging research on developmental stuttering between 1995 and 2016[END_REF][START_REF] Drayna | Genetic approaches to understanding the causes of stuttering[END_REF]. Indeed, functional magnetic resonance imaging studies show anomalies in neural activity before the speech, i.e. during the planning stages of speech production [START_REF] Vanhoutte | When will a stuttering moment occur? the determining role of speech motor preparation[END_REF]. Furthermore, an atypical activation in the left inferior frontal gyrus and right auditory regions has been highlighted [START_REF] Neef | Speech dynamics are coded in the left motor cortex in fluent speakers but not in adults who stutter[END_REF][START_REF] Belyk | Stuttering as a trait or state-an ale metaanalysis of neuroimaging studies[END_REF]. Concerning the genetic aspects, [START_REF] Riaz | Genomewide significant linkage to stuttering on chromosome 12[END_REF] observe an unusual allele on chromosom 12 by PWS. [START_REF] Drayna | Genetic approaches to understanding the causes of stuttering[END_REF] identify 87 genes that could be involved in stuttering, including one called GNPTAB, which was significantly present in a lot of PWS.

• Neurogenic Stuttering: This stuttering can occur after the head trauma, brain stroke, or any kind of brain injury. This results in disfluent speech because of the incoordination of the different regions of the brain which are involved in speaking3 . Even if neurogenic stuttering is rare, it can be observed in children and adults regardless of their age.

Impact of stuttering

Globally, stuttering concerns 1% of the world's population and its incidence rate is between 5% and 17% [START_REF] Yairi | Epidemiology of stuttering: 21st century advances[END_REF]. The difference between the prevalence and incidence rates can be explained by the fact that developmental stuttering disappears in 80% of the cases before adulthood either without any intervention or thanks to the speech therapy. Thus, about 70 million people suffer from this disorder which affects four times males than females [START_REF] Yairi | Epidemiology of stuttering: 21st century advances[END_REF]. As considered by the non-stuttering persons, the disfluency affects the flow of speech only, however for PWS, it is greater than that.

Several studies show that PWS are ignored, teased, and/or bullied by normo-fluent people [START_REF] Iverach | Prevalence of anxiety disorders among children who stutter[END_REF]. The PWS is usually rated less popular than their non-stuttering peers and less likely to be considered leaders [START_REF] Iverach | Prevalence of anxiety disorders among children who stutter[END_REF]. According to the national stuttering association [START_REF] Nsa | The experience of people who stutter: A survey by the national stuttering association[END_REF], 40% of the PWS have been repudiated school opportunities, promotions or job offers and it affects relationships. The data should be assessed in close conjunction with the fact that 85% of businessmen consider stuttering as a negative element during a job interview and prefer offering work that does not require a customer contact [START_REF] Klein | The impact of stuttering on employment opportunities and job performance[END_REF]. All these elements explain that PWS develop social anxieties and negative feelings (fear, shame, etc.,) when they have to speak in public [START_REF] Blood | Long-term consequences of childhood bullying in adults who stutter: Social anxiety, fear of negative evaluation, self-esteem, and satisfaction with life[END_REF]).

Stuttering appears to be complex and several factors that lead to stuttering include stress, delayed childhood development, and speech motor control abnormalities, as there is a strong correlation between stress, anxiety, and stuttering [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Indeed, disfluencies are more frequent in stress or anxiety conditions, in dual tasks including speech and another cognitive charge, and when they speak fast. Conversely, PWS produce fewer disfluencies when they sing in unison or speak with an altered auditory feedback [START_REF] Antipova | Effects of altered auditory feedback (AAF) on stuttering frequency during monologue speech production[END_REF]. In a recent study, [START_REF] Smith | How stuttering develops: The multifactorial dynamic pathways theory[END_REF] postulated the multifactorial dynamic pathways theory, where they asserted that stuttering occurs because of the failure of the central nervous system in generating the necessary patterns of motor commands for fluent speech. Thus, stuttering shows impairment in sensorimotor processes that are responsible for speech production, and its orientation throughout the life of PWS is strongly affected by the linguistic and emotional aspects.

Characteristics of stuttered speech

In this section, we discuss the stuttering problem from the neurogenic and acoustic point of view. This section also discusses the various acoustics properties of stuttered speech and describes how the stuttered speech impacts the various characteristics like formant transitions, pitch, and Voice onset time (VOT).

Neurogenic view

Most brain scan studies show that, during fluent speech and silent rest, there is no difference in the cerebral activity between persons who stutter and normal fluent speakers, however, during stuttering, there is a dramatic change in the cerebral activity [START_REF] Ingham | Functional-lesion investigation of developmental stuttering with positron emission tomography[END_REF]. The right-hemisphere of the brain areas which are normally not active during normal speech become active, and the left-hemisphere areas, which are active during normal speech become less active [START_REF] Ingham | Functional-lesion investigation of developmental stuttering with positron emission tomography[END_REF][START_REF] Kehoe | Speech Language Pathology-Stuttering[END_REF]. It has also been found that there is an under-activity in the central auditory processing area of the brain. [START_REF] Kehoe | Speech Language Pathology-Stuttering[END_REF] suggests that adult persons who stutter have an inability to integrate somatic and auditory processing. A brain study by [START_REF] Foundas | Anomalous anatomy in adults with persistent developmental stuttering: A volumetric MRI study of cortical speech and language areas[END_REF] found that PWS has rightward asymmetry in planum temporale, i.e, their right planum temporale is larger than their left planum temporale, on the contrary, normal people's planum temporale is larger in the left side of their brains. Studies based on motor data have been carried out about stuttering. The PWS also show over-activity in the speech motor control area, in particular in the left caudate nucleus area [START_REF] Smith | How stuttering develops: The multifactorial dynamic pathways theory[END_REF][START_REF] Kehoe | Speech Language Pathology-Stuttering[END_REF]. [START_REF] Conture | Laryngeal behavior during stuttering[END_REF][START_REF] Conture | Laryngeal behavior during stuttering: A further study[END_REF] observe inappropriate vocal folds abductions and adductions which lead to anarchic openings and closure of the glottis. Concerning the supraglottic level, [START_REF] Wingate | Stuttering as phonetic transition defect[END_REF] hypothesizes that stuttering is not a sound production problem but a coarticulation one. He theorizes that disfluencies occur during a fault line, which corresponds to the interval when muscular activity due to a sound that has been produced is going off and muscular movements for the following sound are going on. However in a recent study, [START_REF] Didirková | A two-case study of coarticulation in stuttered speech. an articulatory approach[END_REF] show, thanks to Electromagnetic articulograph (EMA) data, that stuttering is not a coarticulation trouble. They found correct coarticulatory patterns in the fluent and stuttered utterances. Furthermore, another study based on articulatory observation, notes that stuttered disfluencies and non-pathological disfluencies do have common characteristics. However, stuttered disfluencies and nonpathological disfluencies produced by PWS present some particularities, mainly in terms of retention and anticipation, and the presence of spasmodic movements [START_REF] Didirkova | An articulatory study of differences and similarities between stuttered disfluencies and non-pathological disfluencies[END_REF]. PWS tend to develop strategies allowing them to avoid sounds or words which can result in disfluency; such strategies consist in using paraphrases or synonyms instead of the problematic segment [START_REF] Ward | Stuttering and Cluttering: Frameworks for Understanding and Treatment[END_REF].

Acoustic and phonetic view

Concerning stuttering-like disfluencies such as blocks, prolongations, and repetitions as shown in Figure 3.2, some works try to link the locus of disfluencies and phonetic proprieties. [START_REF] Jayaram | Phonetic influences on stuttering in monolingual and bilingual stutterers[END_REF]; [START_REF] Blomgren | Do speech sound characteristics really influence stuttering frequency[END_REF]; [START_REF] Didirkova | Parole, langues et disfluences: Une étude linguistique et phonétique du bégaiement[END_REF] indicate that unvoiced consonants are more disfluent than their voiced counterparts. Furthermore, [START_REF] Blomgren | Do speech sound characteristics really influence stuttering frequency[END_REF] notices that disfluencies are more frequent at the beginning of an utterance or just after a silent pause. Moreover, [START_REF] Didirkova | Parole, langues et disfluences: Une étude linguistique et phonétique du bégaiement[END_REF] observes an important inter-individual variability concerning sounds and/or phonetic features which are the most disfluent.

Acoustic analysis has been carried out about stuttering, including speech rate, stopgap duration, vowel(V)-consonant(C) transition duration, fricative duration, VOT, CV transition duration, vowel duration, formants, glottis constriction, a sharp increase in articulatory power and closure length elongation before the speech segment is released [START_REF] Zebrowski | Acoustic analysis of young stutterers' fluency: Preliminary observations[END_REF]. [START_REF] Dehqan | Formant transitions in the fluent speech of Farsi-speaking people who stutter[END_REF] studied the second formant (F2) transitions of fluent segments of Persian speaking PWS. They concluded that the F2 frequency extent transitions are greater in stuttering speakers than in normal fluent ones. The PWS takes longer to reach vowel steady state, but the overall F2 formant slopes are similar for both stuttering speakers and normal ones [START_REF] Dehqan | Formant transitions in the fluent speech of Farsi-speaking people who stutter[END_REF]. The PWS generally exhibit slower speaking rates when compared to normal speakers. Several other studies have investigated the CV formant transitions in stuttered speech. [START_REF] Yaruss | F2 transitions during sound/syllable repetitions of children who stutter and predictions of stuttering chronicity[END_REF] examined the F2 transitions of children who stutter on syllable repetitions and found no aberrant F2 transitions. However, [START_REF] Robb | Formant frequency fluctuation in stuttering and nonstuttering adults[END_REF] analyzed the fluent speech segments of PWS and showed that F2 fluctuations are longer for voiced and voiceless stops than normal speakers.

In a different study by [START_REF] Chang | Coarticulation and formant transition rate in young children who stutter[END_REF], where 3-5 year aged children were analyzed in the picture-naming task. The results showed that disfluent children produced smaller fluctuations of F2 transitions between the alveolar and bilabial place of articulations than did fluent children, and the overall degree of CV coarticulation is no different among stuttering and control groups. [START_REF] Subramanian | Second formant transitions in fluent speech of persistent and recovered preschool children who stutter[END_REF] analyzed the F2 frequency fluctuations of voiceless stops and revealed that near the onsets of CV, the stuttering children exhibited smaller F2 changes than the normal speakers. [START_REF] Blomgren | A note on vowel centralization in stuttering and nonstuttering individuals[END_REF] found that PWS and normal speaker in /CVt/ token exhibit no differences in the F1 (average). The stutters show significantly lower F2 in /Cit/ tokens than the control groups. The formant spacing for /i/ is significantly lower in PWS than fluent persons [START_REF] Blomgren | A note on vowel centralization in stuttering and nonstuttering individuals[END_REF]. [START_REF] Hirsch | Formant structures of vowels produced by stutterers in normal and fast speech rates[END_REF] conducted a study by analyzing the first two formants (vowel space) in CV sequences between the stuttered and normal groups. At a normal speaking rate, the stuttering group shows a reduction in the vowel space, contrary to the fast speaking rate, where, the latter shows no noticeable deviation.

VOT is the duration of time between the release of a stop consonant and the beginning of vocal fold vibrations [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. [START_REF] Healey | Acoustic measures of stutterers' and nonstutterers' fluency in two speech contexts[END_REF] showed that for voiceless stops, chronic stuttering exhibits longer VOT when compared with normal fluent persons. They showed that consonant and vowel duration were longer only in real-world phrases like take the shape in contrast with nonsense phrases like ipi saw ipi [START_REF] Healey | Acoustic measures of stutterers' and nonstutterers' fluency in two speech contexts[END_REF]. [START_REF] Hillman | Voice onset time for voiceless stop consonants in the fluent reading of stutterers and nonstutterers[END_REF] also found that the PWS reveals longer VOT for voiceless stops than for fluent persons. [START_REF] Adams | Voice onsets and segment durations of normal speakers and beginning stutterers[END_REF] found that not only voiceless stops exhibit longer VOT in PWS, but also, voiced stops display longer VOT than non-stuttering persons. No significant VOT differences have been found in control and PWS groups [START_REF] Watson | A comparison of lrt and vot values between stutterers and nonstutterers[END_REF]. In another study by [START_REF] Jäncke | Variability and duration of voice onset time and phonation in stuttering and nonstuttering adults[END_REF], the PWS show strong variability in repeated production of VOT for voiceless stops, however, there is no significant difference between the two groups. In a study carried out by [START_REF] De Nil | Voice onset times of stuttering and nonstuttering children: The influence of externally and linguistically imposed time pressure[END_REF], it shows that the stuttering children exhibit more variability in VOT than their counterparts. Celeste and de Oliveira Martins-Reis (2015) also found that the stuttering group shows higher VOT for voiceless stops. [START_REF] Brosch | Prognostic indicators for stuttering: The value of computer-based speech analysis[END_REF] examined the VOT in stuttering children.

They found that the children with severe stuttering have higher values of VOT. [START_REF] Borden | Onset of voicing in stuttered and fluent utterances[END_REF] examined the VOT of fluent utterances from PWS. Their study showed that the fluent utterances of PWS exhibit no statistical differences in VOT and are within the limits of normal groups. [START_REF] Fosnot | Prosodic characteristics in children with stuttering or autism during reading and imitation[END_REF] examined the prosodic characteristics of PWS and fluent children. They found that the variability in pitch is greater in the stuttered group, but slightly differs from the normal group. In another study, it has been shown that the normal group and PWS show the same patterns in f0 deviation [START_REF] Ramig | Vocal changes in stutterers and nonstutterers during high-and low-pitched speech[END_REF]. The stuttering occurs less significantly in low-pitched conditions as compared to high-pitched condition [START_REF] Ramig | Vocal changes in stutterers and nonstutterers during high-and low-pitched speech[END_REF]

Conventional stuttering assessment

In conventional stuttering assessment, the Speech language pathologists (SLP) or Speech therapists (ST) manually analyze either the PWS' speech or their recordings [START_REF] Nöth | Automatic stuttering recognition using hidden Markov models[END_REF][START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. The stuttering severity is usually measured by taking the ratio of disfluent words/duration to the total words/duration [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. The most conventional speech therapy sessions involve helping the PWS observe and monitor their speech patterns to rectify them [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. The speech therapeutic success rate recoveries have been reported to be 60-80% when dealt within the early stage [START_REF] Saltuklaroglu | How effective is therapy for childhood stuttering? dissecting and reinterpreting the evidence in light of spontaneous recovery rates[END_REF]. However, this convention of detecting stuttering severity and its improvement due to therapeutic sessions is very demanding and time-consuming and is also biased and prejudiced towards the subjective belief of SLPs. Due to the nature of stuttering, its therapeutic sessions are very intense course, that usually, extend to several months (several years in some cases), which necessitates PWS to see the SLP regularly [START_REF] Roberts | Using emprical benchmarks to assess the effects of a parentimplemented language intervention for children with language impairments[END_REF].

Usually, the speech therapy sessions are private and are very expensive, thus making it unaffordable to some PWS. Thus, it is important to develop interactive automatic SD systems.

Moreover, the ASR are working well for the fluent speech, but they fail to recognize the stuttered speech. So, it would not be feasible for a PWS to easily access virtual assistant tools like Alexa, Siri, etc 4 . The SD may help in adapting the virtual assistant tools for disfluent persons. Therefore, developing interactive SD systems are the need for an hour which provides an objective and consistent measurement of the stuttered speech. Consequently, with the recent developments in natural language processing, machine learning, and deep learning, it became a reality to develop smart and interactive SD and therapy tools. The stuttering detection systems can later on be used in an ASR pipeline to first detect the stuttered samples, and then either correct these stutter speech frames or removing them completely from an utterance.

Automatic stuttering detection

Stuttering speech disorder reflected by its core behaviours: prolongations, repetitions, and blocks impact the acoustic properties of speech which can help to discriminate from 4 For people who stutter, the convenience of voice assistant technology remains out of reach fluent voice as shown in Figure 3.2. Most of the computer based SD methods are based either on ASR systems [START_REF] Alharbi | A lightly supervised approach to detect stuttering in children's speech[END_REF][START_REF] Alharbi | Sequence labeling to detect stuttering events in read speech[END_REF] or language models (Zayats et al., 2016a;[START_REF] Chen | Controllable time-delay transformer for real-time punctuation prediction and disfluency detection[END_REF]. These methods are two stage approaches that first convert the acoustic speech signals into their corresponding spoken textual modality, and then detects stuttering by the application of language models. Even though this ASR based two stage approach of identifying stuttering has shown promising results, however, the dependence on ASR unit makes it computationally costly and prone to error. Moreover, the adaptation towards ASR task results in possible loss of stuttering relevant information such as prosodic and emotional content. In recent decades, the applications of deep learning have grown tremendously in speech recognition [START_REF] Nassif | Speech recognition using deep neural networks: A systematic review[END_REF], speaker recognition (Latif et al., 2020a), speech synthesis (Ning et al., 2019a), emotion detection [START_REF] Akçay | Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers[END_REF], voice conversion [START_REF] Sisman | An overview of voice conversion and its challenges: From statistical modeling to deep learning[END_REF], voice disorder detection [START_REF] Verde | Voice disorder identification by using machine learning techniques[END_REF] including Parkinson's disease detection [START_REF] Almeida | Detecting Parkinson's disease with sustained phonation and speech signals using machine learning techniques[END_REF] and dysarthric speech detection [START_REF] Narendra | Dysarthric speech classification from coded telephone speech using glottal features[END_REF][START_REF] Kodrasi | Temporal envelope and fine structure cues for Dysarthric speech detection using CNNs[END_REF]. However, the application of deep learning in SD is limited.

Due to the presence of acoustic cues in the stuttered embedded speech as shown in Figure 3.2, the deep learning models can be used to exploit these acoustic cues in the detection and identification of stuttering events. Most of the SD existing methods employ spectral features such as MFCCs and Linear prediction cepstral coefficients (LPCCs) or their variants that capture these formant-related information. Other spectral features such as pitch, zero-crossing rate, shimmer, and spectral spread are also used. Finally, those features are modeled with statistical modeling methods such as hidden Markov model, support vector machine, Gaussian mixture model, etc.

Summary

In this chapter, we discussed the underlying principles of stuttering, which makes it easier to grasp the subjects covered in the next chapters. First, speech impairments in general were discussed. Of these speech impairments, stuttering is the most predominant one. Next, we explained the characteristics of stuttering speech disorder from neurogenic, phonetic and acoustic point of view and its impact on PWS and SLP. Finally, we presented the conventional way of stuttering assessment and its limitations. This thesis provides a deep learning framework for SD, where we first describe the state-of-the-art deep learning models followed by our contributions to automatic SD.

Background and state of the art

"Stuttering is no simple speech impediment.

It is a complicated disorder which has both physical and emotional aspects."

Malcolm Fraser

There is a multitude of research content available in each of the domains pertinent to this thesis. It makes use of prior knowledge from speech processing, stuttering, and deep learning. In this chapter, we discuss specific background details of the state-of-the-art features and techniques proposed in this thesis, and we draw on existing literature to paint a complete picture of the problem statement and how the proposed solutions are considered by the speech disorder community in particular stuttering community 1 .

Introduction

Research interest towards stuttered speech is becoming more and more prominent.

Building a trustworthy stuttered speech classification system is still difficult owing to its complicated nature. Stuttering detection (SD) is a complex interdisciplinary problem that involves speech processing, signal processing, neuroscience, psychology, pathology, and machine learning. The recent advancements in the machine and deep learning have significantly transformed the speech domain. However, in SD, it has not been explored eminently. A myriad number of research works for SD (in terms of acoustic feature extraction and classification methods) have been carried out with a focus on developing automatic tools for its detection and identification. Most of the earlier work detects and identifies stuttering either by language models [START_REF] Zayats | Disfluency detection using a bidirectional LSTM[END_REF][START_REF] Chen | Controllable time-delay transformer for real-time punctuation prediction and disfluency detection[END_REF] or by ASR systems [START_REF] Alharbi | A lightly supervised approach to detect stuttering in children's speech[END_REF][START_REF] Alharbi | Sequence labeling to detect stuttering events in read speech[END_REF], which first converts the audio signals into their corresponding textual form, and then by the application of language models, detects or identifies stuttering. However, in this chapter we focus on SD only from the acoustic modality point of view, and discuss the underlying speech processing principles, and state-of-the art SD classification methods.

Speech processing principles and tools

First of all let us go through some of the underlying principles and tools of speech processing that underpin the proposed methods for SD. We commence with a signal, a fundamental representation of speech, and how it is transformed to meaningful relevant feature space based on the phonetic and physiological factors.

Fundamentals of speech processing

Vocal Tract: The vocal tract as shown in Figure 3.1 is the physiological system that generates speech extended from the lungs to the lips and nose. The larynx divides the vocal tract into two physical regions: the lower sublaryngeal area and the upper supralaryngeal region [START_REF] Huang | Spoken Language Processing: A Guide to Theory, Algorithm, and System Development[END_REF][START_REF] Kehoe | Speech Language Pathology-Stuttering[END_REF][START_REF] Hardcastle | The Handbook of Phonetic Sciences[END_REF]. The diaphragm, lungs, and trachea (also called the windpipe) make up the vocal tract's sublaryngeal region. The air flows out of the lungs and encounters the vocal folds, which are a pair of flap-like structures in the larynx. When the vocal folds are kept at an intermediate tension, neither too close or too far apart, the movement of the air causes ripples along their length and makes them to vibrate, resulting in voiced speech. On the contrast, when the vocal folds are maintained sufficiently apart so that air can freely flow through them, they do not vibrate, resulting in voiceless speech. This is visible in the speech signal as aperiodic portions that appear to be random noise and are referred to as unvoiced areas. The supralaryngeal area, which includes the mouth and nasal cavities, is crucial in defining the exact nature and quality of the sounds emitted. Articulators comprise the numerous portions of the supralaryngeal area that contribute to the articulation of various vowels and consonants. The lips, teeth, tongue, alveolar ridge, hard palate, and velum are the principal Speech is a temporal activity, and listeners are sensitive to changes in an utterance's temporal flow. An unlimited number of sounds called phones can be produced by the vocal tract which are independent of the language acquisition [START_REF] Hardcastle | The Handbook of Phonetic Sciences[END_REF]. A given language may only have a limited number of perceptibly unique and distinct sound units called phonemes that can be utilized to build meaningful constructs which facilitate communication. These phonemes are studied under the branch of linguistics known as phonology.

The continuous signal produced by the vocal tract is classified into distinct phoneme classes based on acoustic, articulatory, and perceptual aspects. Most languages have two types of phonemes: vowels, which are voiced sounds formed without restricting the airflow by articulators, and consonants, which are formed by restricting the airflow flowing through the vocal tract [START_REF] Hardcastle | The Handbook of Phonetic Sciences[END_REF]. Some studies try to connect the location of disfluencies with phonetic proprieties as discussed in Section 2.3.2. According to [START_REF] Jayaram | Phonetic influences on stuttering in monolingual and bilingual stutterers[END_REF]; [START_REF] Didirkova | Parole, langues et disfluences: Une étude linguistique et phonétique du bégaiement[END_REF][START_REF] Blomgren | Do speech sound characteristics really influence stuttering frequency[END_REF] unvoiced consonants are more disfluent than their voiced consonants. [START_REF] Wingate | Stuttering as phonetic transition defect[END_REF] hypothesizes that stuttering is a coarticulation2 difficulty rather than a sound production problem. He theorizes that disfluencies develop when muscle activity for a previously generated sound is turned off and muscular movements for the next sound are turned on. However, [START_REF] Didirková | A two-case study of coarticulation in stuttered speech. an articulatory approach[END_REF] demonstrate in a recent study, using EMA data, that stuttering is not a coarticulation problem. They discovered accurate coarticulatory patterns in both fluent and stuttering speech.

Speech representation in time domain:

Speech is a pressure wave that propagates across the medium of air. It can be measured over time by measuring the fluctuations in pressure of air molecules at a single location in space [START_REF] Hardcastle | The Handbook of Phonetic Sciences[END_REF]. Figure 3.2 shows the time-domain signal, spectrogram, and MFCCs of a male speakers saying the word "Communication Disorder". An amplitude with finite precision of bit depth, and the sampling rate/frequency F s are required to digitally represent an acoustic speech signal. Mathematically, the digital audio speech signal can be represented as y[n], where n = 0, 1, ..., N s -1 denotes a sample of instantaneous amplitude.

Speech representation in spectral domain:

The first step that takes us out of the time domain to its spectral domain is short-term analysis. A speech utterance is usually a sequence of phonemes. In each speech utterance, most of its properties such as amplitude, pitch, voicing, etc., fluctuate across time. Contemporary different articulator configurations make different sounds, therefore the mechanism that generates the signal changes with time. As a result, in order to analyse the speech utterance, it is divided into uniform segments known as time frames ( ⃗ f r), each of which is independently processed and analyzed.

The speech utterance y[n] is segmented into T time-frames (each of length L samples). An overlap of L/2 samples is used avoiding artifacts thereby obtaining a processed representation of whole speech utterance as

[ ⃗ f r 0 , ⃗ f r 1 , ..., ⃗ f r T-1 ], where for each time stamp t ∈ [0, 1, ..., T-1] ⃗ f r t = y t * L 2 + i i=L-1 i=0 (3.1)
These small time frames ⃗ f r t are sliced using a small Hamming window ψ[L] as shown in Equation (3.2) (L is same length as time-frame length). A value of L is chosen 20 to 30 ms during which the system (vocal tract) is considered to be stationary throughout the window, hence the signal's spectral characteristics are assumed to remain constant throughout this region [START_REF] Hardcastle | The Handbook of Phonetic Sciences[END_REF].

ψ[i] = 0.54 -0.46 cos iπ L (3.2)

Acoustic features

In developing any speech recognition or stuttering identification system, representative feature extraction and selection is extremely an important task that affects the system performance. The first common step in the speech processing domain is feature extraction. With the help of various signal processing techniques, we aim to extract the features that compactly represent the speech signal and approximate the human auditory system's response [START_REF] Huang | Spoken Language Processing: A Guide to Theory, Algorithm, and System Development[END_REF].

The various feature extraction methods that have been explored in the stuttering recognition systems are envelope parameters [START_REF] Howell | Automatic recognition of repetitions and prolongations in stuttered speech[END_REF], 1 st to 3 rd formant frequencies and their amplitudes [START_REF] Czyzewski | Intelligent processing of stuttered speech[END_REF][START_REF] Khara | A comparative study of the techniques for feature extraction and classification in stuttering[END_REF], spectral measure (fast Fourier transform (FFT) 512) [START_REF] Szczurowska | The application of kohonen and multilayer perceptron networks in the speech nonfluency analysis[END_REF][START_REF] Świetlicka | Artificial neural networks in the disabled speech analysis[END_REF],

MFCCs [START_REF] Chee | Overview of automatic stuttering recognition system[END_REF][START_REF] Khara | A comparative study of the techniques for feature extraction and classification in stuttering[END_REF][START_REF] Hariharan | Classification of speech dysfluencies using LPC based parameterization techniques[END_REF][START_REF] Esmaili | An automatic prolongation detection approach in continuous speech with robustness against speaking rate variations[END_REF],

LPCCs [START_REF] Khara | A comparative study of the techniques for feature extraction and classification in stuttering[END_REF][START_REF] Hariharan | Classification of speech dysfluencies using LPC based parameterization techniques[END_REF]), pitch, shimmer (López-de Ipiña et al., 2018), zero crossing rate (ZCR) [START_REF] Khara | A comparative study of the techniques for feature extraction and classification in stuttering[END_REF], respiratory biosignals [START_REF] Villegas | A novel stuttering disfluency classification system based on respiratory biosignals[END_REF]. [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF] exploited the use of spectrograms (as a gray scale image) as a sole feature extractor for stutter recognition and thus making it suitable for the convolutional neural networks.

For stuttering domain, MFCCs are the most suitable and are the most commonly used features in stuttered speech domain. Thus for all our experimental studies in the following chapters, we use MFCCs features extracted after every 10 ms on a 20 ms window for each speech sample.

MFCCs:

In order to investigate the properties of speech sounds, spectral-domain representation has shown to be quite effective. The spectral envelope, for example, is the smooth curve that follows the peaks of the spectrum for every given analysis frame and is regulated by the configuration of the vocal tract. Each phoneme has its own unique spectral features that may be utilized as a blueprint for its identification [START_REF] Huang | Spoken Language Processing: A Guide to Theory, Algorithm, and System Development[END_REF][START_REF] Hardcastle | The Handbook of Phonetic Sciences[END_REF]. The human auditory system, capable of perceiving sounds within a fixed range of 20 Hz to 20 kHz, is able to discriminate sounds at lower frequencies easily than the higher frequency region with the same frequency distance between the tones. Most people can recognize the difference between a 100 Hz and a 200 Hz sound. By the same argument, we should presume that we can distinguish between a 10000 Hz and 10100 Hz.

Even if the distance between the two sets of sounds is exactly the same, our perception of the frequency range is not, thus, humans find it far more difficult to distinguish between tones at higher frequencies.

To tackle these issues, researchers suggested different spectrum modifications so that the obtained features adhere to a compressed representation driven by the perceptual process of the human ear [START_REF] Jurafsky | Speech and Language Processing[END_REF][START_REF] Huang | Spoken Language Processing: A Guide to Theory, Algorithm, and System Development[END_REF][START_REF] Hardcastle | The Handbook of Phonetic Sciences[END_REF].

Thus, after computing the magnitude spectrum, each frame is transformed to a non-linear perceptual Mel scale using the Equation (3.3).

M el(F ) = 2595 * log 10 1 + F 700 (3.3)
where F is linear frequency. The Mel scale as shown in Figure 3.2 intends to emulate human auditory system's sensitivity by warping the frequency scale using closely packed narrow bandpass filters at lower frequencies and gradually broader and loosely spaced filters at higher frequencies.

Challenges in stuttering domain

Even though a handful number of stuttering datasets (for completed picture of the datasets, please see Appendix A) have been curated, however only three have been made public including UCLASS, LibriStutter, and SEP-28k datasets. Moreover, these datasets come with several flaws. The UCLASS lacks annotations and meta data information such as microphone used, etc. In SEP-28k stuttering datastet, the meta-data information such as speaker information, recording conditions, etc., is missing. This section discusses the various challenges faced by stuttering datasets and SD systems. In this thesis, we try to address some of these challenges in the follow up chapters.

Data collection problem in stuttering domain

One of the most common barriers that need to be addressed is the issue of scarcity of data on stuttering identification research. There are only a few natural stuttered datasets as discussed in Appendix A. Usually, medical data collection is expensive and very demanding, and stuttering is no exception. Thus, having ample speaker and sentence variation adds more complexity to the stuttering domain. In order to make a fine analysis across several speakers, it is appropriate to have the same content (same list of sentences). Unfortunately, in practice, when a PWS is asked to read a list of sentences, the disfluency effects are greatly reduced. For this reason, more spontaneous speech is used in the hope to induce disfluencies.

Moreover, depending on the speaker, the presence of disfluency in a recording is more or less important for several reasons: emotional state, speaking in public or alone, spontaneous or read speech, etc. This makes the collection of a corpus even more difficult and its size from one speaker to another can be variable if one aims at having a comparable number of examples of disfluencies. Moreover, it is extremely difficult, if not impossible, to collect a corpus that contains the same number of examples of each type of disfluency. It is even more challenging to achieve high variability in gender, language, and dialect. It should be noted that the recording of spontaneous speech must be well controlled to comply with the legislation. Due to the sensitivity of medical data and privacy concerns, it can not be applied at a large scale. Currently, we are not dealing with anonymization, as the voice could identify the speaker, but a minimum effort in this direction is required. In order to identify stuttering using deep learning models, the data must be properly labelled. Different background noises can corrupt the stuttered speech data. Likewise, the noise of recording equipment can also degrade the speech signal.

Stuttering data annotation ambiguity

It is no doubt that deep learning has led to enormous advancement in the speech domain, nonetheless, it demands a large amount of labelled data, and also, the dataset bias has plagued current SD methods. Annotating the stuttered speech requires expert SLP/ST, and are also inclined towards the idiosyncratic belief of SLP/ST, thus is expensive and laborious.

Unlike the other datasets such as ImgaeNet [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF], LibriSpeech [START_REF] Panayotov | Librispeech: An ASR corpus based on public domain audio books[END_REF], where it is easy to label a sample, however in stuttering datasets, it is not straightforward due to the fact that a stuttered speech sample can be labelled differently by different speech therapists/phoneticians.

Lack of evaluation protocol in stuttering detection

Different studies have used different evaluation metrics to test the performance of their SD systems. Accuracy [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF], F1-score [START_REF] Bayerl | Detecting dysfluencies in stuttering therapy using wav2vec 2.0[END_REF][START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], UAR [START_REF] Schuller | The ACM multimedia 2022 computational paralinguistics challenge: Vocalisations, stuttering, activity, & mosquitos[END_REF], etc., have been used widely in SD. However, there is no single consistent metric that is being used across the SD community which makes it hard to compare the newly developed SD systems to the existing ones. Even though the works mentioned above show promising results in SD, however, there are no consistent data selection (train/val/test) rules and no clear evaluation protocol for evaluating various SD methods which makes it extremely difficult to compare the proposed SD methods with the state-of-the-art SD techniques. Therefore, it is essential to have a unified data selection procedure for comparing the performance of different SD systems, as well as a single consistent evaluation metric, which considers the class count when evaluating the performance of the SD methods.

Class imbalance in stuttering detection

Stuttering datasets also suffer from class imbalance problems, i.e., the number of samples available for different disfluent categories is not uniform. It is mentioned that in stuttering, the repetitions are the most frequent ones followed by prolongations, and blocks [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. However, in the UCLASS dataset, the block type is present in the majority followed by repetitions and prolongations. In SEP-28k, the majority class is interjections followed by repetitions, blocks, and prolongations respectively. The model trained on these type of imbalanced datasets is biased towards the majority class.

Lack of appropriate stuttering representations

Another issue in the stuttering speech domain is the need for hand-engineered features, which approximate the human auditory system. MFCCs are the principal set of hand-engineered acoustic features that have been used mainly for stuttering identification tasks.

The main drawback of this approach is that being manual is cumbersome and requires human knowledge. Over the past few years in the speech domain, the use of hand-engineered acoustic features is gradually changing and representation learning is acquiring recognition as an effective alternative to learn and capture task-specific features directly from raw speech signals, thus circumvents the hand-engineered feature extraction module from the pre-processing pipeline [START_REF] Latif | Deep representation learning in speech processing: Challenges, recent advances, and future trends[END_REF].

Multimodal stuttering learning

In stuttering identification, deep learning has been successfully applied to single modalities like text and audio. Inspired by the human brain, where the perceptions are carried out through the integration of information from several sensory inputs including vision, hearing, smell, etc., [START_REF] Ngiam | Multimodal deep learning[END_REF] proposed a multi-modal (audio visual) learning and showed how to train deep models that learn effective shared representations across the two modalities. The stuttering itself exhibits an audio-visual problem. Cues are present both in the visual (e.g. head nodding, lip tremors, quick eye blinks, and unusual lip shapes) as well as in the audio modality [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. This multimodal learning paradigm could be helpful in learning robust stutter-specific hidden representations across the cross-modality platform, and could also help in building robust SD systems.

Domain adaptation in stuttering detection

Most of the existing SD techniques proposed so far are evaluated on a dataset of specific languages consisting of limited speakers. The existing SD techniques depend on a probabilistic model to capture domain-specific factors so that any alteration in the input speech domain could have a significant impact (in terms of language or speakers) at the time of inference. It is yet to be explored that, how well an SD technique performs in a cross-domain or cross-language environment. There could be two possible scenarios for the cross-language issue: the first is when the model is trained with a specific-language data, but tested in other languages; the other scenario could be, during training, a disfluent person registered in one language, but evaluated in a different language at the test time. Learning stutter-specific features that are invariant to variabilities in language, speakers, recording conditions, etc., could improve the performance of SD systems.

Multi-stuttering identification

Most of the SD studies focus on utterances, which consist of only one type of stuttering. However, the speech utterance can contain a mix of stuttering such as d-d-d--dog dog is big, which consists of syllable repetition, prolongation and word repetition types of disfluencies [START_REF] Sawyer | By the numbers: Disfluency analysis for preschool children who stutter[END_REF]. There is a lack of studies in detecting multiple stuttering types if present in an utterance, and to the best of our knowledge, [START_REF] Ghonem | Classification of stuttering events using i-vector[END_REF] is the only study, that has been carried out to detect multiple stuttering types (repetition-prolongation) in an utterance.

Neural networks for stuttering detection

Artificial neural networks (ANNs) consist of several connected computing neurons that loosely model the biological neurons [START_REF] Goodfellow | Deep Learning[END_REF]. Like the synapses in biological neurons, each neuron can transmit a signal to other neurons via connections. A neuron receives a signal, processes it, and can transmit a signal to other connected neurons.

The connections have weights associated with them which adjusts the learning procedure as shown in Figure 3.3 [START_REF] Goodfellow | Deep Learning[END_REF]. ANNs are trained by processing examples that maps a given input x = [x 0 , x 1 , x 2 , ..., x n ] to its corresponding result by forming a probability-weighted associations between the two using the Equation (3.4), where α(.)

is a non-linear activation function which transforms the weighted sum of inputs to the probability vector, θ = [w 0 , w 1 , w 2 , ..., w n ] are the synaptic weights with a bias term w 0 .

ŷ = α n i=1 w i * x i = α θ T x (in vector notation) (3.4)
In practice, we employ neural networks with a more complex architecture characterized by numerous connections between thousands of neurons indicated by weights and biases than a simple perceptron model to learn sophisticated non-linear mappings required by the real-time applications such as emotion recognition, ASR, SD, etc. The input to the neural network is passed through a directed mechanism called forward propagation as shown in Figure 3.4, which computes the output class probabilities of the network by processing the input via several layers with the first layer called input layer and the middle layers are called hidden layers.

Training: Give a stuttered dataset D = (X i , d i , s i ) N i=1 , with X i = [x 0 , x 1 , x 2 , ...,
x n ] are speech frames, d i is stutter label, and s i is speaker label, the training is conducted with the help of backpropagation by optimizing the loss function by computing the error difference between the predicted output and its corresponding ground truth. Continuous weight adaptations will cause the ANNs to produce a similar output as the ground truth.

After an adequate number of weight adjustments, the training can be terminated once the optimization criteria is reached [START_REF] Goodfellow | Deep Learning[END_REF].

Loss function: One of the most common loss functions used in deep learning classification tasks is cross-entropy [START_REF] Goodfellow | Deep Learning[END_REF]. It is a metric used to assess how well a classification performs on a given task. The cross-entropy in a true sense measures the difference between the true probability distribution and the predicted distribution that can be defined for multi-class classification task as:

L = - C c=1 y o,c log(p o,c ) (3.5)
where, C is number of classes, y is a binary indicator (0 or 1), if the class label c is the correct classification for the observed sample o, and p is the predicted probability. A mapping is performed by optimizing the loss function using a gradient descent to update each of parameters w ji ∈ θ by the following step:

w ji = w ji -η∆ w ji ∆ w ji = ∇ w ji L (3.6)
where η is the learning rate ∆ w ji is the gradient of the L with respect to the weight w ji connecting the last hidden layer to the output classification layer.

Activation function: There are several possible choices in selecting an activation function for the neural network including sigmoid, rectified linear (ReLU), softmax, etc.

The sigmoid non-linearity activation function σ(x) = 1/(1 + e -x ) saturate and kills the gradient thereby stopping the information to flow through the network and leads to vanishing gradient problem. Moreover, the σ(x) is highly sensitive to the weight initialization with large initial weights forces most of the neurons to get saturated and in that case the neural network model will barely learn [START_REF] Goodfellow | Deep Learning[END_REF]. The ReLU non-linearity activation function has become the most adapted choice in the last few years in speech processing community [START_REF] Nassif | Speech recognition using deep neural networks: A systematic review[END_REF]Latif et al., 2020a;Ning et al., 2019b). In mathematical formulation, it is defined as:

α(x) = max(0, x) (3.7)
with its derivative as follows:

α ′ (x) = 0 if x < 0 1 if x > 0 (3.8)
Compared to the sigmoid and Tanh, the ReLU can simply be thresholded at zero, avoiding expensive operations such as exponentials resulting in faster training of the deep learning models [START_REF] Zeiler | On rectified linear units for speech processing[END_REF]. Another popular activation function is the softmax function, which ensures that the outputs are positive and add up to 1. In classification problems, it is commonly used as the activation for the output layer resulting in a probability distribution vector over the categorical classes.

x 0

x 1

x 2

x 3

x 4

x 5

x 6

Input layer

Relevant artificial neural network models

Deep learning has allowed researchers to investigate a variety of complex network architectures that may be suitable for specific tasks. In this section, we give a brief overview of the models that are suitable for speech processing, where the frame level MFCCs (as described in Section 3.2.2) of the full speech utterance are fed to the network in learning the temporal behaviour of the speech signal. The most basic form of ANNs are the feedforward networks as depicted in Figure 3.4, are the excellent function approximators to discover the relation between the independent and dependent variables of the data. However, they fail to capture the spatial information in images, and temporal information in speech data efficiently. In order to capture the spatial information in images effectively, Convolutional neural networks (CNN) inspired by the mechanism of visual cortex have been put forward [START_REF] Goodfellow | Deep Learning[END_REF]. A CNN's fundamental neuron behaves like a kernel with a predefined 2D receptive field, moving over the input image and performing a convolution operation with the area covered by it. To learn different spatial properties of an image, several kernels convolve with the same input image followed by a non-linear activation such as ReLU and a pooling operation to convert the kernel output to a feature map which represents discriminative features such as shapes, edges in the given image.

The time delay neural network TDNN is another feed-forward suitable architecture to model the temporal information present in speech data. Usually, the initial layers in a standard deep neural network learn wider contexts when processing a temporal input signal.

However in TDNN network as shown in Figure 3.5, the initial layers learn and capture only smaller contexts and the deeper ones compute the activations from a wider context, thus the deeper/higher layers are able to capture and learn the longer temporal contexts [START_REF] Waibel | Phoneme recognition using time-delay neural networks[END_REF]. Because of the large overlap between neighboring contexts, TDNNs can be made computationally efficient by sub-sampling (called dilation) the activations that are passed on to the next layer.

Recurrent neural networks (RNN) belong to a family of deep neural architectures

where connections between neurons/nodes form a directed graph along a temporal sequence, thus allowing it to show temporal dynamic behavior. RNN consist of an internal state (memory) that is used to process variable length input sequences. This structure makes RNN good for modeling sequential tasks like time series, connected handwriting, video, and various speech application tasks [START_REF] Goodfellow | Deep Learning[END_REF].

Adversarial learning for speaker invariant stuttering detection: Domain adversarial training has been extensively applied in speech processing which aids in the adaptation of neural network classifiers to new domains such as different speakers, languages, environmental conditions, etc. (Ganin and Lempitsky, 2015a;Meng et al., 2018b). The main principle of adversarial training allows neural networks to learn latent representations in the hidden layers that are indifferent to data belonging to similar classes but coming from different domains (like speakers, language, microphone). For instance, the latent features for a particular stuttering class such as prolongations will be almost identically distributed when learned by adversarial training on various speakers. An adversarial branch that predicts the domain such as speaker is appended to a neural network classifier. This is accompanied by a gradient reversal layer which uses the negative scalar value to scale the backpropagated gradients as shown in Figure 3.6.

Self supervised learning

Supervised deep learning has been the cornerstone of speech transformation, which has provided remarkable advances for scenarios rich in labeled data. This high dependence on supervised learning paradigm, counter-intuitively, has hindered progress in other speech domains such as stuttering that are limited in resources and do not receive the same level of labeling effort. Inspired by the manner in which young children acquire their first language via involvement with the environment, researchers are attempting to learn speech representations in a self-supervised fashion drawn from the data itself that are able to capture low-level acoustic and lexical information in addition to semantic content [START_REF] Mohamed | Self-supervised speech representation learning: A review[END_REF]. SSL leverages the information from the input data itself to learn latent speech representations that are useful for other downstream tasks [START_REF] Mohamed | Self-supervised speech representation learning: A review[END_REF]. SSL usually is combined of two modes. In the first mode, SSL is used to learn good latent speech representations by pre-training the deep learning models on massive unlabeled datasets with some pre-auxilliary tasks such as predicting future speech frame based on the previous context frames. These learned speech representations are then used in the downstream tasks for low-resource settings either in the same frozen form or by fine-tuning the entire network in a supervised manner [START_REF] Baevski | Wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF][START_REF] Pepino | Emotion recognition from speech using wav2vec 2.0 embeddings[END_REF]. The SSL framework has shown exceptional performance in the tasks like automatic speech recognition (Mohamed Relevant self-supervised learning architectures: There are several state-of-the art SSL models, however we choose Wav2Vec2.0 for our experimental study in stuttering detection. The Wav2Vec2.0 has been proven effective in ASR [START_REF] Baevski | Wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF], emotion detection [START_REF] Pepino | Emotion recognition from speech using wav2vec 2.0 embeddings[END_REF], etc. The Wav2Vec2.0 model is trained in two phases. The first phase takes raw audio as an input and is pre-trained using large unlabeled data and aims at learning good high-level speech representations [START_REF] Baevski | Wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF]. After the first pre- sification approaches) with an emphasis on creating automatic tools for its detection and identification. The majority of work detects and identifies stuttering using language models [START_REF] Zayats | Disfluency detection using a bidirectional LSTM[END_REF][START_REF] Chen | Controllable time-delay transformer for real-time punctuation prediction and disfluency detection[END_REF] or by ASR systems [START_REF] Alharbi | A lightly supervised approach to detect stuttering in children's speech[END_REF][START_REF] Alharbi | Sequence labeling to detect stuttering events in read speech[END_REF], which first translates audio signals into their appropriate textual form, and then by the application of language models, detects or identifies stuttering in it. Even though this method of detecting stuttering has achieved encouraging results and has been proven effective, the reliance on ASR makes it computationally expensive and prone to error. Moreover, the adoption towards ASR task results in possible loss of stuttering relevant information such as prosody and emotional content [START_REF] Bayerl | Detecting dysfluencies in stuttering therapy using wav2vec 2.0[END_REF]). An alternative strategy of SD is to apply the machine/deep learning classifiers directly to the acoustic represented stuttered speech signal without involving ASR and language models to reveal the stuttering type embedded in it. In this section, we discuss the various and state-of-the art classifiers used in SD domain.

SD systems process and classify underlying stuttering embedded speech segments.

Including traditional classifiers, many statistical machine learning techniques have been explored in the automatic detection of stuttering. However, the studies are empirical, so A comprehensive detail on state-of-the-art SD methods can be found in the review paper [START_REF] Sheikh | Machine learning for stuttering identification: Review, challenges and future directions[END_REF].

Summary

This chapter presents an overview of underlying speech processing principles which forms the base for extracting acoustic features. Moreover, this chapter also discusses the various challenges in SD task. In addition to speech processing fundamentals, we also give a brief overview about the relevant deep learning models employed and adapted for SD in this thesis.

StutterNet

"I may be deprived of eloquence, but my mind can never be dumb." [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF][START_REF] Kourkounakis | FluentNet: End-to-end detection of stuttered speech disfluencies with deep learning[END_REF][START_REF] Bayerl | Detecting dysfluencies in stuttering therapy using wav2vec 2.0[END_REF] which can have a huge impact on other disfluent categories. In addition to this, most of the studies have used only a small subset of speakers in their training, which makes it hard to generalize on a large pool of speakers due to their variability in language and accent. The obvious question still remaining in the stuttering domain is whether the stuttering patterns are uniform and consistent across a large pool of speakers.

M
In this chapter, we introduce and describe StutterNet, a novel deep learning based stuttering detection capable of detecting and identifying various types of disfluencies. Most of the existing work in this domain uses ASR combined with language models for stuttering detection. Compared to the existing work, which depends on the ASR module, our method relies solely on the acoustic signal. We use a TDNN suitable for capturing contextual aspects of the disfluent utterances. TDNN has been widely used for different speech classification problems such as speech and speaker recognition [START_REF] Snyder | X-vectors: Robust DNN embeddings for speaker recognition[END_REF], etc. We introduce this for stuttering detection task. The proposed method, referred to as StutterNet is a multiclass classifier with output as stuttering types and fluent. Our method achieves promising results and outperforms the state-of-the-art residual neural network based method.

StutterNet architecture

In developing any speech domain application, the representative feature extraction is the most important that affects the model performance. With the aid of signal processing techniques, several features of the stuttered speech signal can be extracted like raw waveform, spectrograms, mel-spectrograms, and or MFCCs. However, our aim is to compute and extract the features that compactly characterize the stuttering embedded in a speech segment and also which approximates the human auditory system's response. For stuttering domain, MFCCs as described in Section 3.2.2 are the best suitable and are the most commonly used features in stuttered speech domain, thus, we use MFCCs as the sole features to our StutterNet network. These features are generated after every 10 ms on a 20 ms window for each 3-sec (3-sec for SEP-28k and 4-sec for UCLASS dataset) 2 audio sample. This four-second window is used as stuttering lasts on average for four seconds [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF].

The Similarly, layer 3 sees a total context of 15 time frames. The SP layer computes and concatenates the mean and standard deviation by aggregating all T output frames at the end of fifth layer. The SP layer accumulates the information across the temporal dimension which makes it suitable for subsequent layers to operate on the entire temporal speech segment.

consists of five time delay layers with the first three focusing on [t -2, t + 2] (dilation = 1), {t -2 , t, t + 2} (dilation = 2), {t -3 , t, t + 3} (dilation = 3), and the other two on {t} contextual frames, respectively. This is followed by a mean and standard deviation pooling layer, three FC layers, and a softmax layer on top of the network that reveals the classifi-

Experimental evaluation

Datasets

UCLASS:

We have used the UCLASS release 1 stuttering dataset that has been created by the Department of Psychology and Language Sciences, University College London [START_REF] Howell | The university college London archive of stuttered speech (UCLASS)[END_REF]. This dataset consists of monologue samples from 139 participants. Of these, 128 have been chosen in this case study with females 18 and males 110.

Of these, 104 speakers were used for training, 12 for validation and 12 for testing. The audio samples were annotated manually by listening to the recordings of speech segments.

The annotations have been carried out into different categories of stuttering including: core behaviors (blocks, receptions, and prolongations), fluent, repetition-prolongations, blocks-prolongations, repetition-blocks, and blocks-repetition-prolongations. However, for UCLASS dataset, we are focusing only on the core behaviors as the dataset contains only few of the remaining ones. Each monologue audio clip was sliced into 4-second and down sampled to 16 kHz segments, resulting in a total of 4674 speech segment samples. the remaining 10% of the podcast for evaluation in a 10-fold cross-validation scheme. The speaker information is missing from the SEP-28k dataset, so we divided the dataset based on podcast ids (assuming each podcast is having a unique speaker). The SEP-28k dataset is publicly available, but the details about the train, validation, and test set splits are not provided. So, we decided to create a publicly available protocol that ensures no overlap of podcasts between train, validation, and test sets4 .

SEP-28k:

In order to evaluate the StutterNet method on the UCLASS and SEP-28k datasets, we adopted K-fold cross validation technique , where K=10. We conducted 10 experiments, each consisting of random sampling of 80% for training, 10% for validation and last 10% for testing 5 . The reported results are the average of 10 experiments. All experiments were trained with an early stopping criteria of patience 7 on validation loss.

Implementation

We develop StutterNet with PyTorch library in Python. We use a learning rate of 10 -4 , amsgrad optimizer, and cross-entropy loss function. We use Librosa library for the feature extraction. We select models using an early stopping with a patience of seven epochs on validation loss. We compare the results obtained by StutterNet to existing method [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF] in the same experimental framework.

Evlauation metric

To evaluate the model performance, we use the following metrics: macro F1-score, and accuracy which is the standard and are widely used in the stuttered speech domain. The macro F1-score (F 1 ) (which combines the advantages of both precision and recall in a single metric unlike unweighted average recall which only takes recall into account) from (Equation (4.1)) is often used in class imbalance scenarios with the intention to give equal importance to frequent and infrequent classes, and is also more robust towards the error type distribution [START_REF] Opitz | Macro F1 and macro F1 a note[END_REF].

F 1 = 1/C k F 1 k = 1/C k 2.P k R k P k + R k (4.1)
where C is the number of classes and P k , R k , and F 1 k denotes the precision, recall, and F1-score with respect to class k.

Results and discussion

The results of our 4.5 .

Compared to [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF], our optimized proposed method gains a margin of 4.69% in overall average accuracy. For detecting the core behaviours and the fluent part, the margins are also substantial (improvements of 7.49%, 14.69% in repetition and fluent speech segments, respectively). Most previous work tends to avoid block disfluencies because of their similar nature to silence and prolongation (blocks are prolonged without audible airflow). As shown in Table 4.5, the proposed StutterNet can detect and classify the block stuttering with an average accuracy of 47.14%. Moreover, our technique relies on the assumption that stuttering usually lasts for four-second window size [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Note that some of the stuttering (in particular prolongation and repetition) can exceed more than four seconds in speech [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF], thus causing those prolongation stuttering likely to be misclassified. We also perform experiments on SEP-28k dataset and compared our proposed Stutter-Net with the convolutional LSTM (ConvLSTM) model proposed by [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF]. From Table 4.6, it can be clearly seen that the proposed StutterNet outperforms ConvLSTM [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] in overall macro F1 (F 1 ) score by a relative margin of 14.71%. The StutterNet is a single branched SD system which suffers from a class imbalance problem as can be seen from the major classes of interjection and fluent classes where the detection performance is greater as compared to minority classes. The ConvLSTM SD model performs better in repetitions and interjection disfluencies most likely because of the class balanced training using separate branching for each disfluency type.

Summary

This chapter addresses the stuttering detection as a multi-class classification problem and introduces StutterNet which is based on time delay neural network for stuttering detection. Next, this chapter discusses the StutterNet architecture in detail followed by experimental setup. Next, we give the details about UCLASS and SEP-28k datasets, and their protocol for splitting the data by ensuring there is no overlap of speakers across train, validation and test sets. This is followed by experimental results and discussion in comparison to the baselines.

Multi-branch and multi-contextual StutterNet

"I like to be multi-contextual, which is much more important than being uni-cultural."

Cornel West

Introduction

Most of the earlier work employed only a small set of disfluent speakers in their experimental studies, and has approached the stuttering detection problem as a binary classification problem: disfluent vs fluent identification or one vs other type 1 . The StutterNet we propose as discussed in previous Chapter 4, tackled the SD as a multi-class classification problem and evaluated the proposed methodology on a medium sized dataset (100+ speakers) as compared to the earlier work [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF]. Although this network has shown promising results in SD, it has several deficiencies. First, it doesn't generalize well on unseen data and leads to overfitting due to the limited amount of data available for training. In addition to data scarcity, the stuttering dataset was collected from podcasts in a clean environment, which makes the trained StutterNet difficult to generalize in other environmental conditions. Second, obtaining class-balanced datasets is extremely difficult and very expensive in the speech domain and the stuttering datasets are no exception. Deep learning classifiers trained on highly class-imbalanced datasets are usually biased towards the majority class, which results in poor modeling of minority classes [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF]. In addition to the above deficiencies, we found in our previous Chapter 4 that the 1 The following article forms the basis of this chapter: context is very important in SD. A larger context increases the performance of prolongation and repetition types of disfluencies but decreases the recognition performance of fluent speech segments on the UCLASS dataset. Moreover, we found some annotation issues with the UCLASS dataset, so we have shifted to the newly created SEP-28k stuttering dataset for our experimental studies (SEP-28k is the largest dataset available in stuttering domain) [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF].

Shakeel

Addressing deficiencies

Class Imbalance: In class imbalance learning, the distribution of samples across each class is not uniform. The deep neural network (DNN) classifiers, trained on highly imbalanced datasets generally perform poorly for the minority class and favor the majority class [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF]. In the worst case, where, there is an extreme imbalance in the training set, the majority class dominates the learning, and samples among the minority class may go undetected, thus affecting the performance [START_REF] Weiss | Foundations of Imbalanced Learning[END_REF].

The class-imbalance is one of the major problems in real-world applications, including multi-lingual speech recognition [START_REF] Winata | Adapt-and-adjust: Overcoming the long-tail problem of multilingual speech recognition[END_REF] Over the years, the class imbalance problem is one of the main concerns due to its prevalence, especially in the biomedical domain. Several methods have been proposed to tackle the class imbalance problem, which are mainly categorized into three groups: datalevel, cost-level, and architecture-level [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF]. Data-level approaches attempt to re-balance the class distribution by means of some re-sampling methods which include: under-sampling, over-sampling, or combined over-sampling and under-sampling [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF], [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF]. The architecture-level approaches attempt to modify the existing algorithm or develop a new one to tune and adapt it for imbalanced datasets [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF], [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF],. Cost-level approaches attempt to influence the loss/objective function by providing comparatively higher misclassification cost penalties to the minority class in order to force the model to learn about minority classes [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF][START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF]. In DNNs, addressing the class-imbalance problem by resampling may either get rid of sensitive speech samples that are extremely important in training when under-sampling. It can also add numerous quantities of duplicated speech samples under the over-sampling strategy, which eventually makes the training expensive and makes the DNN model likely to overfit [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF]. Because of these limitations, we investigate cost-level and architecture-level approaches in this work.

For the cost-based approach, we modify the standard cross entropy loss by assigning weights to different classes [START_REF] Cui | Class-balanced loss based on effective number of samples[END_REF]. We set the class weights inversely proportional to the number of samples. We define the weight for class i as

w i = N C * N i (5.1)
where N is the number of training samples, C is number of classes, N i is the number of training samples for class i.

Therefore, the weighted cross-entropy (WCE) over the train set can be defined as,

L W CE = 1 B B b=1 M i w i * log(p i ) M i, i∈B w i (5.2)
where B is the number of batches, M is number of stuttered speech samples in a batch

b i , p i = e c i C j=1 e c j
is the predicted probability of class c i of sample i.

For architecture-level, we propose a multi-branch approach similar to the work by [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] .2.: Repetition stuttering with utterance "said that, that" and the effect of various data augmentations. From 0 to 0.25 seconds, the speaker is saying "said", then followed by two repetitions "that".

recognition (Latif et al., 2020a), emotion recognition [START_REF] Akçay | Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers[END_REF]. However, as a drawback, DL based models are hungry for the data and require a substantial amount of annotated data, especially in the speech domain to address the over-fitting and robustness issues and StutterNet is no exception, and perhaps it needs more data than other speech processing areas, as the presence of examples of disfluencies in speech corpus is not frequent.

Data augmentation is a popular technique that increases the quantity and diversity of the existing annotated training data, improves robustness, and avoids the overfitting of DNNs. For normal speech recognition, data augmentation demonstrates to be an effective approach for dealing with data scarcity and enhancing the performance of various DNN acoustic methods [START_REF] Park | SpecAugment: A simple data augmentation method for automatic speech recognition[END_REF]. Several data augmentation techniques have been investigated including pitch adjust [START_REF] Meng | Mixspeech: Data augmentation for low-resource automatic speech recognition[END_REF], spectral distortion [START_REF] Kanda | Elastic spectral distortion for low resource speech recognition with deep neural networks[END_REF], tempo perturbation [START_REF] Ko | Audio augmentation for speech recognition[END_REF], speed perturbation, cross-domain adaptation [START_REF] Bell | Transcription of multi-genre media archives using out-ofdomain data[END_REF], adding noise to clean speech [START_REF] Ko | Audio augmentation for speech recognition[END_REF], spectrogram deformation with frequency and time masking [START_REF] Park | SpecAugment: A simple data augmentation method for automatic speech recognition[END_REF], mixspeech [START_REF] Meng | Mixspeech: Data augmentation for low-resource automatic speech recognition[END_REF], etc.

On the contrary, so far, very limited attention has been given to data augmentation targeting the speech disorder domain. In [START_REF] Vachhani | Data augmentation using healthy speech for dysarthric speech recognition[END_REF], speed and tempo perturbation based data augmentation were used to convert the normal speech to dysarthric speech impairment. In [START_REF] Jiao | Simulating dysarthric speech for training data augmentation in clinical speech applications[END_REF], a voice conversion adversarial training based framework was used to simulate dysarthric speech from healthy speech. In [START_REF] Christensen | Combining in-domain and out-of-domain speech data for automatic recognition of disordered speech[END_REF], normal speech samples (also called out-of-domain) were used as a data augmentation for dysarthric speech in the bottleneck feature extraction stage. In [START_REF] Xiong | Phonetic analysis of dysarthric speech tempo and applications to robust personalised dysarthric speech recognition[END_REF], the speaker-dependent parameter was computed, which was then used for augmentation of scarce dysarthric speech in tempo adjustment. In [START_REF] Woszczyk | Data augmentation for Dementia detection in spoken language[END_REF], several data augmentation techniques such as noise, time stretching, pitch shifting, time shift, masking, etc. were analysed in Dementia detection. Even though there are some studies on data augmentation targeting other speech dirsorders such as speech dysarthric detection, however, in the case of stuttering/disfluency detection, there is none.

In the stuttering domain, the employment of data augmentation is not straightforward, because most of the data augmentations like time stretch, speed perturbation, etc., alter the underlying structure of the stuttering speech sample completely. Our approach employs reverberation and additive noises because it reflects the real-world scenario and doesn't change significantly the underlying stuttering in the speech sample as shown in Figure 5.2. Reverberation consists of convolving speech samples with room impulse responses (RIR). We utilize the simulated RIRs as described in [START_REF] Ko | A study on data augmentation of reverberant speech for robust speech recognition[END_REF]. For additive noises, we utilize the MUSAN dataset, comprised of 60 hours of speech from 12 languages, 42 hours of music from various genres, and 900 hours of noises.

To augment the original speech samples, we combine the training "clean" set with below augmented copies from the following:

1. music: A single music sample file randomly chosen from MUSAN is added to the original clean stuttering speech sample (SNR: 5-15dB) (The music sample file is trimmed or repeated as required to match the duration of the clean stuttered speech sample).

2. noise: Throughout the stuttered speech, samples from MUSAN noises are added at 1 sec intervals (SNR: 0-15dB).

3. babble: Speech samples from randomly three to seven speakers are summed together, then added to the original clean stuttered speech sample (SNR: 13-20dB).

reverb:

The "clean" train set is convolved with simulated RIRs.

All the data augmentation shown in Figure 5.2 were performed using Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] tool.

Multi-contextual StutterNet:

The multi-contextual framework is based on the way humans perceive speech. In the cochlea, the input acoustic speech signal is partitioned into several frequency bands so that the information in each band can be filtered independently and thus processed in parallel in the human brain [START_REF] Allen | How do humans process and recognize speech? In Modern methods of speech processing[END_REF].

For a long time, multi-contextual has been studied for action recognition in videos (Simonyan and Zisserman, 2014), robust ASR mostly in noisy environments [START_REF] Han | Multistream CNN for robust acoustic modeling[END_REF]Bourlard and Dupont, 1996;Bourlard et al., 1996;[START_REF] Hennansky | Towards ASR on partially corrupted speech[END_REF][START_REF] Naderi | Multiresolution convolutional neural network for robust speech recognition[END_REF], speech separation [START_REF] Zhang | A deep ensemble learning method for monaural speech separation[END_REF], where the input speech signal is processed in multiple streams/contexts (multiple time or frequency resolutions), etc. Han

Experimental setup

We evaluate our proposed architecture thoroughly on the newly released SEP-28k stuttered dataset [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], and for cross copora, we used the FluencyBank dataset [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF]. for training, 10% of podcasts for validation, and the remaining 10% of the podcast for evaluation in a 10-fold cross-validation scheme. The speaker information is missing from the SEP-28k dataset, so we divided the dataset based on podcast ids (assuming each podcast is having a unique speaker).The SEP-28k dataset is publicly available, but the details about

Datasets

L(θ e , θ f , θ d ) = L f (θ f ) + L d (θ d ) 15:
Backward Pass:

16:

∇θ b = backward_propagation(L(θ), ŷ, d) // ŷ is predicted label. 17:
Parameter Updates:

18: We have considered those samples where at-least two annotators agree with the same labelling of stuttered speech sample.

θ b ← θ b -η *
LibriStutter: The original LibriStutter is having 6 classes including fluent, interjection, prolongation, sound, word and phrase repetitions. To make our experiments consistent with the SEP-28k dataset, we treated all repetitions as one class. After extracting samples based on the class label, we didn't find any interjection samples in the dataset, so we only trained with three classes including fluent, prolongation, and repetitions(for details of dataset split, please see Appendix A). We randomly selected 80% (40) speakers for training, 10% (5) speakers for validation , and remaining 10% (5) speakers for testing.

Training setup

We implement MC StutterNet using the PyTorch library. The acoustic input features used in this case study are 20-dimensional MFCC features, which are generated after every 10 ms on a 20 ms window size, extracted using the Librosa library. For 3-dim pitch and phoneme features, we use Pykaldi [START_REF] Can | PyKaldi: A Python wrapper for Kaldi[END_REF] and Phonexia [START_REF] Fer | Multilingually trained bottleneck features in spoken language recognition[END_REF] tools respectively. For training, we use Adam optimizer and cross-entropy loss function with a learning rate of 10 -2 and batch size of 128. All the results reported in this paper are the average of the 10-fold validation technique and all the training experiments were stopped by an early stopping criteria with a patience of 7 on validation loss.

Evaluation metrics

To evaluate the model performance, we use the following metrics: macro F1-score and accuracy which are the standard and are widely used in the stuttered speech domain [START_REF] Schuller | The ACM multimedia 2022 computational paralinguistics challenge: Vocalisations, stuttering, activity, & mosquitos[END_REF][START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF][START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF]. The macro F1-score (F 1 ) (which combines the advantages of both precision and recall in a single metric unlike unweighted average recall which only takes recall into account) from (Equation (5.4)) is often used in class imbalance scenarios with the intention to give equal importance to frequent and infrequent classes, and also is more robust towards the error type distribution [START_REF] Opitz | Macro F1 and macro F1 a note[END_REF].

F 1 = 1/C k F 1 k = 1/C k 2.P k R k P k + R k (5.4)
where C is the number of classes and P k , R k , and F 1 k denotes the precision, recall, and F1-score with respect to class k.

Results

In this section, we present the results with class balanced training, data augmentation, and multi-contextual learning. We propose two modifications to the vanilla StutterNet to address the class imbalance issue and one of them involves the loss function on the SEP-28k dataset. We further present the results of cross-corpora experiments on the FluencyBank dataset.

Baselines

In addition to our single branch StutterNet baseline (BL5), we first implement the state-of-the-art ConvLSTM and ResNet+BiLSTM as our baseline models for comparison purposes in the same settings. Our baseline model StutterNet is performing well in almost all the disfluent classes except blocks as compared to the baseline model used in SEP-28k paper [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] ConvLSTM (MFCC features) (we call it BL1 5 ), ConvLSTM (phoneme features) (we call it BL2), and ConvLSTM (pitch+MFCC features) (we call it BL3). In addition, we also used one more model i.e, ResNet+BiLSTM classifier (we call it BL4) from [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF]. Comparing our single branch baseline StutterNet to BL4, the model performs well only in interjection and repettion classes as shown in Table 5.1. In order to evaluate our proposed methods, we choose BL1 (best among BL1, BL2, and BL3), and BL4 from baselines for comparison purposes.

Class balanced training

Weighted cross entropy

This section discusses the impact of applying weighted cross entropy to StutterNet (which we call StutterNet W CE ) in order to improve the detection performance of minority classes including repetitions, blocks, and prolongations. stuttering information and improves the detection performance only in the repetition class by a relative margin of 84% over BL4.

In applying class balanced training, we found that there is drop in macro F 1 score.

Using

Analysis of confusion matrix

Using branches. We label this training scheme as M f rz enc . By exploiting this method, we achieved an overall detection improvement of 5.11% in F 1 over MB StutterNet (StutterNet M B ). Secondly, we apply freezing to the base encoder and StutterNet W CE (DisfluentBranch) and appended with one more FluentBranch (to distinguish between fluents and stutter samples). We label this training scheme as M f rz enc,disf and it results in an overall improvement of 6.01% in F 1 over StutterNet M B .

Data augmentation

The main experimental results obtained with various data augmentation techniques are shown in Table 5.2, where we compare the detection performance obtained with data augmented training to the baseline clean dataset. To the best of our knowledge, this is the first-ever study that investigates the impact of data augmentation in SD. We first separately train the StutterNet M B on different data augmentation techniques which are described in Section 5.2. We found that the training MB ResNet+BiLSTM and StutterNet M B with Kaldi augmentation increases the overall F 1 performance. From Table 5.2, it can be seen that the data augmentation does help in improving the F 1 in almost all the cases. Applying data augmentation with a single branched and weighted cross entropy StutterNet, there is a relative improvement of 5.74% and 3.50% in F 1 respectively, over the clean versions of the training. When data augmentation is applied to MB training, there is a relative improvement of 4.17% and 0.61% in F 1 using StutterNet M B and ResNet+BiLSTM over clean training, respectively. Moreover, applying Kaldi data augmentation on top of the M f rz enc and M f rz enc,disf training scheme, there is a relative improvement of 7.28%, and 6.81% in overall accuracy respectively, however, M f rz enc based augmented training results in a slight drop of F 1 score. In addition to Kaldi augmentation, we also applied pitch scaling and bandpass filter as data augmentation, however, we didn't achieve much improvement in SD.

Multi-contextual StutterNet

Different contexts show different optimized class accuracies. In order to exploit these different variable contexts and to improve the detection performance further, we propose a multi-contextual ( MC) StutterNet for SD as shown in Figure 5.3. We jointly train and optimize the MC StutterNet on the clean and augmented data using variable contexts of 5 and 9.

Cross corpora evaluation

FluencyBank: By optimizing on SEP-28k dataset in terms of data augmentation and multi-contextual, we aim to evaluate our proposed methodology MC StutterNet on a cross corpora scenario. We trained MC StutterNet on SEP-28k dataset and evaluated it on Flu-encyBank dataset which comprises samples from 33 podcasts. We found that the model trained on one corpus (SEP-28k) fails to generalize and performs poorly on cross-domain corpora. As can be seen from the block classes show more degradation in their performance in cross corpora scenarios. Furthermore, we applied data augmentation in cross corpora evaluation, and we found that it boosts the detection performance of all the classes, which results in an improvement of 13.23% in F 1 . The block samples from FluencyBank and SEP-28k dataset show very low detection performance and seem to be the hardest ones to detect.

Summary of proposed methods

This chapter discusses and addresses the deficiencies of StutterNet model that was presented in Chapter 4. In brief, we experiment with the following new modifications to our StutterNet baseline by first addressing the class imbalance problem in stuttering domain via weighted cross entropy and multi-branch training schemes. On top of that, we also investigated the impact of data augmentation in stuttering detection. And lastly, we exploited the different contexts for the identification of stuttering classes. The systematic improvement of all the proposed modifications can be seen in the Table 5.6, where the MC StutterNet outperforms against its single contextual framework in stuttering detection.

Multi-task and adversarial learning in stuttering detection

"Stuttering shows too much thinking. The human brain con only multitask efficiently at a certain level, beyond that the thoughts and signals get mixed up."

Roger W. Hancock

Introduction

The recent works have made significant strides in deep learning based stuttering detection. However, all the existing systems focus only on single task stuttering learning strategy without focusing on the auxiliary tasks. Unfortunately, this is not the approach that we human beings process the speech utterances, rather we simultaneously decode the primary task with other meta contents like speaker characteristics, linguistic content, emotion, etc.

This multi-tasking (MTL) have been proven effective in computer vision, ASR, emotion detection, speaker recognition, etc., by jointly learning several tasks through a learned shared encoding [START_REF] Vandenhende | Multi-task learning for dense prediction tasks: A survey[END_REF][START_REF] Pironkov | Multi-task learning for speech recognition: an overview[END_REF]. In this chapter, we investigate the impact of MTL 1 approach in stuttering detection, by jointly learning stuttering and metada information. SD systems are further affected by source diversity, be that linguistic content, speaker, gender, accent, or encompassing acoustic conditions as discussed in Section 3.3. In addition to MTL, we also investigate an adversarial (ADV) framework with the aim to learn robust stuttering representations. Among the various source variabili- 1 The following articles forms the basis of this chapter:

Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "Robust stuttering detection via multi-task and adversarial learning". In Proc. 3oth European Signal Processing Conference (EUSIPCO), (2022) (peer-reviewed).

ties, we specifically focus on eliminating meta-data podcast (In this thesis, we assume each podcast is having a unique speaker and we use them as speaker ids, and we use the terms podcast and speaker interchangeably) information with the intention to improve the detection performance of stuttering and its types. Inspired by unsupervised domain adaptation training [START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF] and speaker invariant affective learning [START_REF] Li | Speaker-invariant affective representation learning via adversarial training[END_REF], we explore and provide the first ever deeper analysis by using MTL and ADV in the context of stuttering detection.

Motivation

MTL is basically an inductive transfer scheme which simultaneously learn multiple tasks by optimizing several objectives using a shared hidden representation [START_REF] Caruana | Multitask learning[END_REF][START_REF] Vandenhende | Multi-task learning for dense prediction tasks: A survey[END_REF]. It provides an effective way of improving generalization. MTL aims to exploit the integrated knowledge across multiple domains with the intention of improving the performance of primary task [START_REF] Crawshaw | Multi-task learning with deep neural networks: A survey[END_REF].

A contrasting approach is to presume that robust acoustic representation of stuttering should be invariant to secondary tasks, in particular speaker recognition, as stuttering speech task should not depend on a particular set of speakers. The way to achieve such robust invariances is ADV learning [START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF]. The ADV framework learns invariances to metada characteristics in a min-max fashion that the main branch is trained to maximize the metadata (podcast) classification loss and the speaker branch is trained to minimize its classification loss. Robust representation of speech signals have been studied via ADV training in various speech domains such as emotion detection [START_REF] Li | Speaker-invariant affective representation learning via adversarial training[END_REF], acoustic matching [START_REF] Su | Acoustic matching by embedding impulse responses[END_REF], and ASR (Meng et al., 2018a). However, there is no prior study on the application of ADV frameworks in the domain of acoustic SD.

6.3. Proposed framework

6.3.1. Multi-task learning Within the MTL framework, given a training set of D = (X i , s i , d i , f i ) n i=0
, where each sample is comprised of three components: a sequence of R 20×T acoustic MFCC fea- (podcast here) loss L speaker (θ e , θ s ). Therefore, the total objective function in MTL as shown in Figure 6.1 can be represented as:

L(θ e , θ f , θ d , θ s ) =(1 -λ) * L stutter (θ e , θ f , θ d ) + λ * L speaker (θ e , θ s ) L stutter (θ e , θ f , θ d ) =L fluent (θ e , θ f ) + L disfluent (θ e , θ d ) (6.1)
where, λ ∈ (0, 1), is a weighting parameter between the losses tuned manually.

Adversarial learning

In applying ADV framework to stuttering domain, we aim to learn acoustic representations which are robust and metada invariant, but at the same time are stuttering discriminative. For this purpose, similar to MTL, we consider an encoder E having parameters θ e , which maps T input acoustic features to representation Z by Z i = E(X i ). This Z i representation is then fed to the DisfluentBranch (denoted by D) and FluentBranch decoders (denoted by F) to output the stuttering and fluent class probabilities p(d i /Z i ; θ e , θ d ) and p(f i /Z i ; θ e , θ f ) respectively. In addition, this Z i is fed to another branch speaker decoder (denoted by S), to output metadata (podcast) probabilities p(s i /Z i ; θ e , θ s ). To learn metadata-invariant representations, our goal is to minimize stuttering classification loss, L stutter (θ e , θ d ) and to optimize θ e to maximize the speaker classification loss, L speaker (θ e , θ s ) and, at the same time minimize the speaker classification loss by optimizing θ s . The total objective loss becomes:

L(θ e , θ f , θ d , θ s ) = L stutter (θ e , θ f , θ d ) -λ * L speaker (θ e , θ s ) L speaker = - (X i ,f i ,d i ,s i )∈D log(p(s i /Z i )) L stutter = - (X i ,f i ,d i ,s i )∈D log(p(d i /Z i )) + log(p(f i /Z i )) (6.2)
where, λ ∈ (0, 1), is a weighting trade-off hyper-parameter, and s i , f i and d i are the labels.

We optimize the parameters using Adam by applying the gradient reversal layer [START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF] in speaker branch during backpropagation as shown by red curve in

Experimental setup

Dataset: For our experimental studies, we have used the SEP-28k stuttering dataset discussed in Appendix A. Out of 28,177 samples, after leaving out non stuttering samples, we used only 23573 speech segments for our case study which constitutes of 19. 65 for test set which comprises of 309, 37 and 39 podcasts respectively. For MTL and ADV, we mix the train and validation sets from baseline, and then randomly selected 90% samples for train set and 10% samples for validation set from each podcast to have at-least each disfluency from each podcast in both the sets, and the test set remains same in all the cases. In this study, we use podcast ids as meta deta information to learn robust stutter representations.

Evaluation metrics:

To test and evaluate the model performance, we use the widely and standard metrics that are used in stuttering and other speech tasks which include recall, precision, F1-score and accuracy. In addition, we also used macro F1 score as well.

The models are evaluated and compared to our previous work StutterNet discussed in previous Chapter 2. The results reported are the average of 10 experiments with 10-fold cross validation technique. For ADV, we first train the SpeakerNet for 25 epochs by freezing the other two branches, then we freeze the SpeakerNet and train the FluentBranch and Disflu-entBranch for 25 epochs. From epoch 50-75, we jointly train all the branches by applying gradient reversal layer in the speaker branch during backpropogation, and after epoch 75, we freeze all the weights and fine tune the FluentBranch and DisfluentBranch branches for the recovery phase until the stopping criteria is achieved.

Implementation:

We have used PyTorch library for our implementation purposes. We trained the models using Adam optimizer, cross entropy loss function with a learning rate of 10 -2 . The training was stopped with an early stopping criteria having a patience of 7 on validation loss.

Baseline: Table 6.1 and Table 6.2 show the results of our baseline StutterNet along with the proposed methods. In comparison with the single branch (SB), multibranch (MB) outperforms in all the disfluency class predictions with an overall improvement of 26%. However, the SB shows better performance on fluent class because of the class imbalance issue. After addressing the class imbalance issue via two branch (FluentBranch and Disfluent-Branch) training scheme, we found that the blocks and repetitions are still being confused with fluent speech as shown in Figure 6.3. This is intuitive because the repetitions are fluent speech with the same word being repeated more than once. Similarly, the blocks are fluent speech with some small initial pause followed by fluent speech. From the Table 6.1, we also observe that compared to the single task learning approach, our model based on simple MTL approach helps to boost the detection performance for stuttering types but The block class follows a trend that by increasing the λ, its detection performance also improves. For other classes, as the value of λ is increased from 0.1 → 0.9, the identification performance decreases. Following the work in (Ganin and Lempitsky, 2015a), we also tried varying λ by 2/(1 + e p i ) -1, where p i is the scaled version of epoch. However, it degrades performance.

In addition, the two class accuracy of disfluent samples in the FluentBranch decreases as we increase the value of λ in the MTL framework. The λ acts as a control parameter for the podcast information to flow through the network. As λ increases, it is allowing more and more podcast-related information to flow through the network and thus decreasing the detection performance of stutter samples in the FluentBranch. To this contrary, the podcast identification branch in the ADV framework is trying to play a good adversary and tries to remove more irrelevant information as λ value increases. This, in turn, helps in boosting the detection performance of stutter samples in the FluentBranch branch. We can conclude that the podcast information is entangled with the stuttering characteristics, and by reducing podcast-related information, the model is able to improve the discrimination between the fluent and disfluent classes, but it fails to generalize within the disfluent classes.

By analysing the macro F1 score (F 1 ) from Figure 6.5, we can see that the F 1 score decreases sharply in MTL stuttering detection setting as the λ value increases than in adversarial learning framework. As the λ value increases, the network allows more and more speaker information to pass through it during backward propagation, thus trying to dilute the latent stutter embeddings in feature encoder and results in decrease of F 1 sharply.

In contrast to MTL, with the increase in λ value, adversarial learning framework tries to unlearn more and more speaker information with an aim not to dilute the latent stutter embedidngs learned from FluentBranch and DisfluentBranch training setup.

Visualization: Figure 6.6 shows the embeddings computed from the statistical pooling layer. In MTL scheme, it is evident from the well formed podcast clusters that the model is trying to learn podcast dependent stuttering information. On the other hand, for the adversarial setting, the clusters are not visible and the model is trying to learn this meta-data invariant robust stutter features.

Summary

Due to the diversity and uniqueness of stuttering, it continues to be the most demanding and challenging to detect due to its inherent nature of huge class imbalance across different types of disfluencies. Training a model on such a type of dataset will bias the majority class. To this end, we employ an MB StutterNet (BL) in thic chapter, where, we introduce From the results, we found that the MTL framework improves the detection performance of blocks, repetitions, and interjections. We also found that the confusion rate increases for the fluent and prolongation classes. With the ADV scheme, we found that the fluent detection performance increases with a relative improvement of 3.59%. In addition, we also found that by decreasing the λ hyper-parameter in both the frameworks, the detection rate of disfluent classes increases.

There might be issues with the meta-information of the dataset, as there is a possibility of having the same speakers across multiple podcasts.

Self-supervised learning in stuttering detection

"Without big data, you are blind and deaf and in the middle of a freeway." Geoffrey Moore

Introduction

The automated stuttering detection 1 task suffers from a lack of unlabeled data and thus limits the application of large deep models. Over the years, several stuttering datasets including: SEP-28k [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], UCLASS [START_REF] Howell | Automatic recognition of repetitions and prolongations in stuttered speech[END_REF], FluencyBank [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], and LibriStutter [START_REF] Kourkounakis | FluentNet: End-to-end detection of stuttered speech disfluencies with deep learning[END_REF] have been developed for investigating different SD models. Due to a number of variable characteristics, such as language, speaker, gender, age, accent, speech rate, and others, the identification of stuttering is still a challenging task. We exploited the speaker variable and employed an adversarial multitasking learning framework to learn speaker invariant stuttering features (discussed in previous Chapter 6). Moreover due to this varying nature of stuttering from person to person, these small datasets are inclined to be biased towards these small pool of speakers [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF] datasets, which are unable to capture different speaking styles, accents, linguistic content, etc. In addition, collecting medical data requires big-budget and is very taxing, and stuttering data collection is no exception. To address this issue, we introduce a self-supervised learning framework that first learns a feature extractor for a pre-text task on a large amount of non-stuttering audio data and then employs the learned feature extractor for the downstream stuttering detection task with limited audio data. We mainly focus on speaker and contextual embeddings where the self-supervised learning (SSL) models are pre-trained on massive datasets with the hypothesis in capturing various speaking styles and other paralinguistic information which are extremely important to stuttering detection.

Speaker and contextual embeddings

Speaker embeddings

Speaker embeddings are usually computed from trained neural networks to identify and classify speakers from a group of speakers [START_REF] Desplanques | ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification[END_REF]. The pre-trained speaker embeddings have been successfully applied in speaker diarization [START_REF] Dawalatabad | ECAPA-TDNN embeddings for speaker diarization[END_REF]. The (ECAPA-TDNN) emphasized channel attention, propagation and aggregation TDNN is the state-of-the-art for extracting speaker embeddings. As depicted in Figure 7.1, the ECAPA-TDNN is composed of 1D convolution followed by three 1D

Wav2Vec2.0 contextual embeddings

The WaveVec2.0 model is a self-supervised representation learning framework of raw audio, and is comprised of three modules including feature encoder E : X → Z, contex- The paper uses two different transformer networks with the base model consisting of 12 blocks having eight attention heads at each block, and the large model is comprised of 24 blocks with 16 attention heads at each block. The feature representations Z are also fed to the quantization module which is comprised of two codebooks having 320 possible entries in each. For each vector representation z i ∈ Z, a logit of R 2×320 is chosen using (Equation (7.1)) by concatenating the corresponding entries from each codebook, which is then followed by linear transformation to produce the quantized vector q i of the local feature encoder representation z i ∈ Z.

p g,v = exp(l g,v + η v )/τ V k=1 exp(l g,v + η v )/τ (7.1)
emotion detection [START_REF] Pepino | Emotion recognition from speech using wav2vec 2.0 embeddings[END_REF]. In the stuttering speech, most parts of the speech utterance are perturbed, and it seems a plausible way to employ and explore the role of the contextual and encoder representations in SD. In this work, without fine-tuning, we employ a total of 13 contextual embeddings extracted from a local encoder and 12 layers of the contextual transformer block as depicted in Figure 7.2.

Classifier description

k-nearest neighbourhood

A non-parametric supervised algorithm based on distance metric is used mostly for classification tasks. The prediction of the query sample depends on the voting majority of its k nearest neighbors [START_REF] Murphy | Machine Learning: a Probabilistic Perspective[END_REF]. In this work, we use the Minkowski metric distance from (Equation (7.3)) with p = 2 (Euclidean distance) to fit the k-NN on the SEP-28k dataset using embeddings computed from pre-trained ECAPA-TDNN and Wav2Vec2.0 models.

Dist = k i=1 (∥x i -y i ∥) p 1/p (7.3)

Gaussian back-end

A NBC is simply a Bayesian network to handle continuous features by representing the likelihood of features using Gaussian distribution [START_REF] Murphy | Machine Learning: a Probabilistic Perspective[END_REF]

Dataset description

In this case study, we use SEP-28k stuttering dataset [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] which consists of 28,177 speech samples from 385 podcasts. After removing non-stuttered samples, we are left with 23573 annotated speech segments. We randomly selected 80% podcasts (without mixing the podcasts) for training, 10% podcasts for validation and the remaining 10% of the podcast for evaluation in a 10-fold cross-validation scheme. The speaker information is missing from the SEP-28k dataset, so we divided the dataset based on podcast ids (assuming each podcast is having a unique speaker). This dataset contains two different types of annotations including stuttering and non-stuttering. We considered only stuttering annotations (repetitions, blocks, interjections, prolongations, and fluent speech) and avoided the non-stuttering annotations (unsure, no speech, poor audio quality, music, unintelligible and natural pauses) in this experimental study. The SEP-28k dataset is publicly available, but the details about the train, validation and test set splits are not provided. So, we decided to create a publicly available protocol that ensures no overlap of podcasts between train, validation and test sets3 . The experimental results mentioned are the average of 10fold cross-validation experiments, and are compared to the baseline results from Chapter 4 which is trained on MFCC features.

Implementation

For implementing the proposed pipeline for NNet, we use PyTorch library, and for LDA, k-NN, and NBC, we have used the Scikit-learn toolkit. We have chosen a value of k = 5 using the elbow method in the k-NN classifier. The downstream NNet is trained with

Results and discussion

For thorough evaluation, we train each of the proposed models 10 times. Table 7.1.

shows the macro

Speaker embeddings:

We see from Table 7. 1 (b), that the downstream classifiers trained on ECAPA-TDNN embeddings perform poorly in all the stuttering classes as compared to the baseline results from Table 7.1. This is evident from the Figure 7.3 as well, where the different stuttering type utterances are mixed and no clear cluster is visible among the disfluency classes. Furthermore, applying magnitude normalization on ECAPA-TDNN embeddings before passing them to the downstream classifiers, improves the SD performance marginally in the majority of classes. The ECAPA-TDNN is trained and adapted for the speaker identification task and it is likely possible that the information (such as linguistic content, prosody, and emotion state) which isn't essential for that task but could be crucial for SD gets removed from the latent embeddings.

Wav2Vec2.0 contextual embeddings: Table 7. 1 (c) shows the results with the last but two-layer of contextual transformer block C of Wav2Vec2.0 with and without prior application of LDA. From the results, we can observe that for SD, the contextual embeddings from Wav2Vec2.0 outperform in all the classes with an overall relative improvement (TA) of 5.82% using k-NN, 11.54% using NBC, 10.38% using NNet over StutterNet and over MB StutterNet by 11.92% using k-NN, 17.97% using NBC and 16.74% using NNet downstream classifiers. We hypothesize that the lower layers including local encoder (L0) representations contain speech information only from the local window of size 25 ms, and, in addition, passing the representations to the downstream classifiers after applying the statistical pooling layer further restricts it in capturing more stutter specific patterns. In addition, the results show that the contextual layers from L5 to L11 of the Wav2Vec2.0 model trained in a self-supervised fashion are able to capture rich stuttering patterns as also depicted in Figure 7.3. As for the last layer (L12), it slightly degrades performance in SD in comparison to its previous layer due to the fact that the transformer block C was fine-tuned and adapted towards the ASR task. By fine-tuning towards ASR, it is possible that the Wav2Vec2.0 model has not focused on the information which is relevant to stuttering, resulting in the loss of rich stuttering 4 L0 is a local encoder, L12 is the last layer of C information. Consider such an example of prosodic information, which is very essential in SD, but not that important for ASR. Using Wav2Vec2.0 embeddings with NNet in SD, there is an overall relative improvement of 61.36%, 47.91%, 14.65%, and 9.02% in repetitions, blocks, interjections, and fluents respectively over the MB StutterNet, thus outperforms over the state-of-the-art results.

Moreover, the prior application of LDA on Wav2Vec2.0 representations further boosts the detection performance in repetitions by 8.29%, prolongation by 10.41%, blocks by 27.24%, and interjections by 8.72%. Fusion: In addition, we fuse the ECAPA-TDNN and Wav2Vec2.0 embeddings via score and embedding fusion schemes, the results of which can be seen in Table 7. 1 (d, e).

While computing the final score p from ECAPA-TDNN and Wav2Vec2.0 prediction probabilities, we empirically optimize the weighting parameter α on test set in p = α * p w2v2 + (1-α) * p ecapa and we found α = 0.9 gives the best results. The ECAPA-TDNN representations which contain rich information about speakers' identity further enhances the overall detection performance of 2.1% using k-NN, 0.65% using NBC, and 1% using NNet. However, it doesn't contain enough rich information about suprasegmental, emotional content, etc., which are incredibly important for disfluent classes, thus acting as a negative transfer for some of them. Moreover, concatenating the ECAPA-TDNN and Wav2Vec2.0 speech embeddings results in a further relative improvement of 8.66% in UAR using NNet. We found that combining speaker embeddings with contextual Wav2Vec2.0 embeddings acts as a negative transfer for most of the disfluent classes except fluent category as shown in Table 7. 1 (d, e). The stuttering characteristics vary from person to person and speaker embeddings in ECAPA-TDNN are trained in a manner to capture unique speaker attributes without focusing or giving much attention to other meta-data such as linguistic content and prosodic information. Usually, the Wav2Vec2.0 model captures acoustic information in the initial layers of contextual block, while the middle layers are more suitable for semantic, phonetic and class-related information [START_REF] Mohamed | Self-supervised speech representation learning: A review[END_REF][START_REF] Baevski | Unsupervised speech recognition[END_REF]. Based on the results from Table 7.1 (f) and Figure 7.4 (d) where the middle layers show more F 1 score for stuttering detection, it seems reasonable to hypothesize that the stuttering is mostly related to phonetic and semantic information than the acoustic information which are captured in initial layers of Wav2ec2.0 contextual embeddings. An interesting question might arise "finding the impact of finetuning Wav2Vec2.0 model in stuttering detection domain in the same language and cross language settings i.e., Is stuttering uniform across cross lingual settings or it does vary with the language"?. This is a very interesting research question and it needs to be studied and evaluated deeply in collaboration with the speech therapists and phoneticians.

KSoF

The KSoF is a German stuttered dataset collected recently and is comprised of 5597 three-second stuttered speech clips extracted from 214 recordings of 37 PWS with 28 males and 9 females (Bayerl et al., 2022a). This dataset contains a recording from three types of environments including telephonic conversation, read speech, and spontaneous speech. After collecting the stuttered recordings, the samples were downsampled to 16 kHz and were labelled by three annotators (please refer Appendix A). More than one label was assigned to some of the stuttered samples.

Results and discussion

Evaluation

We experiment with different speech embeddings extracted from various layers of a Wav2Vec2.0 pre-trained model. The extracted speech embeddings improve the detection performance of almost all the classes over the baseline. Similar to [START_REF] Schuller | The ACM multimedia 2022 computational paralinguistics challenge: Vocalisations, stuttering, activity, & mosquitos[END_REF], we also found that the word repetitions are the most challenging ones to recognize. Among the various layers, layer five (L5) shows the best performance on UAR with all the downstream classifiers except for NBC, where, speech embeddings from layer nine (L9) perform better. fluency (e.g., speech rate, pauses, etc.,) and pronunciation (e.g., vowels, consonants, syllables, stress, etc.,) features. The trend also shows that the UAR curve decreases for the last layers. This is most likely due to the reason that the Wav2Vec2.0 model is fine-tuned and adapted towards the ASR task similar to SEP-28k dataset as dicusses previously, and it is quite possible that the information such as prosody, stress, and emotion state gets diluted, as also can be confirmed from the study by [START_REF] Shah | What all do audio transformer models hear? probing acoustic representations for language delivery and its structure[END_REF]. We also observe that the improvement gap between MFCC-based systems and Wav2Vec2.0 embedding-based systems is lower than the study carried out above in Section 7.5. The most obvious reason seems the linguistic information between the two datasets. In previous Section 7.5, we used the SEP-28k corpus for the downstream SD task, which is also an English language stuttering dataset that coincides with the language of the LibriSpeech on which the Wav2Vec2.0 model was trained. Contrary to the SEP-28k dataset, there is a possibility of the loss of linguistic information while extracting embeddings of the KSoF German dataset (provided in this ComParE challenge) from the English-based Wav2Vec2.0 pre-trained model.

In addition, we experiment by concatenating the MFCC features to each speech embeddings layer of Wav2Vec2.0 after applying statistical pooling to both the features and using the downstream classifiers k-NN, SVM, NBC, and shallow NNet for SD, the plots of which are shown in Figure 7.7. We observe from the curves that the concatenation of ment of 31.32%, and 33.81% on the validation set, and 1.49%6 and 5.45% on the test set respectively. We make use of the fact that each layer of the Wav2Vec2.0 model has different information and passing the summation over all layers ( i L i ) to MB StutterNet results in a relative improvement of 45.91% on the validation set and 5.69% on the test set in UAR over the challenge baseline (CBL) as shown in Table 7.3.

After analyzing various confusion matrices (one shown in Figure 7.6) from the SD systems mentioned in Table 7.3, we found the trend that the most of the disfluent classes are still being falsely classified as ND (no_disfluencies class), and confusion in garbage is maximum followed by word repetitions, sound repetitions and blocks. This makes intuitive sense because the repetitions contain certain words or phrasal parts that, when examined individually, reveal themselves to be fluent utterances. Consider the phrase, "for for the movement".

The word for is repeated twice, however, it is a fluent component if two fors are examined separately. For garbage, it is hard to understand the reason since it is either unintelligible or contains no speech. Furthermore, we observe that switching from inputting speech embeddings from a single layer to summing information across all layers ( i L i ) reduces the confusion rate of various classes for MB StutterNet.

Summary

The automated stuttering detection task suffers from a lack of unlabeled data and thus limits the application of large deep models. To address this issue, we introduced a selfsupervised learning framework that first learns a feature extractor for a pre-text task on a large amount of audio data and then employs the learned feature extractor for the downstream stuttering detection task with limited audio data. We investigated ECAPA-TDNNbased speaker recognition and Wav2Vec2.0-based speech recognition as two separate pretext tasks trained on VoxCeleb and LibriSpeech, respectively. Our study reveals that contextual embeddings associated with speech recognition tasks are more appropriate for SD than speaker embeddings. We found that Wav2Vec2.0-based contextual embeddings yielded at least 19.17% relative improvement in UAR over the competitive state-of-the-art systems trained only on limited labeled data. We further improved SD performance by combining embedding from different layers of the Wav2Vec2.0 model. We also found that postprocessing the extracted embeddings with LDA improves classification performance. Our benchmarking experiments with three different classifiers for downstream tasks reveal that a simple MLP-based neural network performs best and it opens up opportunities for further advancements.

Even if the proposed self-supervised framework substantially improved the SD performance over baselines, the performance is still relatively low as compared to other speech applications, possibly due to the presence of a mismatch effect in podcast recordings and class imbalance during training. The future work includes further investigations of the proposed approach by compensating for those effects. In our present work, we did not utilize the stuttering labeled dataset for training the pre-text task. This work can also be extended by including this dataset in this stage which may mitigate the audio domain mismatch between pre-text and downstream tasks.

Conclusion and future research perspectives

We conclude this thesis by summarizing the key findings and overall contributions, followed by a list of open challenges in stuttering detection domain. Finally, we give a broader perspective for the practical application of the methods presented in this thesis from the clinical point of view.

Conclusion

The primary purpose of this thesis is to create deep learning based stuttering classification systems that might help speech pathologists and PWS in distinguishing fluent and disfluent speech. Recall the main question of this thesis that we highlighted in Section 1.1: Chapter 6 presents MTL framework motivated by the way humans perceive speech and decode its meta characteristics such as emotional content, speaker traits, etc. In this thesis, we particularly exploit speaker characteristics as an auxiliary task in a MTL SD.

In addition, this chapter also provides an adversarial learning based MTL framework for stuttering detection based on the hypothesis that stuttering is affected by various meta-data contents such as speaker information, recording conditions, age, accent, etc. We investigate speaker information and try to learn speaker invariant stutter features.

In Chapter 7, we employ SSL to stuttering detection domain. We first use pre-trained

Wav2Vec2.0 SSL model as a feature extractor, which is trained on massive amounts of nonstuttering unlabelled data with an assumption to capture different recording conditions, gender, age, linguistic content, paralinguistic information, and so on. These extracted features of stuttering labeled dataset from Wav2Vec2.0 model are then benchmarked with shallow neural network and MB StutterNet. We extensively evaluate our methodology and compared with the MFCC based input baseline features. We found that the features learned using SSL setup are rich enough to capture different types of stuttering and outperformed over the baseline MFCC based features. We also found that the middle layers of Wav2Vec2.0

model are good at capturing information such as paralinguistics, phonetic features, etc., which helps the model to capture more stutter related representations and its types. In addition, we found that the last layers of Wav2Vec2.0 model slightly degrades the stuttering detection performance due to the fact that the model was adapted towards ASR task, and there is a possibility of loss of prosodic, emotional content that are extremely important for stuttering detection.

Even though the proposed models in this thesis have shown great improvement in the stuttering detection domain, there are some limitations associated with these models. The performance of advanced StutterNet is yet to be investigated on the actual clinical stuttering datasets, the collection of which is under process. The issue with the current stuttering datasets is the noise in annotations, which are labeled by phoneticians not by expert speech therapists. So there is a possibility that a model such as StutterNet trained on noisy labels can perform poorly in actual clinical settings. Moreover, our methodology is based on an assumption that each stuttering speech sample is having a dominant unique disfluency in it, however stuttering rarely comes alone. This can cause variations in the outcome predictions of the trained models such as StutterNet and its variants, when fed with such kind of disfleunt speech samples having multiple disfluencies in them. In addition, models trained in one domain such as language, recording conditions, accents, speakers, linguistic content, etc., preform poorly in another domains such as cross corpora settings, etc. Analysing the performance of our methodology in cross corpora settings as discussed in Chapter 5, we observe a drop in the detection performance of stuttering and its types. The current thesis focuses only on the global detection and identification of disfluencies, if embedded in a speech utterance, however, it didn't take into account where actually the stuttering occurs i.e., which frames in particular, in the stuttered speech signal are the disfluent ones.

Based on our initial investigations, we also found that some stuttering types are not clearly discernible in acoustic domain, and the proposed methodology in this thesis is likely to misrepresent those samples, However, those samples were clearly visible by the visual cues (e.g., head nodding, lip tremors, quick eye blinks, and unusual lip shapes) that can act as a complementary information to the acoustic domain, thus opens the door for multi-modal stuttering detection systems.

Future perspectives

The techniques presented in this thesis have a lot of potential for expansion. In this section, we discuss some possible future perspectives.

Overall, this thesis is one of the several potential beginning steps towards developing automatic stuttering detection systems for clinical settings. As discussed above, disfluencies rarely come alone, we can exploit multi-label training schemes by involving expert speech therapists to direct the StutterNet to make robust predictions in classifying the multiple disfluent samples into respective multiple labels. In addition, soft/fuzzy clustering algorithms can also be exploited for multiple disfluent samples. Exploiting speaker information, which we study in Chapter 6 in learning speaker-invariant stutter representations, additional meta-data such as language, linguistic content, age, recording conditions, etc., can also be explored in adversarial MTL setup to make stutter representations more robust and invariant to these meta-contents.

In learning frame-wise stuttering detection, the methodology proposed in this thesis can be adapted like sequence-to-sequence learning paradigm in conjunction with CTC loss to reveal the disfluent type at each frame-level in the stuttered speech utterance. Similar to part of speech tagging, language models such as bidirectional encoder representations from transformers models can also be adapted for frame-level disfluency detection, which later on can be auto-corrected on the fly and can be tuned towards ASR task.

Stuttering can also be characterized as an audio-visual problem. Cues are present both in the visual (e.g., head nodding, lip tremors, quick eye blinks, and unusual lip shapes) as well as in the audio modality. This multimodal learning paradigm could be helpful in learning robust stutter-specific hidden representations across the cross-modality platform, and could also help in building robust automatic stuttering detection systems. Self-supervised learning such as AV-HuBERT, Whisper models can be exploited to capture acoustic stutter-specific representations based on guided video frames. This framework could be helpful in learning stutter-specific features from audio signals guided by visual frames or vice versa [START_REF] Shukla | Visually guided self supervised learning of speech representations[END_REF]. Textual and magnetic resonance imaging modalities can also be added on top to enrich the stuttering latent representations. In addition, functional magnetic resonance imaging modality of stuttering can be used to explore and explain the stutter latent embedding space, which the model is trying to learn from acoustic input feature space.

For privacy preserving, adversarial and federated learning are the tools that can be employed with the models proposed in this thesis in developing privacy preserving stuttering detection systems. In federated learning paradigm, the SD model can be downloaded from the cloud onto the edge devices such as mobile phones and trains the model on each mobile device separately using their local disfluent data only. These locally trained models will then be transmitted from the devices to the main server in encrypted form, where they will be integrated, by weight averaging, etc,. and a single consolidated and enhanced global model will be delivered back to the devices without compromising their privacy.

Different studies have used different evaluation metrics to test the performance of their SD systems. Accuracy, F1-score, UAR, etc., have been used widely in SD. However, there is no single consistent metric that is being used across the SD community which makes it hard to compare the newly developed SD systems to the existing ones. Moreover the unbiased nature of these metrics doesn't reflect the true evaluation performance of SD methods. There are no consistent data selection (train/val/test) rules and no clear evaluation protocol for evaluating various SD methods which makes it extremely difficult to compare the proposed SD methods with the state-of-the-art SD techniques. Therefore, it is essential to have a unified data selection procedure for comparing the performance of different SD systems, as well as a single consistent evaluation metric, such as macro F1-score, which considers the class count when evaluating the performance of the SD methods.

One more potential work could be the synthesis of stuttered speech using generative Malgré le fait qu'il ait de nombreuses applications potentielles, très peu d'attention a été accordée à la recherche dans le domaine du SD et de la mesure, ce qui est l'objectif principal de cette thèse. La détection et l'identification des épisodes de bégaiement peuvent être un problème difficile et complexe en raison de plusieurs facteurs variables, notamment la langue, le sexe, l'âge, l'accent, le débit de parole, etc. La plupart des travaux antérieurs sur les méthodes de SD sont basés soit sur des systèmes de ASR [START_REF] Alharbi | A lightly supervised approach to detect stuttering in children's speech[END_REF][START_REF] Alharbi | Sequence labeling to detect stuttering events in read speech[END_REF] ou des modèles de langage (Zayats et al., 2016a;[START_REF] Chen | Controllable time-delay transformer for real-time punctuation prediction and disfluency detection[END_REF]. Ces méthodes sont des Introducing self-supervised learning to stuttering detection: En raison de la rareté des données étiquetées, la tâche de détection automatique du bégaiement est limitée dans son utilisation des modèles d'apprentissages profonds de grand taille. Comme indiqué dans Appendix A, plusieurs ensembles de données sur le bégaiement, tels que SEP-28k [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], UCLASS [START_REF] Howell | Automatic recognition of repetitions and prolongations in stuttered speech[END_REF], LibriStutter [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF] et FluencyBank [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], ont été créés au fil du temps pour examiner divers modèles SD. Cependant, l'amélioration est limitée, très probablement en raison de la petite taille et du nombre de données non étiquetées des ensembles sur le bégaiement. Ces ensembles sont incapables d'encapsuler plusieurs facteurs variables tels que le contenu linguistique, le style de parole, le sexe, l'âge, l'accent, le tempo de la parole, les conditions d'enregistrement, etc. Notre contribution à ce défi est que nous avons introduit le tout premier cadre d'apprentissage auto-supervisé au domaine de la détection du bégaiement. Le cadre d'apprentissage auto-supervisé entraîne d'abord un extracteur de caractéristiques pour une tâche de pré-texte en utilisant une grande quantité de données audio non bégayantes non étiquetées pour capturer différents styles de parole, d'accents, de groupes d'âge, de contenu linguistique, de contenu paralinguistique, de prosodie, conditions environnementales, etc. Puis, il applique l'extracteur de caractéristiques appris à une tâche de détection du bégaiement en aval en utilisant des données audio de bégaiement étiquetées limitées.

First

  & foremost, with great admiration, I would like to express my deepest gratitude to jury members, my supervisor Prof. Slim Ouni, & to my collaborators Dr. Md Sahidullah, & Prof. Fabrice Hirsch without whom this piece of work would not have been possible. To be sincere, I am unable to find words to describe my gratitude towards them. To continue this tradition, I try to make an attempt by following the verse "Is there any reward for goodness except goodness?" (Al-Quran 55.60) in thanking my supervisor & collaborators. With great humility & congeniality, they provide me such an excellent, constructive & encouraging environment which consistently pushed my motivation & confidence in this interdisciplinary stuttering detection research domain. Every now & then, they provide unbiased constructive critical reviews & recommendations, which has essentially helped me to improve more & more over the years. They not only trained me from the concerns of migrating to a new nation, but also taught me to be patient, endure, & lead good scientific endeavors, which is the quintessence of the LORIA's & INRIA's work culture. With considerable reverence, I would also like to show my sincere & deepest prayers & gratitude to my sagacious & profound mentors especially Dr. Dawood Ashraf Khan, Mr. Khwaja Mohammad Shafi, & Dr. Ajaz Ahmad Bhat, whose role in my life has been phenomenal & inconceivable since I met them. They not only put me in the right direction, but rather, they molded me like their younger brother by guiding & advising me from time to time & by showing great empathy towards me. Ahh!, how can I forget the beautiful vii viii moment of concern & care Dr. Dawood showed to me when I was going through tough time during the first year of my stay in France & during the first year of my PhD thesis. Recollecting my memorable moments with Mr. Shafi at the time of my admission in France, where he selflessly motivated me to explore the new educational adventures in France by offering financial support. Coming to Dr. Ajaz with whom there are no less memorable moments of my life especially in my first journal publication. Working at Multispeech Inria/Loria, Nancy with such a supportive & pundit group is a once-in-a-lifetime opportunity. I am privileged to work with such a pleasant & courteous cohort. My colleagues at Inria with whom I form close bonds have been my stress relievers & constant companions, without them, I could not have completed my thesis. Special thanks to my lively C137 office gang -Louis Abel (the boss of C137 gang), Ashwin D'sa Geet (South Indian photographer), Michel Olvera (Mexican ping-pong champ), Joris Cosentino (French chess champ & an adorable enemy of the boss), Marie-Anne Lacroix (French computational ornithologist), & Mickaëlla Grondin-Verdon (French keyboard warrior), Nasser-Eddine Monir (Moroccan Tajine expert) and , Sewade Ogun (other peoples's food taster) for making it "the most exciting & amusing office", with white board discussions & games such as wor(l)dle & chess. Thanks to my ping-pong gang Ali Golmakani, Hugo, Vinicius Ribeiro, Prerak Srivastava, Sofiane Azzouz, Louis Delebecque, Hubert Nourtel. I would like to thank our senior scientists as well including Denis Jouvet, Emmanuel Vincent, Mostafa Sadeghi, Antoine Deleforge, Sheikh Imran, Paul Magron, Yves Laprie, Vincent Colotte, Romain Serizel & other colleagues including Nicolas Furnon, Pierre Champion, Théo Biasutto-Lervat, S&ipana Dowerah, Tulika Bose, Can Cui, & Marina Krémé.

1 .

 1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3. Contribution summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4. Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1. Peer-reviewed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.2. Under revision . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5. Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. 1 . 1 .

 11 A diagram of vocal tract with articulators [source: (Ahmed et al., 2022)]. . 23 3.2. Speech waveforms (top), spectrograms (middle), and Mel-frequency cepstral coefficients (MFCC)s (bottom) of a male speaker saying Communication disorder. Left is fluent speech and right is stuttering (repetitions), occurring at the D (between 1.25 second to 2.25 second window) in the disorder utterance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3. Simple perceptron Artificial neural networks (ANNs) with one hidden layer. 33 3.4. Feed forward fully connected neural network. . . . . . . . . . . . . . . . . 34 3.5. Time delay neural network model. The first layers of Time delay neural network (TDNN) model focuses only on the small context of speech frames. Layer 2 takes input as the sliced output of layer 1 which results in capturing a more temporal context at layer 2 with the help of the previous layer's context. The statistical pooling layer computes and concatenates the mean and standard deviation by aggregating all T output frames at the end of fifth layer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.6. General adversarial learning of neural network classifiers. The network includes a feature encoder E (yellow), with primary classifier in light blue, and domain classifier in red. The black and right red arrows show forward propagation of gradients in the network. All other backward arrows represent backward propagation of gradients with gradient reversal layer (GRL) scales the gradients by reversing using the coeffient λ. . . . . . . . . . . . 37 3.7. Block diagram of Wav2Vec2.0 SSL architecture. We extract embeddings from the contextual block which is trained in a self-supervised fashion using quantisation module (Baevski et al., 2020). . . . . . . . . . . . . StutterNet with layer based context-wise computation. The first five layers of StutterNet focus on the small context of speech frames. For example, layer 2 takes input as the sliced output of layer 1 at time frames of {t-2 , t, t+2}, which results in capturing a total temporal context of 9 with the help of the previous layer's context of [t -2, t + 2]. Similarly, layer 3 sees a total context of 15 time frames. The Statistical pooling (SP) layer computes and concatenates the mean and standard deviation by aggregating all T output frames at the end of fifth layer. The SP layer accumulates the information across the temporal dimension which makes it suitable for subsequent layers to operate on the entire temporal speech segment. . . . . . . . . . . . 43 4.2. Mathews correlation coefficient (MCC) and accuracy of StutterNet with varying context, layer size and mel filter bank size (Acc is normalized in [0,1]). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.3. t-SNE visualization of the output of last fully-connected layer for ResNet + BiLSTM (left) and StutterNet (right). . . . . . . . . . . . . . . . . . . . . . 48 5.1. Stuttering data distribution in SEP-28k dataset. In this thesis, we considered only those samples with single stuttering label. . . . . . . . . . . . . 52 5.2. Repetition stuttering with utterance "said that, that" and the effect of various data augmentations. From 0 to 0.25 seconds, the speaker is saying "said", then followed by two repetitions "that". . . . . . . . . . . . . . . . . 55 5.3. A schematic diagram of multi-contextual StutterNet, which is a multi-class classifier that exploits different variable contexts of (5, 9) in SD. The Flu-entBranch and DisfluentBranch are composed of 3 fully connected layers followed by a softmax layer for prediction of different stuttering classes, CB: Context block, SP: Statistical pooling layer, TDNN: Time delay neural network layer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.4. Class-wise loss values for different probabilities of ground truth classes. Here block, fluent, repetition, prolongation, and interjection classes arecorrespondingly denoted by B, F, R, P, and I. The standard cross entropy (CE) is shown by solid curves and its weighted version is shown by dashed ones (WX represents the weighted loss of class X). . . . . . . . . . . . . . 64 List of figures xxiii 5.5. Confusion matrices showing the repetitions and blocks are mostly confused with fluent speech utterances. F: Fluent, R: Repetition, B: Block, P: Prolongation, In: Interjection, BL: Baseline, WCE: Weighted cross-entropy, MB: Multi-branch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.6. Impact of data augmentation (A4) in cross corpora FluencyBank dataset with MC StutterNet (R: Repetition, P: Prolongation, B: Block, In: Interjection, F: Fluent, XA: X is Disfluency Class and A is accuracy, MC: Multi contextual StutterNet, SC: Same corpora, CC: Cross corpora, A4: Augmentation). The bar plot clearly shows that the model MC StutterNet trained on clean SEP-28k dataset fails to generalize on FluencyBank cross corpora data. Applying data augmentation improves the stuttering detection in cross domain corpora as shown by navy blue bars (1 st bar is clean training with SC, 2 nd is augmented training with SC, 3 rd is clean training with CC, 4 th is augmented training with CC). . . . . . . . . . . . . . . . . . . . . . 69 6.1. Stuttering detection using MTL with light blue curve indicating secondary task which is speaker/podcast classification" CE: Cross entropy, θ's are the parameters of each module, X: MFCCs input is R 20×T dimensioanl features, Z: encoder output = E(X). . . . . . . . . . . . . . . . . . . . . . . 75 6.2. Robust stuttering detection using adversarial MTL. Black arrows show forward propagation of primary task (stuttering detection). The top arrows including light blue, light yellow, and light green show backward propagation of primary classifier. The bottom red arrows show forward propagation in speaker classifier with bold ones are backward propagation, and GRL block indicates gradient reveresal CE: Cross entropy, θ's are the parameters of each module, X: MFCC input R 20×T , Z: encoder output, E(X). The primary classifier contains two sub-branches including Fluent-Branch (F) and DisfluentBranch (D). . . . . . . . . . . . . . . . . . . . . . 77 6.3. Confusion in disfluency pairs in baseline (BL), MTL and Adversarial learning (ADV). XasY refers class X identified as class Y. For better visualization, we did not show in percent. . . . . . . . . . . . . . . . . . . . . . . . . . . 80 6.4. Effect of lambda in MTL (left) and ADV (right) settings in SD with RA: Repetition accuracy, BA: Block accuracy, PA: Prolongation accuracy, InA: Interjection accuracy, SA: Average stutter accuracy, FA: Fluent accuracy, , AnyDisf: Any disfluent accuracy from two class FluentBranch. . . . . . . . 81 6.5. Effect of lambda in MTL (left) and ADV (right) settings with overall macro F1 score in SD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.6. t-SNE embeddings from the shared encoder MTL (left) and ADV (right). We selected four podcasts/speakers (type) randomly for visualisation purposes. Clear speaker clusters are visible in MTL framework where the adverarial MTL is trying to learn speaker independent stutter latent representations. The labels represent the speakers with HeStutters is a male speaker and WomeneWhoStutter is a female speaker. . . . . . . . . . . . . . . . . . 83 7.1. ECAPA-TDNN block diagram, k is kernel size, d is dilation, C is channel, T is temporal dimension, FC is fully connected, BN is batch normalization, AAM is additive angular margin, input is 80-dim MFCC features, SE-Res2Block is squeeze-excitation block with variant of residual (Res2) dilated convolution in between two one-dimensional convolution layers (Desplanques et al., 2020). . . . . . . . . . . . . . . . . . . . . . . . 87 7.2. Block diagram of Wav2Vec2.0 SSL architecture. We extract embeddings (shown in purple) from the contextual block which is trained in a selfsupervised fashion using quantisation module with X is input raw signal. . 88 7.3. t-SNE embeddings from the Fully connected (FC) layer of Emphasized channel attention, propagation and aggregation TDNN (ECAPA-TDNN) (left) and layer L10 of the contextual block C of Wav2Vec2.0 (right). The Wav2Vec2.0 embeddings show clear visible clusters among disfluent classes. 92 7.4. Impact of various Wav2Vec2.0 contextual layers in stuttering detection with L0 is local encoder layer, L1 → L12 are contextual layers, RA is repetition accuracy, PA is prolongation accuracy, BA is block accuracy, InA is interjection accuracy, FA is fluent accuracy, and TA is total accuracy. . . . 94 List of figures xxv 7.5. Block diagram of the proposed pipeline for stuttering detection on KSoF ComParE stuttering challenge (NNet: Shallow neural network, SVM: Support vector machine, NBC: Naive Bayes classifier, KNN: k-nearest neighbor, MB: Multi branch). For shallow NNet, SVM, NBC, and KNN, the input is speech embeddings only extracted from Wav2Vec2.0. . . . . . . . 99 7.6. Accuracy confusion matrix of Support vector machines (SVM) on Val. set with L5 Embeddings (G: Garbage, Fi: Fillers, P; Prolongation, SR: SoundRepetition, B: Block, M: Modified, WR: Word Repetition, ND: no_disfluencies, TA; Total accuracy) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 7.7. Comparison of various Wav2Vec2.0 contextual layers and its concatenation with MFCCs in SD with k-nearest neighbor (k-NN), Naive Bayes classifier (NBC), SVM, and shallow Shallow three layered fully connected neural network (NNet) slassifiers where Y-axis represents Unweighted average recall (UAR):Unweighted average recall, and X-axis represents Wav2Vec2.0 layer embeddings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 List of tables 4.1. StutterNet architecture. Except statistical pooling layer, each layer is followed by a ReLU activation function and a batch normalization (T C: Total context, TDNN: Time delay neural network layer, FC: Fully connected layer, T : Temporal dimensional of speech utterance, t: Current time frame). 44 4.2. Results in recall on UCLASS dataset (B: Block, F: Fluent, R: Repetition, P: Prolongation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.3. Results in precision on UCLASS dataset (B: Block, F: Fluent, R: Repetition, P: Prolongation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.4. F1-score on UCLASS dataset (B: Block, F: Fluent, R: Repetition, P: Prolongation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.5. Results in accuracies and F 1 score on UCLASS dataset (B: Block, F: Fluent, Rept: Repetition, P: Prolongation. F 1 : Macro F1 score). . . . . . . . . . . 47 4.6. Results in accuracies and macro F 1 score on SEP-28k dataset (B: Block, F: Fluent, R: Repetition, P: Prolongation, In: Interjection, F 1 : Macro F1 score). 50 5.1. Results using class imbalance learning (B: Block, F: Fluent, R: Repetition, P: Prolongation, In: Interjection, TA: Total Accuracy, F 1 : Macro F1-Score), BLX: Baseline and X is Number, F M F CC : MFCC input features, F F 0 : 3-Dim (Pitch, Pitch-delta, voicing) features, F phone : Phoneme peatures, MB: Multi branch, WCE: Weighted cross entropy, M f rz enc :Freezing encoder, M f rz enc,disf : Freezing encoder and DisfluentBranch. . . . . . . Results using data augmentation (B: Block, F: Fluent, R: Repetition, P: Prolongation, In: Interjection, F 1 : Macro F1-Score), A4: Babble + Reverberation + Music + Noise augmentation, MB: Multi branch, WCE: Weighted cross entropy, M f rz enc :Freezing encoder, M f rz enc,disf : Freezing encoder and Dis-fluentBranch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.3. Results using MC StutterNet with clean and data augmentation. The MC StutterNet also contains MB training. (B: Block, F: Fluent, R: Repetition, P: Prolongation, In: Interjection, TA: Total accuracy, Bb: Babble, Rv: Reverberation, Mu: Music, No: Noise, A4: Bb + Mu + No + Rv augmentation) . . 69 5.4. Results on cross corpora FluencyBank. The first row is from Table 5.3 which show the results on the same corpora SEP-28k dataset. The last two rows show the results, where the model is trained on SEP-28K and tested on FluencyBank in cross corpora setting. (B: Block, F: Fluent, R: Repetition, P: Prolongation, In: Interjection, F 1 (%): Macro F1 score), Bb: Babble, Rv: Reverberation, Mu: Music, No: Noise, A4: All four Bb + Rv + Mu + No augmentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.5. Results on simulated LibriStutter dataset with RA: Repetition accuracy, PA: prolongation accuracy, TA: Total accuracy, F 1 (%): Macro F1 score, A4: Babble + Reverberation + Music + Noise Augmentation. . . . . . . . . . . 71 5.6. Summary of macro F 1 score of proposed methods (MB: Multi branch, WCE: Weighted cross entropy, M f rz enc : Freezing encoder, M f rz enc,disf : Freezing encoder and DisfluentBranch, MC: Multi-contextual, A4: All four augmentation (Bb + Mu + No + Rv). . . . . . . . . . . . . . . . . . . . . . . . 72 6.1. Stuttering detection results (SA: Stutter accuracy, TA: Total accuracy, SB: Single branch, MB: Multi branch, F 1 : macro F1 score, BL: baseline, MTL: Multi-task learning, ADV: Adversarial learning ) . . . . . . . . . . . . . . 78 6.2. Stuttering detection results (B: Block, F: Fluent, R: Repetition, P: Prolongation, I: Interjection, MB: Multi branch, BL: Baseline, SB: Single branch). xxix 7.1. SD results on SEP-28k stuttering dataset (TA: Total accuracy, B: Block , F: Fluent , R: Repetition , P: Prolongation , I: Interjection, UAR: Unweighted average recall, F 1 : Macro F1 score, BLs: Baselines, MB: Multi branch) for different methods. The results reported for Wav2Vec2.0 are from L10. . . 95 7.2. Distribution summary of KSoF ComParE stuttering dataset (G: Garbage, Fi: Fillers, P; Prolongation, SR: SoundRepetition, B: Block, M: Modified, WR: WordRepetition, ND: no_disfluencies). . . . . . . . . . . . . . . . . 98 7.3. SD results in accuracy and UAR (G: Garbage, Fi: Fillers, P; Prolongation, SR: SoundRepetition, B: Block, M: Modified, WR: Word Repetition, ND: no_disfluencies (fluent), TA; Total accuracy, V: Validation set, T: Test set, CBL: Challenge baseline, CE: Cross entropy, WCE: Weighted cross entropy, UAR: Unweighted average recall, V: Validation set, T: Test set, DS: Deep spectrum features, L i : Embeddings from layer i, ":" is concatenation, i L i : Summing information over all embedding layers). . . . . . . . . . 102 A.1. Stuttering datasets (audio) (C: Conversational speech, RS: Read speech, SS, Spontaneous speech, Ac: Accessibility, Eng: English, Gr: German, Pb: Public, Pr: Private, Un: Unannotated, M: Male, F: Female) . . . . . . . . . 112 A.2. LibriStutter dataset split with number of train speakers = 40, validation speakers = 5, testing speakers = 5. We randomly selected 80% (40) speakers for training, 10% (5) speakers for validation , and remaining 10% (5) speakers for testing. The numbers describe the count of samples in 10-fold cross validation scheme. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 List of acronyms ADV Adversarial learning ANNs Artificial neural networks ASR Automatic speech recognition BN Batch normalisation CNN Convolutional neural networks ComParE Computational Paralinguistics ChallengE CTC Connectionist temporal classification ECAPA-TDNN Emphasized channel attention, propagation and aggregation TDNN EMA Electromagnetic articulograph ED Emotion detection FC Fully connected GMM Gaussian mixture models HMM Hidden markov models KSoF Kassel state of fluency k-NN k-nearest neighbor LDA Linear discriminant analysis LPCCs Linear prediction cepstral coefficients MB Multi-branch MC Multi-contextual MFCC Mel-frequency cepstral coefficients MLP Multi-layer perceptron MTL Multi-task learning NBC Naive Bayes classifier NNet Shallow three layered fully connected neural network PWS Persons who stutter xxxi 1. Introduction

  Stuttering can be expressed in multiple modalities including text, audio, visual, and bio-signals, we mainly focus on developing acoustic modality based stuttering classification systems by addressing some of the challenges stuttering domain is facing in terms of speaker variabilities, data availability, class imbalance, etc., when approached from deep learning paradigm. This PhD manuscript presents my contribution in proposing a new model called StutterNet, its variants, and their extensive evaluation on stuttering SEP-28k dataset curated recently. This thesis also presents my contribution about the adoption of self-supervised learning (SSL) framework in stuttering detection domain by exploiting Wav2Vec2.0 model trained on large amounts of non-labelled data. StutterNet: The first main contribution of this thesis is the development of StutterNet, a time delay neural network based stuttering detection system, which is a multi-class SD classifier capable of detecting several stuttering types using one single system on a large set of speakers, unlike state-of-the-art SD systems which are mostly binary SD classification systems, and only considered few stuttered speakers. The proposed StutterNet first computes the Mel-frequency cepstral coefficient (MFCC) features from raw audio samples, which are then passed to the TDNN to learn and capture the temporal context of various types of disfluencies/stuttering classes. The proposed StutterNet has been evaluated on University college London's archive of stuttered speech (UCLASS) and SEP-28k datasets. Moreover, we, also consider fluent speech samples of PWS that greatly effects the stuttering classification systems unlike state-of-the-art SD systems, where they focus mostly within the stuttering classes. Multi-branch (MB) StutterNet: Due to inherent nature of stuttering, obtaining class-balanced stuttering datasets is both extremely challenging and expensive. Deep neural network classifiers are typically biased towards the majority class when trained on highly class-imbalanced datasets, which leads to subpar modeling of minority classes. Moreover, the stuttering data is very limited, and, is collected usually in clean clinical environments, thus limits the adoption of advanced deep learning methods to stuttering detection, and SD can further be greatly effected by noisy environments. To contribute to the previously mentioned issues, we proposed MB StutterNet (addressing class-imbalance), which comprises a base encoder (E) followed by two parallel branches referred as DisfluentBranch (D) and FluentBranch (F). The embeddings generated by (E) are simultaneously passed to both the FluentBranch and DisfluentBranch, where the FluentBranch is trained to distinguish between fluent and disfluent samples, and the DisfluentBranch is trained to differentiate and classify the disfluent sub categories. Moreover, we also contributed by providing extensive analysis on employing class-balanced training by applying more weights to minority stuttered classes in the loss function, which in turn greatly improves their detection performance.

Figure 3

 3 Figure 3.1.: A diagram of vocal tract with articulators [source: (Ahmed et al., 2022)].

  Figure 3.2.: Speech waveforms (top), spectrograms (middle), and MFCCs (bottom) of a male speaker saying Communication disorder. Left is fluent speech and right is stuttering (repetitions), occurring at the D (between 1.25 second to 2.25 second window) in the disorder utterance .

1 Figure 3

 13 Figure 3.3.: Simple perceptron ANNs with one hidden layer.

  Figure 3.4.: Feed forward fully connected neural network.

Figure 3

 3 Figure 3.5.: Time delay neural network model. The first layers of TDNN model focuses only on the small context of speech frames. Layer 2 takes input as the sliced output of layer 1 which results in capturing a more temporal context at layer 2 with the help of the previous layer's context. The statistical pooling layer computes and concatenates the mean and standard deviation by aggregating all T output frames at the end of fifth layer.

  Figure 3.6.: General adversarial learning of neural network classifiers. The network includes a feature encoder E (yellow), with primary classifier in light blue, and domain classifier in red. The black and right red arrows show forward propagation of gradients in the network.All other backward arrows represent backward propagation of gradients with gradient reversal layer (GRL) scales the gradients by reversing using the coeffient λ.

Figure 3

 3 Figure3.7.: Block diagram of Wav2Vec2.0 SSL architecture. We extract embeddings from the contextual block which is trained in a self-supervised fashion using quantisation module[START_REF] Baevski | Wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF].

  no generally accepted technique can be used. The traditional classifiers that were explored in stuttering identification include SVM, Hidden markov models (HMM), Multi-layer perceptron (MLP), NBC, Gaussian mixture models (GMM), and k-NN.Most of the SD existing methods employ spectral features including MFCCs and spectrograms (please refer Section 3.2.1) or their variants that capture the stuttering-related information. The earlier studies in neural network based stuttering detection explored shallow architectures.[START_REF] Howell | Automatic recognition of repetitions and prolongations in stuttered speech[END_REF];Howell et al. (1997a,b) employed two separate artificial neural networks for the identification of repetition and prolongation disfluencies. This work used autocorrelation features, envelope parameters, and spectral information input to the neural network. The experiments were conducted with a dataset of 12 speakers. Ravikumar et al. (2009) attempted MLP for the detection of repetition disfluencies. They used MFCCs as input features from 12 disfluent speakers. Szczurowska et al. (2014) employed Kohonen network and MLP for discriminating fluent and disfluent speech. The experiments were conducted with eight speakers. Villegas et al. (2019a) proposed a respiratory-based stuttering classifier. They trained MLP on the respiratory air volume and pulse rate features for the detection of block stuttering. The network is trained on 68 Latin American Spanish speakers and reported promising results with bio-respiratory modality.In recent decades, the application of deep learning have grown tremendously in speech recognition[START_REF] Nassif | Speech recognition using deep neural networks: A systematic review[END_REF], speaker recognition(Latif et al., 2020a), speech synthesis(Ning et al., 2019a), emotion detection[START_REF] Akçay | Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers[END_REF], voice conversion (Sisman et al., 2020), voice disorder detection[START_REF] Verde | Voice disorder identification by using machine learning techniques[END_REF]) including Parkinson's disease detection[START_REF] Almeida | Detecting Parkinson's disease with sustained phonation and speech signals using machine learning techniques[END_REF] and dysarthric speech detection[START_REF] Narendra | Dysarthric speech classification from coded telephone speech using glottal features[END_REF][START_REF] Kodrasi | Temporal envelope and fine structure cues for Dysarthric speech detection using CNNs[END_REF]. Inspired by the human auditory temporal mechanism, Kodrasi (2021) recently proposed a convolutional neural network based dysarthric speech detection method by factoring the speech signal into two discriminative representations including temporal envelope (stress, voicing, and phonetic information) and fine structure (breathiness, pitch, and vowel quality) and reported the state-of-the-art results in dysarthric detection. However, the application of deep learning in SD is limited. Due to the presence of acoustic cues in the stuttered embedded speech, the deep learning models can be used to exploit these acoustic cues in the detection and identification of stuttering events as the stuttering can be reflected in representative feature space as shown in Figure3.2. Kourkounakis et al. (2020) recently proposed residual network in conjunction with bi-directional long short-term memory (ResNet+BiLSTM) based binary deep learning classifiers for the detection of six different types of disfluencies including prolongation, word repetition, sound repetition, phrase repetition, and false starts. They used spectrograms as an input features and reported promising results on a small subset (25 speakers) from the UCLASS dataset. Lea et al. (2021) develop a large stuttering dataset (SEP-28k) and utilized convolutional long short-term memory (ConvLSTM) model for the detection and identification of six different stuttering types which include: blocks, prolongations, sound repetitions, word repetitions, and interjections. The model takes 40-dimensional input MFCCs, 8-dimensional articulatory features, 41-dimensional phoneme feature vector, and three dimensional pitch features. The model was trained using a cross entropy loss function with a batch size of 256. In another study by Jouaiti and Dautenhahn (2022), where they introduced phoneme based BiLSTM for SD by mixing the SEP-28k and UCLASS datasets.

  . B. Johnson 4.1. Introduction People with the stuttering problem face several difficulties in social and professional interactions. Due to the very limited research in the field of stuttering detection SD, the idea is to design a single network that can be used to detect and identify various types of stuttering disfluencies. This chapter 1 is about the automatic detection of stuttering with several important applications. For example, it could facilitate the speech therapist's work, since they have to carry out a manual calculation to evaluate the severity of stuttering; to give a feedback to PWS about their fluency. Most of the classifiers proposed for SD so far are either single binary classifiers (distinguishing between fluent v/s stutter) or multiple binary classifiers (requiring a separate model to train for each disfluency category, thus making it computationally expensive). Moreover, the fluent samples of PWS were not taken into consideration during training

  proposed StutterNet first computes the MFCCs features from audio samples, which are then passed to the TDNN to learn and capture the temporal context of various types of stuttering disfluencies. Usually, the initial layers in a standard deep neural network learn wider contexts when processing a temporal input signal. However in StutterNet network as shown in Table 4.1 and Figure 4.1, the initial layers learn and capture only smaller contexts and the deeper ones compute the activations from a wider context. Thus the deeper/higher layers are able to capture and learn the longer temporal contexts. The network 2 Each audio sample in UCLASS in of 4-seconds long and in SEP-28k, it is only 3-seconds long.

Figure 4 . 1 .:

 41 Figure 4.1.: StutterNet with layer based context-wise computation. The first five layers of StutterNet focus on the small context of speech frames. For example, layer 2 takes input as the sliced output of layer 1 at time frames of {t -2 , t, t + 2}, which results in capturing a total temporal context of 9 with the help of the previous layer's context of [t -2, t + 2].Similarly, layer 3 sees a total context of 15 time frames. The SP layer computes and concatenates the mean and standard deviation by aggregating all T output frames at the end of fifth layer. The SP layer accumulates the information across the temporal dimension which makes it suitable for subsequent layers to operate on the entire temporal speech segment.

Figure 4

 4 Figure 4.2.: Mathews correlation coefficient (MCC) and accuracy of StutterNet with varying context, layer size and mel filter bank size (Acc is normalized in [0,1]).

Figure 4

 4 Figure 4.3.: t-SNE visualization of the output of last fully-connected layer for ResNet + BiLSTM (left) and StutterNet (right).

Figure 4 .

 4 Figure 4.3 shows a visualization of the latent feature embeddings learned by StutterNet using t-SNE projection. Both ResNet+BiLSTM and StutterNet present very good discrimination of the different types of disfluencies. However, the latent feature embeddings learned by StutterNet are more distinctive for fluent and less distinctive for prolongations and accurately capture the stutter-specific information than the state-of-the-art ResNet+BiLSTM method. Interestingly for ResNet+BiLSTM, the fluent and repetition category's embeddings are widely spread and more overlapped with the other classes. The prolongations and blocks are well clustered in BiLSTM+ResNet as compared to StutterNet.

  Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. ". Advancing stuttering detection via data dugmentation class-balanced loss and multi-contextual deep learning". Journal of IEEE Biomedical Informatics, (2022).

Figure 5 . 1 .

 51 Figure 5.1.: Stuttering data distribution in SEP-28k dataset. In this thesis, we considered only those samples with single stuttering label.

  , and stuttering is no different as shown in Figure 5.1. In fact, stuttering is extremely imbalanced across fluent and other speech disfluencies. The Figure 5.1. shows that the interjections are the most common disfluency present in the SEP-28k dataset followed by repetitions, blocks, and prolongations and the overall distribution is approximately equivalent to the fluent distribution. Collecting a balanced dataset is difficult and expensive for the stuttering detection task. Other datasets such as Kassel State of Fluency (KSoF)[START_REF] Schuller | The ACM multimedia 2022 computational paralinguistics challenge: Vocalisations, stuttering, activity, & mosquitos[END_REF], FluencyBank[START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF]) also suffer from this issue (discussed in the Appendix A).

  Figure5.2.: Repetition stuttering with utterance "said that, that" and the effect of various data augmentations. From 0 to 0.25 seconds, the speaker is saying "said", then followed by two repetitions "that".

  et al. (2021) recently proposed a multi-stream 3 convolutional neural network for robust acoustic modeling. Chiba et al. (2020) recently proposed multi-stream attention-based BiL-STM network for speech emotion recognition. Li et al. (2018) extracted deep features by training multi-stream hierarchical DNN for acoustic event detection. Moreover in Chapter 4, we found that settings like context frame size optimized for one stuttering class are not good for other stuttering types. Exploiting this fact, we also investigate how the multicontextual neural networks will impact classification performance in the speech disorder domain, and in particular stuttering identification. In our preliminary study, we found that the context window improves the identification performance of two types of disfluencies on the UCLASS dataset. As the context frame size increases in the StutterNet, the performance detection of prolongation and repetition also increases, but decreases for fluent speech segments, and almost remains unaffected for a block type of stuttering. The prolongation and repetition last longer than other types of disfluencies. So to address this deficiency, we exploit the variable contexts of MB StutterNet by training the model jointly on different contexts as shown in Figure 5.3. The pseudo-code of MC StutterNet is provided in Algorithm 1.

Figure 5 Algorithm 1

 51 Figure 5.3.: A schematic diagram of multi-contextual StutterNet, which is a multi-class classifier that exploits different variable contexts of (5, 9) in SD. The FluentBranch and Disfluent-Branch are composed of 3 fully connected layers followed by a softmax layer for prediction of different stuttering classes, CB: Context block, SP: Statistical pooling layer, TDNN: Time delay neural network layer.

Figure 5

 5 Figure 5.4.: Class-wise loss values for different probabilities of ground truth classes. Here block, fluent, repetition, prolongation, and interjection classes are correspondingly denoted by B, F, R, P, and I. The standard cross entropy (CE) is shown by solid curves and its weighted version is shown by dashed ones (WX represents the weighted loss of class X).

  Figure 5.4. illustrates the class-wise training loss curves for normal and weighted cross-entropy loss functions. We can observe that for the standard cross entropy loss function, the majority class (i.e., fluents) exhibit higher loss values and it dominates the overall loss. Therefore, during training with backpropagation, the number of updates for fluent class dominates the gradient values that in turn forces the model to focus mainly on correctly classifying/predicting the majority class. Thus, the minority classes including the blocks, prolongations, and repetitions are given less importance during training, which leads to their poor detection performance. Table 5.1 confirms that the detection performance of blocks and repetitions is very poor, as they are mostly predicted as fluent samples due to the class imbalance nature of the problem. In order to increase the detection performance of minority classes, we penalize the majority class by reducing its contribution to loss computation, and vice-versa for minority classes. The loss functions for the weighted cross entropy are shown by dashed curves in Figure 5.4. The figure indicates that applying weights to standard CE loss function proportional to the inverse of class frequency forces the model to give balanced importance to each disfluency class while optimizing the parameters of the network during backpropagation. This, in turn, helps in boosting the gradient updates for minority classes during training, and thus increases their detection performance in both the baselines as shown in

  Figure 5.5.: Confusion matrices showing the repetitions and blocks are mostly confused with fluent speech utterances. F: Fluent, R: Repetition, B: Block, P: Prolongation, In: Interjection, BL: Baseline, WCE: Weighted cross-entropy, MB: Multi-branch.

  the WCE training scheme, the detection performance of minority classes improved remarkably at the cost of fluent accuracy. From the Figure 5.5 (a), we analyze that most of the repetitions and blocks are being classified as fluent speech. Initially, we hypothesize that it is most likely because of the imbalanced nature of the problem. Despite addressing the class imbalance problem in the stuttering domain using WCE and multibranched training, we found that the block and repetition type of stuttering disfluencies are the ones that are still getting confused with the fluent class as depicted in Figure 5.5 (b) and Figure 5.5 (c). This makes intuitive sense because the blocks are closely related with fluent segments having just different initial silence or gasp followed by fluent utterances like fluent speech. The repetitions, on the other hand, contain some word or phrasal repetitions, which are actually fluent utterances, if we carefully analyze their individual parts. Consider an utterance he he is a boy. The word he is being repeated twice but is a fluent part if two he's can be analyzed on an individual basis. 5.4.2.4. Exploiting advantage of WCE and MB StutterNet Since StutterNet W CE and StutterNet M B address the class-imbalance issue differently, we combine them to exploit both of their advantages. We first pre-train the WCE based StutterNet (StutterNet W CE ) and used it as a DisfluentBranch in our multi-branched Stutter-Net. After the pre-training step, we applied freezing in two ways. Firstly, we freeze the parameters of the contextual encoder block basenet only and fine-tune only the two output

Figure 6 . 1 .

 61 Figure 6.1.: Stuttering detection using MTL with light blue curve indicating secondary task which is speaker/podcast classification" CE: Cross entropy, θ's are the parameters of each module, X: MFCCs input is R 20×T dimensioanl features, Z: encoder output = E(X).

FigureFigure 6

 6 Figure 6.2.

Figure 6 Figure 6

 66 Figure 6.3.: Confusion in disfluency pairs in baseline (BL), MTL and ADV. XasY refers class X identified as class Y. For better visualization, we did not show in percent.

Figure 6

 6 Figure 6.6.: t-SNE embeddings from the shared encoder MTL (left) and ADV (right). We selected four podcasts/speakers (type) randomly for visualisation purposes. Clear speaker clusters are visible in MTL framework where the adverarial MTL is trying to learn speaker independent stutter latent representations. The labels represent the speakers with HeStutters is a male speaker and WomeneWhoStutter is a female speaker.

  squeeze and excitation (SE) Res2Blocks, 1D convolution, attentive statistical pooling, and an FC layer. A non-linear ReLU activation and a batch normalisation is applied after each layer in SE-Res2Block. The model is fed with 80 dimensional mean normalized log Melfilterbank energy features and is trained on a large Voxceleb dataset with additive angular margin (AAM)-softmax loss function using Adam optimizer having a cycling learning rate between 1e-8 and 1e-3. In this work, we use ECAPA-TDNN as a feature vector on SEP-28k stutter dataset to exploit it for SD. We extract R 1×192 speaker embedding feature vector from the FC layer of ECAPA-TDNN as shown by the purple block in Figure7.1.

  tual block transformer g : Z → C and quantization block Z → Q as depicted in Figure 7.2[START_REF] Baevski | Wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF]. The feature encoder is comprised of multi-layer 1D convolution blocks followed by Batch normalisation (BN) and Gaussian error linear unit (GELU) activation functions, takes the normalized raw input X and encodes it into local feature representations Z = E(X). These encoded feature representations of size Z T ×768 are then fed to contextual transformer block to learn contextual speech representations C = g(Z).

Wav2Vec2. 0 :

 0 . Given a data set D = (X i , d i ) of N samples with R 1×K (K is 768 for Wav2Vec2.0 and 192 for ECAPA-TDNN) pre-trained features, NBC assumes that the likelihood of class conditional densities is normally distributed byp(e|C = c, µ c , Σ c ) = N (e|µ c , Σ c ) (7.4)where e ∈ X, is extracted pre-trained representation, µ and Σ are class-specific mean vector and covariance matrix respectively. The posterior probability for each target class is computed then by Bayes' formula:p(C = c|e, µ c , Σ c ) = c, µ c , Σ c ) p(C = c) K i=1 p(e|C = i, µ i , Σ c )p(C = i) (7.5) and then the query sample e is classified by taking argmax over the classes. ŷ(e) = argmax p(C = c|e, µ c , Σ c ) (7.6) 7.3.3. Shallow neural network for stuttering detection A NNet with just a few layers can also be applied on top of the pre-trained embeddings extracted from ECAPA-TDNN and Wav2Vec2.0 models. In this work, we use twobranched NNet with FluentBranch differentiating between fluent and stuttered utterances, and DisfluentBranch classifying stuttered speech utterances into several disfluency types. Each branch is composed of three FC layers with each layer followed by ReLU activation and 1D BN functions. A dropout of 0.2 is applied for first two FC layers. A softmax layer is used in the end to get the desired classes. Following the approach in (Lea et al., 2021), for FluentBranch, we employ a pseudo labelling scheme, where we re-label all different disfluent speech samples as one class and train the binary FluentBranch branch to differentiate between fluent and stuttered utterances. For the multi-class DisfluentBranch branch, we train it by penalizing the fluent class with zero loss. During evaluation phase, the outcome from FluentBranch is considered if the prediction is fluent class, otherwise predictions from DisfluentBranch are taken into consideration. 7.4. Experimental setup 7.4.1. Embedding extractors ECAPA-TDNN: For each three-second speech utterance of the SEP-28k dataset, we extract speaker embeddings of dimension R 1×192 after the FC layer of ECAPA-TDNN as shown by the purple line in Figure 7.1, resulting in R N ×192 data (N is total samples). We use SpeechBrain toolkit (Ravanelli et al., 2021) for the extraction of 192-dimensional speaker embeddings. In addition, we also use Linear discriminant analysis (LDA) for dimensionality reduction with the component size of four resulting in R N ×4 dimensional data before passing it to the downstream classifiers. For contextual embedding extraction of each SEP-28k sample, we extract R T ×768 (T is a temporal dimension) dimensional representations from local feature encoder Z and from each layer of contextual transformer block C, each of which yields differ-ent contextual representation as shown by purple block in Figure 7.2. Before passing each layer representation separately to the downstream classifiers, we apply statistical pooling across the temporal domain and concatenated the mean and standard deviation resulting in a feature vector of R 1×768×2 . Moreover, we also concatenate the contextual embeddings of the local encoder (L0), L6, and layer L10 of C after applying the statistical pooling and LDA (with component size of four) which results in R N ×12 feature vector. We extract embeddings using the PyTorch version of Wav2Vec2.0.

Figure 7

 7 Figure 7.3.: t-SNE embeddings from the FC layer of ECAPA-TDNN (left) and layer L10 of the contextual block C of Wav2Vec2.0 (right). The Wav2Vec2.0 embeddings show clear visible clusters among disfluent classes.

F 1 ,

 1 average UAR and accuracy results of different stuttering and fluent classes on exploiting different features extracted from ECAPA-TDNN and Wav2Vec2.0 models. We compare the proposed speaker and contextual embeddings based SD models to the work ResNet+BiLSTM (Kourkounakis et al., 2020), StutterNet (discussed previously in Chapter 4) (For a fair comparison, we trained these on the SEP-28k dataset with MFCC input features) and to the multi-branched StutterNet (discussed previously in Chapter 5) having two different branches with one branch differentiating between fluent and stutter utterances and the other branch distinguishing among different disfluency types. We use LDA for dimensionality reduction of features in each case before passing them to the classifiers for prediction.

  Figure 7.4. shows the impact of different contextual embeddings in the detection of various stuttering classes. The plot shows almost a close trend in all the stuttering class accuracies. The detection accuracy of stuttering classes increases with the layer 4 number.

Figure 7

 7 Figure 7.4.: Impact of various Wav2Vec2.0 contextual layers in stuttering detection with L0 is local encoder layer, L1 → L12 are contextual layers, RA is repetition accuracy, PA is prolongation accuracy, BA is block accuracy, InA is interjection accuracy, FA is fluent accuracy, and TA is total accuracy.

  multi-branched (MB) StutterNet 5 , where we pass the 768 × T -dimensional speech embeddings directly. The block diagram of proposed framework is shown inFigure 7.5.7.6.3. Training detailsWe train the MB StutterNet and a shallow NNet (discussed in section 7.3) with Adam optimizer on a cross entropy loss function (L = Lf + L d , L f : FluentBranch loss, L d : Dis-fluentBranch loss similar to (Equation (7.7)) using a learning rate of 1e-2, with a batch size of 128 over 50 epochs. The shallow NNet is composed of two branches including FluentBranch and DisfluentBranch with each branch having three FC layers. Each FC layer is followed by a ReLU activation function and a 1D-batch normalization. A dropout of 0.3 is applied to the first two FC layers. A patience of seven is applied on a validation loss to stop the training criteria. We train the MB StutterNet with two input features including 20 × T MFCCs and 768 × T speech embeddings extracted from Wav2Vec2.0. For SVM[START_REF] Murphy | Machine Learning: a Probabilistic Perspective[END_REF], we experiment with different kernels including linear, polynomial, radial basis function, and sigmoid, however, in this thesis, we report the results only with the linear kernel. For k-NN[START_REF] Murphy | Machine Learning: a Probabilistic Perspective[END_REF], we choose the value of k = 5 empirically by grid search using the elbow method with p = 2 (Euclidean) distance metric. A statistical pooling (mean and standard deviation) is applied to Wav2Vec2.0 768 × T speech embeddings before pass-5 The MB StutterNet is discussed in Chapter 5.

Figure 7

 7 Figure 7.5.: Block diagram of the proposed pipeline for stuttering detection on KSoF ComParE stuttering challenge (NNet: Shallow neural network, SVM: Support vector machine, NBC: Naive Bayes classifier, KNN: k-nearest neighbor, MB: Multi branch). For shallow NNet, SVM, NBC, and KNN, the input is speech embeddings only extracted from Wav2Vec2.0.

Figure 7

 7 Figure 7.6.: Accuracy confusion matrix of SVM on Val. set with L5 Embeddings (G: Garbage, Fi: Fillers, P; Prolongation, SR: SoundRepetition, B: Block, M: Modified, WR: Word Repetition, ND: no_disfluencies, TA; Total accuracy)

Figure 7

 7 Figure 7.7. shows the impact of various speech layer embeddings in SD with downstream classifiers. As presented by the plots, the first initial and last layers show lower UAR in comparison to the middle layers. We hypothesize that the lower layer speech representations contain information only from smaller contexts, and passing this information after statistical pooling to SVM, k-NN, NBC, and NNet further inhibits it in learning stutterspecific patterns. Furthermore,[START_REF] Shah | What all do audio transformer models hear? probing acoustic representations for language delivery and its structure[END_REF] have also shown that the middle layers are good at capturing fluency and pronunciation features which are extremely important for stuttering detection, while initial layers of Wav2Vec2.0 exhibit more deviation in learning

  scheme. Inspired by the fact that the number of fluent class samples is almost equal to the total number of samples in disfluent classes as described in Figure 5.1, we simultaneously classify fluent vs disfluent classes in one output branch and subcategories of disfluent classes in another output branch. Furthermore, we contribute by offering thorough study on how to use class-balanced training by adding additional weights to minority stuttering classes in the loss function, which considerably enhances their detection performance. In addition, we thoroughly test our models under simulated noisy ambient conditions by applying data augmentation to the stuttered speech. The employment of data augmentation is not straightforward, because most of the data augmentations like time stretch, speed perturbation, etc., alter the underlying structure of the stuttering speech sample completely. Our approach employs techniques such as noise, babble, music, and reverberation reflecting realworld scenarios and doesn't change the underlying structure of the stuttering speech completely in order to further test SD methods in real-world settings. As discussed above, we further employ variable contexts, and presented MC StutterNet (please refer to Figure 5.3) to jointly learn different contexts of stuttered speech utterance, where a context optimised for one disfluent class is not good for another class. The experiments show ≈ 7% relative improvement in macro-F1 score over the vanilla StutterNet.

  ils peuvent ne pas être disponibles en permanence. De plus, les ASR fonctionnent bien pour une parole normale et fluide, mais ils ne parviennent pas à reconnaître les bégaiements[START_REF] Mitra | Analysis and tuning of a voice assistant system for dysfluent speech[END_REF], ce qui ne permet pas aux PWS d'accéder facilement aux assistants virtuels, comme indiqué précédemment. Par conséquent, les systèmes SD automatiques interactifs qui fournissent une évaluation impartiale et cohérente de la parole bégayée sont fortement encouragés. Nous pensons que ces systèmes de détection automatique du bégaiement fourniront également une évaluation juste, cohérente et impartiale de la parole bégayée qui, plus tard, pourra également être adaptée aux assistants numériques vocaux pour les PWS.

  approches en deux étapes qui convertissent d'abord les signaux acoustiques de la parole en leur modalité textuelle correspondante, puis détectent le bégaiement par l'application de modèles de langage. Bien que cette approche en deux étapes basée sur le ASR pour identifier le bégaiement ait donné des résultats prometteurs, la dépendance à l'égard de l'unité ASR la rend coûteuse en calcul et sujette aux erreurs. De plus, l'adaptation à la tâche ASR entraîne la perte possible d'informations relatives au bégaiement telles que les informations prosodiques, émotionnelles et autres informations paralinguistiques. En résumé, Nous essayons de développer des systèmes de détection de bégaiement basés sur l'audio qui capturent avec succès les différentes variabilités des énoncés audio de bégaiement et qui permettent de distinguer entre la parole fluente et diverses catégories de parole disfluente telles que les blocages, les prolongements, les répétitions et les interjections.B.3. Résumé de la contributionL'objectif principal de cette thèse est de développer des systèmes de classification du bégaiement basés sur l'apprentissage profond. Ils pourront être utilisés ultérieurement dans des environnements cliniques pour aider les orthophonistes à analyser les différents types de bégaiement des PWS. Cependant, comme énoncé exprimé dans la Section 1.1, le bégaiement peut être exprimé dans de multiples modalités, y compris le texte, l'audio, le visuel et les biosignaux. Nous nous concentrons principalement sur le développement de systèmes de classification du bégaiement basés sur la modalité acoustique en abordant certains des défis auxquels le domaine du bégaiement est confronté en termes de variabilité des locuteurs, de disponibilité des données, de déséquilibre des données, etc. lorsqu'il est abordé à partir du paradigme de l'apprentissage profond. Ce manuscrit de thèse présente ma contribution en proposant StutterNet, ses variantes et leur évaluation extensive sur le jeu de donnée SEP-28k de bégaiement SEP-28k dataset récemment créé. Cette thèse présente également ma contribution à l'adoption de l'architecture SSL dans le domaine de la détection du bégaiement.StutterNet: La première contribution de cette thèse est le développement de StutterNet.Ce système de détection du bégaiement basé sur un réseau neuronal à retard temporal est un classificateur SD multi-classes capable de détecter plusieurs types de bégaiement à l'aide d'un seul système sur un grand ensemble de locuteurs. Ce contrairement aux systèmes SD de l'état de l'art qui sont pour la plupart des systèmes de classification SD binaires, qui ne prennent en compte que quelques locuteurs bègues. Le réseau proposé StutterNet calcule d'abord les caractéristiques MFCC à partir d'échantillons audio bruts, qui sont ensuite transmises au TDNN pour apprendre et capturer le contexte temporel de divers types de classes de disfluences et bégaiements. StutterNet a été évalué sur les jeux de données UCLASS et SEP-28k. De plus, nous prenons également en compte les échantillons de parole fluente des PWS qui influencent considérablement les systèmes de classification du bégaiement, contrairement aux systèmes SD de l'état de l'art, qui se concentrent principalement sur les classes de bégaiement. MB StutterNet: En raison de la nature inhérente du bégaiement, l'obtention d'ensembles de données équilibrées par classe est à la fois extrêmement difficile et coûteuse. Les classificateurs de réseaux neuronaux profonds sont généralement biaisés en faveur de la classe majoritaire lorsqu'ils sont entraînés sur des ensembles de données fortement déséquilibrées en termes de classe, ce qui entraîne une modélisation médiocre des classes minoritaires. De plus, les données sur le bégaiement sont très limitées et sont généralement recueillies dans des environnements cliniques contrôlés, ce qui limite l'adoption de méthodes avancées d'apprentissage profond pour la détection du bégaiement et peut être grandement affecté par les environnements bruyants. Pour contribuer aux problèmes mentionnés précédemment, nous avons proposé MB StutterNet (traitant le problème des équilibres de classes), qui comprend un encodeur de base (E) suivi de deux branches parallèles appelées Disfluent-Branch (D) et FluentBranch (F). Les representations générées par (E) sont simultanément transmises au FluentBranch et au DisfluentBranch, où le FluentBranch est entraîné à distinguer les échantillons fluents et disfluents et le DisfluentBranch est entraîné à différencier et à classer les sous-catégories disfluentes. De plus, nous avons également contribué en fournissant une analyse approfondie sur l'utilisation d'entraînement à classes équilibrées en appliquant plus de poids aux classes de bégaiement minoritaires dans la fonction de perte, ce qui améliore considérablement leur performance de détection. Pour tester d'avantage nos propositions de StutterNet et de MB StutterNet dans des environnements réels, nous les évaluons de manière approfondie dans des conditions d'environnements bruyants simulées en utilisant des techniques d'augmentation des données telles que le bruit, le babillage, la musique et la réverbération sur les échantillons de parole bégayée. L'avantage d'appliquer l'augmentation des données à la parole bégayée est double. Premièrement, elle augmente la quantité de données dans les bégaiements, ce qui permet d'utiliser des méthodes avancées d'apprentissage profond. Deuxièmement, elle permet de simuler des conditions environnementales bruyantes en temps réel, ce qui rend les systèmes SD proposés robustes à ces facteurs. MC StutterNet: En outre, nous avons également constaté précédemment que le contexte optimisé pour une catégorie n'est pas bon pour une autre classe de bégaiement. En exploitant ce fait, nous proposons un MC StutterNet, qui est une architecture basée sur un réseau neuronal à TDNN multi-contexte de multi-branché SD. L'architecture multicontexte est basée sur la façon dont les humains perçoivent la parole. Dans la cochlée, le signal acoustique d'entrée de la parole est divisé en plusieurs bandes de fréquences afin que les informations de chaque bande puissent être filtrées indépendamment et traitées en parallèle par le cerveau humain. Multi-task and adversarial learning in stuttering detection: Durant ces dernières années, le domaine de la détection du bégaiement basé sur l'apprentissage profond a connu une évolution continue. Pourtant, tous les systèmes actuels se concentrent uniquement sur la méthode d'apprentissage du bégaiement sans tenir compte des tâches auxiliaires. Malheureusement, ce n'est pas de cette manière que nous percevons la parole. Au contraire, nous décodons simultanément la tâche principale avec divers méta-contenus tels que les caractéristiques du locuteur, le contenu linguistique, l'émotion, etc. de manière multitâche. Dans cette optique, nous avons exploité les caractéristiques du locuteur dans la détection du bégaiement. La troisième contribution de cette thèse est d'apprendre et d'exploiter ces caractéristiques du locuteur en combinaison avec la tâche primaire de détection du bégaiement dans un paradigme d'apprentissage multi-tâches. Comme la tâche de détection du bégaiement ne devrait pas dépendre d'un ensemble spécifique de locuteurs, une approche alternative consiste à supposer qu'une représentation acoustique robuste du bégaiement doit être invariante par rapport aux tâches secondaires, en particulier la reconnaissance du locuteur. Nous avons également contribué à cette approche en proposant une méthode d'apprentissage contradictoire MTL, qui apprend des représentations latentes et qui est invariante par rapport au locuteur, mais qui est en même temps discriminantes pour le bégaiement.

  my Rafeeqs Faisal Rashid, Mushtaq Ahmad Wani, Waheed Lone, Ijaz ul Haq Kanth, Rizwan uz Zaman Wani, to my Najiyys Dr. Ahmet Nicolas Cuny, Ahmet Yavuz hoca for their moral & emotional support. I would also like to offer my prayers & thanks to Mr. Nazir Ahmad Wani, Mr. Mohammad Amin Padder, & to my high school teachers Mr. Ghulam Mohammad & Mr. Abdul Aziz. I unequivocally state that it won't be fair in expressing my prayers & gratitude to my Maymoona Akhter (whom I lost in 2014), my neonatal deceased nephew and niece. I also express my sincere thanks to my maternal uncle Abdul Waheed Sheikh, paternal uncles, cousin brothers including Bhat Altaf, Sheikh Ubaid, Sheikh Parvez, Bhat Tahir, Bhat Owais, & to my little cousin princesses Zehra Waheed, Safoora Zahoor, & Adeena Zahoor. Le bégaiement est un trouble de la parole qui est le plus fréquemment observé parmi les troubles de la parole et se traduit par des caractéristique spécifiques. La fastidieuse tâche de détection et d'analyse des schémas de parole des personnes qui bégaiement «Persons who

	Résumé
	Qareen in words.
	Shakeel Ahmad Sheikh

Ultimately, this thesis would be incomplete without expressing prayers, devotion, gratitude & heartfelt thanks to my family members including my parents Mr. Bashir Ahmad Sheikh, Mrs. Haleemah Akhter, my brother Mr. Tareq Ahmad Sheikh, my sister Mrs. Ishrata Bashir, my newphew Mohammad Yahya, my grandmothers Mrs Aisha Jan & Mrs Taja Bano (I lost these two beautiful souls in this PhD journey), my grandfathers Mr. Ghulam Ahmad Sheikh & Mr Ghulam Mohammad Sheikh, my aunts Mrs. Tasleema Akhter, Mrs. Yasmeena Akhter, & deceased Mrs. stutter (PWS)», dans le but de les rectifier, est souvent traitée manuellement par les orthophonistes et est biaisée par leurs croyances subjectives. De plus, les systèmes de reconnaissance automatique de la parole, «Automatic speech recognition (ASR)», ne parviennent pas non plus à reconnaître les bégaiements. Cela empêche les PWS d'accéder à des assistants vocaux tels que Siri, Alexa, etc.

  .3.2. Acoustic and phonetic view . . . . . . . . . . . . . . . . . . . . . 2.4. Conventional stuttering assessment . . . . . . . . . . . . . . . . . . . . . 2.5. Automatic stuttering detection . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3. Background and state of the art 21 3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.2. Speech processing principles and tools . . . . . . . . . . . . . . . . . . . . 22 3.2.1. Fundamentals of speech processing . . . . . . . . . . . . . . . . . 22 3.2.2. Acoustic features . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3. Challenges in stuttering domain . . . . . . . . . . . . . . . . . . . . . . . Speaker and contextual embeddings . . . . . . . . . . . . . . . . . . . . . 86 7.2.1. Speaker embeddings . . . . . . . . . . . . . . . . . . . . . . . . . 86 7.2.2. Wav2Vec2.0 contextual embeddings . . . . . . . . . . . . . . . . 86 7.3. Classifier description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7.3.1. k-nearest neighbourhood . . . . . . . . . . . . . . . . . . . . . . 89 7.3.2. Gaussian back-end . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7.3.3. Shallow neural network for stuttering detection . . . . . . . . . . 90 7.4. Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 7.4.1. Embedding extractors . . . . . . . . . . . . . . . . . . . . . . . . 90 La motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 B.2. Objectifs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 B.3. Résumé de la contribution . . . . . . . . . . . . . . . . . . . . . . . . . . 117

	xvi 2.6. xviii 7.2. Contents Bibliographie	Contents Contents	xix 123
			xv

. . . . . 2. Stuttering problem 2.1. Speech disorders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2. Stuttering speech disorder . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1. Stuttering categories . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2. Impact of stuttering . . . . . . . . . . . . . . . . . . . . . . . . . 2.3. Characteristics of stuttered speech . . . . . . . . . . . . . . . . . . . . . . 2.3.1. Neurogenic view . . . . . . . . . . . . . . . . . . . . . . . . . . . 27.4.2. Dataset description . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.4.3. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.5. Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 7.6. Evaluation on Kassel state of fluency (KSoF) dataset in Computational Paralinguistics ChallengE (ComParE) 2022 stuttering sub-challenge . . . . . 97 7.6.1. KSoF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.6.2. Proposed framework . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.6.3. Training details . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 7.7. Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 7.7.1. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 7.8. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 8. Conclusion and future research perspectives 8.1. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 8.2. Future perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 A. Appendix: Stuttering datasets B. Résumé étendu B.1.

  To further test our proposed StutterNet and MB StutterNet in real-life environments, we extensively evaluate them in simulated noisy environmental conditions by employing data augmentation techniques such as noise, babble, music, and reverberation to the stuttered speech samples. The advantage of applying data augmentation to stuttered speech is twofold. First, it increases the amount of data in stuttered speech, thus makes it suitable for advanced deep learning methods. Second, it can simulate real-time noisy environmental conditions, thus making the proposed SD systems robust to such factors.

Table 4 .

 4 cation scores of stuttering disfluencies. A ReLU nonlinearity activation function and a 1D batch normalization are applied after each layer except the SP layer. The two FC layers are followed by a dropout of 0.3. Consider an input speech sample with T frames. The first five layers of StutterNet focus on the small context of speech frames. For example, layer 2 takes input as the sliced 1.: StutterNet architecture. Except statistical pooling layer, each layer is followed by a ReLU activation function and a batch normalization (T C: Total context, TDNN: Time delay neural network layer, FC: Fully connected layer, T : Temporal dimensional of speech utterance, t: Current time frame).

	Layer	Output layer size Layer context TC
	TDNN 1	512	[t -2, t + 2] 5
	TDNN 2	512	{t-2, t, t+2}	9
	TDNN 3	512	{t-3, t, t+3}	15
	TDNN 4	512	{t}	15
	TDNN 5	3*512	{t}	15
	Statistical Pooling	3 * 512×2	[0, T )	T
	FC1	512	-	T
	FC2	512	-	T
	FC3	NumClasses	-	T

output of layer 1 at time frames of {t-2, t, t+2}, which results in capturing a total temporal context of 9 with the help of the previous layer's context of [t -2, t + 2]. Similarly, layer 3 sees a total context of 15 time frames. The SP layer computes and concatenates the mean and standard deviation by aggregating all T output frames at the end of fifth layer. The SP layer accumulates the information across the temporal dimension which makes it suitable for subsequent layers to operate on the entire temporal speech segment. The StutterNet is trained to classify different types of stuttering including core behaviors and the fluent part of the PWS.

  The original SEP-28k dataset 3 contains 28,177 samples out of which, we used only 23573 segments, among which 3286 are repetitions, 1770 are prolongations, 2103 are blocks and 12419 are fluent segments, and 3995 are interjections. This results in a total of 19.65 hours of data which includes 2.74 hours of repetition, 1.48 hours of prolongation, 1.75 hours of block, 10.35 hours of fluent speech, and 3.34 hours of interjections. After labeling, each 3-sec sliced speech segment is downsampled to 16 kHz. We randomly selected 80% of podcasts (without mixing podcasts) for training, 10% of podcasts for validation, and

  StutterNet for different stuttering recognition are presented in Table4.2, Table 4.3, Table 4.4, and, Table 4.5 where we compare our technique to[START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF]. All the considered disfluencies and the fluent speech are recognized with good scores. As can be seen from the Table4.4, F1-Scores show clearly the good performances of StutterNet method in comparison to residual network and bi-directional long short-term memory (ResNet+BiLSTM). Table4.5 also shows that the StutterNet surpasses the state-of-the-art in most disfluency detection cases, but shows slightly lower performance in prolongation and block detection with an average accuracies of 17.13%, 42.43% in comparison to 23.17%, 53.33% average accuracies of ResNet+BiLSTM respectively. The StutterNet outperforms ResNet+BiLSTM in correctly detecting fluent speech with a difference of 11.63 points (66.63% for the StutterNet and 55.00% for ResNet+BiLSTM).

				Precision
	Method		R	P	B	F
	(Kourkounakis et al., 2020) (baseline) 0.33 0.42 0.43 0.63
	StutterNet		0.36 0.43 0.42 0.59
	StutterNet (Optimized)	0.35 0.31 0.47 0.59
	Table 4.2.: Results in recall on UCLASS dataset (B: Block, F: Fluent, R: Repetition, P: Prolongation).
				Recall
	Method		R	P	B	F
	(Kourkounakis et al., 2020) (baseline) 0.20 0.23 0.53 0.55
	StutterNet		0.28 0.17 0.42 0.67
	StutterNet (Optimized)	0.24 0.13 0.47 0.70
				F1-Score
	Method		R	P	B	F
	(Kourkounakis et al., 2020) (baseline) 0.22 0.28 0.44 0.52
	StutterNet		0.30 0.23 0.42 0.62
	StutterNet (Optimized)	0.27 0.16 0.46 0.63
	Method		Accuracy		Tot. Acc. F 1
		Rept	P	B	F
	Kourkounakis et al. (2020) (baseline) 20.39 23.17 53.33 55.00	46.10	0.36
	StutterNet	27.88 17.13 42.43 66.63	49.26	0.39
	StutterNet (Optimized)	23.98 12.96 47.14 69.69	50.79	0.38
	Table 4.5.: Results in accuracies and F 1 score on UCLASS dataset (B: Block, F: Fluent, Rept: Repe-We separately optimize the StutterNet by varying the filter bank size (10, 30, 40, 50), context window (3, 7, 9, 11) and layer size (64, 128, 256, 1024). We found that context win-tition, P: Prolongation. F 1 : Macro F1 score).

Table 4.3.: Results in precision on UCLASS dataset (B: Block, F: Fluent, R: Repetition, P: Prolongation). Table 4.4.: F1-score on UCLASS dataset (B: Block, F: Fluent, R: Repetition, P: Prolongation).

  the train, validation, and test set splits are not provided. So, we decided to create a publicly available protocol that ensures no overlap of podcasts between train, validation, and test sets 4 . FluencyBank: For cross corpora, we used FluencyBank dataset (for details, please see Appendix A). Out of 4, 144 samples, we only use 3355 samples (ignoring the non-stuttering and multiple samples), among which 542 are repetitions, 222 are prolongations, 254 are blocks and 1584 are fluent segments, and 753 are Interjections. This results in a total of 2.80 hours of data which includes 0.45 hours of repetition, 0.19 hours of prolongation, 0.21 hours of block, 0.63 hours of interjection samples, and 1.32 hours of fluent samples.

		∇θ b
	19:	end for
	20:	Compute CE loss on validation set
	21:	if CE val ≤ CE prev val then
	22:	if patience ≤ 7 then
	23:	Continue training → Go to step 2
	24:	else
	25:	Stop training
	26:	end if
	27:	end if
	28: end for

Table 5 .

 5

	Accuracy

1.: Results using class imbalance learning (B: Block, F: Fluent, R: Repetition, P: Prolongation, In: Interjection, TA: Total Accuracy, F 1 : Macro F1-Score), BLX: Baseline and X is Number, F M F CC : MFCC input features, F F 0 : 3-Dim (Pitch, Pitch-delta, voicing) features, F phone : Phoneme peatures, MB: Multi branch, WCE: Weighted cross entropy, M f rz enc :Freezing encoder, M f rz enc,disf : Freezing encoder and DisfluentBranch.

Table 5

 5

	.1. The StutterNet using the WCE

  WCE with StutterNet and ResNet+BiLSTM, the macro F 1 score drops from 42.84% and 43.12% to 41.02% in StutterNet and 41.02% in ResNet+BiLSTM respectively. By employing multi-branch training, the macro F 1 score drops very slightly from 42.84% to 42.26% in StutterNet but it drops remarkably from 43.12% to 39.20% in ResNet+BiLSTM.

  The embeddings extracted from each context as depicted by CB block are passed Multi contextual StutterNet, SC: Same corpora, CC: Cross corpora, A4: Augmentation). The bar plot clearly shows that the model MC Stut-terNet trained on clean SEP-28k dataset fails to generalize on FluencyBank cross corpora data. Applying data augmentation improves the stuttering detection in cross domain corpora as shown by navy blue bars (1 st bar is clean training with SC, 2 nd is augmented training with SC, 3 rd is clean training with CC, 4 th is augmented training with CC).to a two-layered BiLSTM unit, and then concatenated after applying SP, resulting in a 1 × 2 × (2 × N )-dimensional feature vector (where N is layer size), which is then fed parallelly to two different branches including FluentBranch and DisfluentBranch for class predictions. This results in a relative improvement of 13.40%, 15.67%, 6.41%, and 1.38% in prolongation, block, interjection, and fluent classes over StutterNet M B (clean), thus an improvement of 3.79% in macro F 1 score, however, employing multi-contextual training, we see a drop from 35.26% to 33.36% in repetition accuracy over the StutterNet M B . In comparison to the baseline BL5 (vanilla single branch StutterNet), there is a relative improvement of 22.92%, 14.13%, 181.81%, and 3.27% in repetition, prolongation, block, and interjection classes respectively. Thus a relative improvement of 2.38% in macro F 1 score. The MC Stut-terNet also performs better in macro F 1 score in comparison to state-of-the-art baselines BL1 and BL4. Applying data augmentation on top of the MC StutterNet, we found that except noise augmentation, all the other data augmentation helps in improving the macro F 1

						Accuracy
		Method		R	P	B	In	F	TA F 1 (%)
		MC StutterNet (Clean) 33.36 37.15 08.34 59.63 78.34 59.77 43.86
		MC StutterNet + Bb	32.29 37.11 09.85 66.50 74.54 58.71 44.40
		MC StutterNet + Mu 32.08 36.57 07.98 65.68 77.67 59.96 44.00
		MC StutterNet + No	28.54 32.39 04.75 64.20 80.42 59.99 42.80
		MC StutterNet + Rv	31.98 37.02 05.83 67.63 77.41 60.08 44.20
		MC StutterNet + A4	34.65 39.75 10.12 67.91 77.89 61.72 46.00
	Table 5.3.: Results using MC StutterNet with clean and data augmentation. The MC StutterNet also
		contains MB training. (B: Block, F: Fluent, R: Repetition, P: Prolongation, In: Interjec-
		tion, TA: Total accuracy, Bb: Babble, Rv: Reverberation, Mu: Music, No: Noise, A4: Bb +
		Mu + No + Rv augmentation)
		MC SC:Clean 1
	80	MC SC:A4 2 MC CC:Clean 3
		MC CC:A4 4
	60
	40
	20
	0
		RA	PA		BA	InA		FA	F 1
	Figure 5.6.: Impact of data augmentation (A4) in cross corpora FluencyBank dataset with MC Stut-
		terNet (R: Repetition, P: Prolongation, B: Block, In: Interjection, F: Fluent, XA: X is Dis-
		fluency Class and A is accuracy, MC:

score in comparison MC StutterNet (clean) The noise augmented samples help in improving the detection accuracy of fluent class. For interjection, we found that all the data augmentation helps, and for block, only babble augmentation helps in improving their detection accuracy. We also found applying all four data augmentation in MC StutterNet results in an accuracy improvement in prolongation and repetition classes, however, with individual data augmentation, a drop in their accuracies can be observed in Table 5.3. By applying all four data augmentation in MC StutterNet training, there is an overall improvement of 1.76%, 17.15%, and 134% in repetitions, prolongations, and blocks respectively, which is a 4.48% relative gain in macro F 1 score over the augmented StutterNet M B training.

Table 5

 5

	.4 and Figure 5.6, the F 1 detection performance
	decreases remarkably from 43.86% to 38.92% employing clean training. The repetition and

Table 5 .

 5 4.: Results on cross corpora FluencyBank. The first row is from Table 5.3 which show the results on the same corpora SEP-28k dataset. The last two rows show the results, where the model is trained on SEP-28K and tested on FluencyBank in cross corpora setting. (B: Block, F: Fluent, R: Repetition, P: Prolongation, In: Interjection, F 1 (%): Macro F1 score), Bb: Babble, Rv: Reverberation, Mu: Music, No: Noise, A4: All four Bb + Rv + Mu + No augmentation.

  The experimental results on LibriStutter dataset are shown in Table5.5. Experimenting and evaluating our MC StutterNet model on simulated LibriStutter dataset[START_REF] Kourkounakis | FluentNet: End-to-end detection of stuttered speech disfluencies with deep learning[END_REF] results in a macro F 1 score of ≈ 91%. However, the performance shows a considerable drop in macro F 1 score when evaluated in cross corpora setting in comparison to real stuttering dataset such as FluencyBank. The MC StutterNet trained on SEP-28k dataset shows extremely poor performance when tested on simulated LibriStutter dataset. The simulated datasets such as LibriStutter looks like a choppy speech and doesn't reflect the actual nature and characteristics of real world stuttered speech. So, a model that performs very well on simulated stuttered datasets can not be used in actual settings as such, as the speaker characteristics change to a very high extent in real clinical conditions. Table 5.6.: Summary of macro F 1 score of proposed methods (MB: Multi branch, WCE: Weighted cross entropy, M f rz enc : Freezing encoder, M f rz enc,disf : Freezing encoder and DisfluentBranch, MC: Multi-contextual, A4: All four augmentation (Bb + Mu + No + Rv).

		Method		F 1 (%)
	StutterNet (baseline) (Clean) (Sheikh et al., 2021) 42.84
		Class Imbalance
		StutterNet W CE (Clean)		41.02
		StutterNet M B (Clean)		42.26
		M f rz enc (Clean) M f rz enc,disf (Clean)		44.42 44.80
		Data Augmentation
		StutterNet + A4		45.30
		StutterNet W CE + A4		44.34
		StutterNet M B + A4		44.03
		M f rz enc + A4 M f rz enc,disf + A4		44.06 45.76
		Multi-Contextual
		MC StutterNet (Clean) MC StutterNet + A4		43.86 Accuracy 46.00
	Model	TrainSet	TestSet	RA	PA	FA	TA F 1 (%)
	MC StutterNet	LibriStutter	LibriStutter 93.36 76.26 98.19 96.11 91.0
	MC StutterNet SEP-28k (clean) LibriStutter 0.65 0.48 99.95 77.25 30.0
	MC StutterNet SEP-28k + A4 LibriStutter 1.11	2.39 97.35 75.43 31.0
	MC StutterNet	LibriStutter	SEP-28k	24.24 55.11 60.60 53.21 41.0
	Table 5.5.: Results on simulated LibriStutter dataset with RA: Repetition accuracy, PA: prolongation
	accuracy, TA: Total accuracy, F 1 (%): Macro F1 score, A4: Babble + Reverberation + Music
	+ Noise Augmentation.

Simulated LibriStutter:

Table 6 .

 6 1.: Stuttering detection results (SA: Stutter accuracy, TA: Total accuracy, SB: Single branch, MB: Multi branch, F 1 : macro F1 score, BL: baseline, MTL: Multi-task learning, ADV: Adversarial learning ) data. The distribution of stuttering data is highly imbalanced, among which the majority of the samples (12419) are fluent segments constitutes 10.35 hours, 3995 are interjections constitutes 3.34 hours, 3286 are repetitions constitutes 2.74 hours, 2103 are blocks constitutes 1.75 hours, and 1770 are prolongations constitutes 1.48 hours of data. For our baseline, we randomly selected 80% of podcasts for training, 10% for validation and remaining 10%

	hours of

  . Although deep learning has shown considerable improvement in several areas, this improvement in SD is nevertheless limited, most likely due to the limited size of stuttering

		This chapter is based on the following articles:
		Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "Introducing ECAPA-
		TDNN and Wav2Vec2.0 embeddings to stuttering detection". Under review in International Journal
	1	of Speech Technology, (2022) (under review).
		Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "End-to-end and self-
		supervised learning for ComParE 2022 stuttering sub-challenge". In Proc. 30th ACM International
		Conference on Multimedia, (2022), pp 7104-7108 (peer-reviewed).

  Figure 7.1.: ECAPA-TDNN block diagram, k is kernel size, d is dilation, C is channel, T is temporal dimension, FC is fully connected, BN is batch normalization, AAM is additive angular margin, input is 80-dim MFCC features, SE-Res2Block is squeeze-excitation block with variant of residual (Res2) dilated convolution in between two one-dimensional convolution layers (Desplanques et al., 2020).

  Activity, & Mosquitoes[START_REF] Schuller | The ACM multimedia 2022 computational paralinguistics challenge: Vocalisations, stuttering, activity, & mosquitos[END_REF]. Among these, we only focus on the Stuttering (KSF-C) sub-challenge. In this sub-challenge, the task was to classify the speech segments into one of the eight categories including filler, garbage, prolongation, sound repetition, block, modified, word repetition, and no_disfluency (fluent).

	7.6. Evaluation on KSoF dataset in ComParE
	2022 stuttering sub-challenge
	The ACM Multimedia 2022 ComParE proposes four sub-competitions including Vo-
	calisations, Stuttering,

  MFCCs with layer embeddings only helps slightly for the SVM classifier while for k-NN and NBC, it remains almost unchanged. For shallow NNet, the UAR degrades while concatenating MFCC and speech embeddings. Over the CBL baseline, the SVM with speech embeddings and concatenated (MFCCs+speech embeddings) results in a relative improve-

							Accuracy
	Model	Feature	G	Fi	P	SR	B	M	WR ND	TA UAR(V) UAR(T)
	(Schuller et al., 2022) (CBL)	DS	NA	NA	NA	NA	NA	NA	NA NA	NA	28.1	40.4
	MB StutterNet	MFCCs	21.21 14.71 35.85 18.42 43.27 78.38 0.00 20.27 33.40	29.0	NA
	SVM (Linear)	L5	9.09 69.61 22.64 23.68 32.69 70.27 0.00 67.57 56.93	36.9	41.0
	SVM (Linear)	MFCCs:L5 21.21 59.80 26.42 26.32 39.42 71.35 0.00 56.08 52.34	37.6	42.6
	NBC	L9	45.46 41.18 30.19 26.32 39.42 70.81 0.00 08.56 29.84	32.7	NA 7
	NNet	L5	6.06 59.80 26.42 10.53 0.00 75.14 4.35 81.08 59.17	32.9	NA
	MB StutterNet	L5	15.15 71.57 32.08 15.79 39.42 76.22 0.00 57.43 54.79	38.5	40.3
	MB StutterNet	L5:L9	03.03 81.37 28.30 28.95 50.96 74.60 4.35 54.50 55.40	40.8	42.6
	MB StutterNet	i L i	12.12 78.43 39.62 26.32 58.65 80.00 0.00 32.88 47.86	41.0	42.7
	Table 7.3.: SD results in accuracy and UAR (G: Garbage, Fi: Fillers, P; Prolongation, SR: SoundRep-
	etition, B: Block, M: Modified, WR: Word Repetition, ND: no_disfluencies (fluent), TA;
	Total accuracy, V: Validation set, T: Test set, CBL: Challenge baseline, CE: Cross entropy,
	WCE: Weighted cross entropy, UAR: Unweighted average recall, V: Validation set, T: Test
	set, DS: Deep spectrum features, L

i : Embeddings from layer i, ":" is concatenation, i L i : Summing information over all embedding layers).

Stuttering is also called stammering. In this manuscript, we will use stammering, stuttering or disfluency interchangeably

This chapter is based on the following article: Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "Machine learning for stuttering identification: Review, challenges and future directions". Journal ofNeurocomputing, 514 (2022), pp 385-402 (peer-reviewed).

https://www.nidcd.nih.gov/health/stuttering

We recall that the coarticulation refers to the influence of one speech segment on another, that is, the influence of a phonetic context on a given segment. The phonetic context is more varied in a connected segment speech than it is in isolated speech segments.

The following article forms the basis of this chapter: Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. "StutterNet: Stuttering detection using time delay neural network". In Proc.

29th EUSIPCO, (2021), pp 426-430 (peerreviewed).

For details, please refer to Appendix A

For splitting of the dataset, please refer https://shakeel608.github.io/protocol.pdf

Speaker disjoint split.

multi-stream, multi-scale, multi-resolution are the different names of multi-context.

For splitting of the dataset, please refer https://shakeel608.github.io/protocol.pdf

We are using (768-dimensional) base version of Wav2Vec2.0

For splitting of the dataset, please see protocol https://shakeel608.github.io/protocol.pdf

We were only allowed to submit five UAR results on the test set due to the restriction in this ComParE challenge.

Acknowledgements

And Verily My Success is Only by Allah.

Speech

Feature Encoder

Contextual Block

Contrastive Loss

Embedding Extraction

Figure 7.2.: Block diagram of Wav2Vec2.0 SSL architecture. We extract embeddings (shown in purple) from the contextual block which is trained in a self-supervised fashion using quantisation module with X is input raw signal.

where l is logit, v is v-th codebook entry, g is codebook group, η = -log(-log(u)) with u are uniform samples from U(0, 1), and τ is the temperature which controls the randomness.

Similar to the masked language modelling, the model is pre-trained in a self-supervised fashion using (Equation (7.2)) by randomly masking certain time stamp representation vectors of the feature encoder and the training objective is to reproduce the quantized qt latent speech representation from a set of K+1 distractors including candidate vector q t and K distractors ∈ Q for masked time stamp vectors at the end of contextual transformer block.

The distractors are uniformly sampled from masked frames of the same speech utterance.

where sim(c t , q t ) = c T t qt ∥ct∥∥qt∥ computes the cosine similarity between the quantized vector q t and contextualized transformer vector c t .

The authors have released several pre-trained feature embeddings with dimensions of 768 (base) 2 and 1024 (large) pre-trained on 960 hours of LibriSpeech dataset and then finetuned for ASR using Connectionist temporal classification (CTC) loss function by adding a linear layer on top of the contextual block. The Wav2Vec2.0 showed remarkable improvement in ASR [START_REF] Baevski | Wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF], techniques such as generative adversarial networks and diffusion models that can act as a stutter-specific data augmentation for the models proposed in this thesis. This can be further studied by generating a cross lingual/domain stuttered speech samples that aims to generate the stuttered speech from the same speaker but in another language, or from a different speaker in the same language, or from a different speaker in a different language.

Last but not least, it is really interesting to evaluate and analyse how the methodology presented in this thesis performs in other speech disorders such as apraxia, cluttering, lisping and dysarthria.

A. Appendix: Stuttering datasets

Stuttering datasets

Data is an indispensable component of any deep learning model. Deep learning saves feature engineering costs by automatically generating relevant features, however, requires substantial amounts of annotated data. Most stuttering identification studies so far are based on in-house datasets [START_REF] Howell | The university college London archive of stuttered speech (UCLASS)[END_REF][START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF][START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF][START_REF] Ratner | Fluency bank: A new resource for fluency research and practice[END_REF]Bayerl et al., 2022a) with limited speakers. In the stuttering domain, there is a lack of datasets and several stuttering datasets that have been collected so far are discussed below:

UCLASS

The most common concern in stuttering research is the lack of training data. UCLASS English public dataset (although very small and unlabelled) is the most commonly used amongst the stuttering research community [START_REF] Howell | The university college London archive of stuttered speech (UCLASS)[END_REF]. The UCLASS comes in two releases from the University College London's department of psychology and language sciences. This contains monologues, conversations, and readings with a total audio recording of 457. Some of these recordings contain transcriptions like orthographic, phonetic, and standard ones. Of these, orthographic are the ones that are best suitable for stutter labelling. The UCLASS 1 release 1 contains 139 monologue samples from 81 PWS, aged from 5 to 47 years. The male samples are 121 and the female samples are 18.

LibriStutter

The availability of a small amount of labelled stuttered speech led to synthetic Lib-riStutter's creation [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory[END_REF]. The LibriStutter (English) consists of 50

SEP-28k

The public unlabelled stuttering datasets are too small to build well generalizable SD systems. To address this, [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] recently curated a public version of stuttering events in podcasts (SEP-28k) dataset. This dataset contains total three-second samples of 28,177 extracted from 385 podcasts which contain utterances from PWS conversing with other PWS. The SEP-28k dataset is the first publicly available annotated dataset. The original SEP-28k dataset contains two types of labels: stuttering and non-stuttering labels.

The stuttering labels include blocks, prolongations, repetitions, interjections, and fluent segments, whereas the non-stuttering labels include unintelligible, unsure, no speech, poor audio quality, and music which are not relevant to our study.

FluencyBank

This is a shared database for the study of fluency development which has been developed by Nan Bernstein Ratner (University of Maryland) and Brian MacWhinney (Carnegie Mellon University) [START_REF] Ratner | Fluency bank: A new resource for fluency research and practice[END_REF]. The platform proposes stutter files with transcriptions of adults and children who stutter. The FluencyBank is an interview data of 32 PWS. In this thesis, we use the annotations developed similar to SEP-28k paper from [START_REF] Lea | Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF]. The total samples are 4,144 including stuttering and non-stutering labels.

KSoF

The KSoF is a German stuttered dataset collected recently and is comprised of 5597 three-second stuttered speech clips extracted from 214 recordings of 37 PWS with 28 males and 9 females (Bayerl et al., 2022a). This dataset contains a recording from three types of environments including telephonic conversation, read speech, and spontaneous speech.

After collecting the stuttered recordings, the samples were downsampled to 16 kHz. This dataset also provides meta information in terms of gender, microphone type, and speaker identity. Like SEP-28k, the KSoF dataset also contains two types of labels: stuttering and non-stuttering labels. The stuttering labels contain core behaviors with blocks, prolongations, repetitions, interjections, and fluent segments, while non-stuttering labels include natural pause, unintelligible, unsure, no speech, poor audio quality, and music.

In this manuscript, we used mainly SEP-28k dataset for experimental purposes and FluencyBank for cross corpora evaluation purposes. For StutterNet, we tested it on both UCLASS and SEP-28k datasets.

Smith and Weber, 2017).

Les troubles de la parole font qu'il est difficile pour les PWS de former des paroles correctes et nécessaires et altèrent la capacité d'une personne à produire ces paroles correctes qui leurs permettent de communiquer avec les autres [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF][START_REF] Ward | Stuttering and Cluttering: Frameworks for Understanding and Treatment[END_REF].

Ces troubles peuvent toucher tous les groupes d'âge et prennent généralement la forme d'un bégaiement, d'une apraxie (le cerveau est incapable d'envoyer un message exact aux muscles de la parole), d'une dysarthrie (causée par une faiblesse musculaire), ou encore d'un encombrement (parole trop saccadée ou trop rapide) [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF][START_REF] Ward | Stuttering and Cluttering: Frameworks for Understanding and Treatment[END_REF][START_REF] Duffy | Motor Speech Disorders e-Book: Substrates, Differential Diagnosis, and Management[END_REF]. Parmi ces troubles, le bégaiement ou balbutiement est le plus fréquemment rencontré [START_REF] Minnis | Speech disorders[END_REF]. Le dysfonctionnement des régions sensorimotrices responsables de la production de la parole entraîne le bégaiement sous la forme de comportements principaux comprenant les répétitions, les blocages et les prolongations et son évolution est dirigée par des aspects lexicaux et psychologiques [START_REF] Smith | How stuttering develops: The multifactorial dynamic pathways theory[END_REF][START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Contrairement à d'autres troubles de la parole, le bégaiement peut être traité si une intervention précoce est mise en place pour aider les PWS à atteindre une fluidité normale [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Avec

B.2. Objectifs

Lors des séances de traitement et de thérapie conventionnelles, les orthophonistes ou les audiologues analysent manuellement la parole du patient [START_REF] Nöth | Automatic stuttering recognition using hidden Markov models[END_REF]. Les orthophonistes observent et surveillent les schémas d'élocution des PWS pour les rectifier [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Cette convention de SD est très laborieuse, prend beaucoup de temps et est également influencée par la croyance idiosyncrasique des orthophonistes. Par ailleurs,