Je Remercie

Jean-Marc Vesin

Thomas Hueber

V P Very

GMM Gaussian Mixture Model HMM Hidden Markov Model IUGR IntraUterine Growth Retardation KNN K-Nearest Neighbors algorithm LFB Linear-Filter Bank LPCCs Linear Prediction Cepstral Coefficients MFB Mel-Filter Bank MFCCs Mel-Frequency Cepstral Coefficients MLP Moderate to Late preterm NICU Neonatal Intensive Care Unit NIDCAP Newborn Individualized Developmental Care and Assessment Program PCA Principal Component Analysis PMA Post Menstrual Age PNA Post Natal Age PR-AUC Area Under Precision-Recall Curve

Keywords: Audio-Video segmentation Chapter, Classification for cry detection Chapter

Factors that influence the behavioral pain responses of premature

Résumé en français

Les naissances prématurées sont définies par l'Organisation mondiale de la santé comme -La méthode de segmentation développée s'est inspirée de celle proposée par Orlandi et al. qui repose sur un calcul de l'énergie à court terme suivi d'un seuillage par la méthode d'Otsu [ 6]. Après avoir éliminé les fichiers audio de minutes ne contenant pas de son, deux étapes sont ajoutées à la méthode pour l'améliorer. La première étape est un double filtre fréquentiel, la seconde est une re-segmentation.

Résumé en français III L'évaluation de la méthode de segmentation en comparaison à des annotations manuelles, réalisées sur trois fichiers, a donné de bons résultats. En effet, nous avons montré qu'elle permet une bonne extraction des évènements contenant des pleurs tout en réduisant le nombre de segments audio extraits. Pour aller plus loin, nous avons également proposé d'utiliser les informations de mouvements des nouveau-nés calculés par une autre équipe du LTSI au cours du projet 8]. En nous limitant aux sons apparaissant dans les intervalles détectés comme du mouvement, nous avons montré qu'il était possible de réduire considérablement la quantité de données à traiter tout en conservant des périodes riches en vocalisations. Ces dernières sont également très présentes dans les périodes de présence d'adultes. Cependant, nous avons choisi de les ignorer par souci de quantité et de complexité des données (superposition des voix et de pleurs, beaucoup de bruits liés aux soins, etc). L'évaluation de cette stratégie sur heures d'enregistrements audio réalisés auprès de nourrissons a montré que les nourrissons sont très peu en mouvement ( % du temps) et qu'en ne retenant que les sons issus de ces seules périodes cela permettait de supprimer jusqu'à 8 % des segments initialement extraits.

fi -La classification, après l'étape de segmentation, est nécessaire pour identifier les pleurs parmi les segments sonores extraits. Nous avons choisi d'utiliser une représentation temps-fréquence des pleurs (spectrogrammes) en entrée d'un algorithme de réseau de neurones convolutifs Resnet. La classification est ainsi réalisée en quatre étapes : ) calcul du spectrogramme par transformée de Fourier à l'aide de fenêtres de Hamming successives de . ms et d'un recouvrement de %, ) découpage des spectrogrammes en images de même durée avec un recouvrement de %, ) utilisation du réseau de neurones convolutifs pour la prédiction de la présence de pleurs dans les images, et ) reconstitution des prédictions pour les sons en retenant la prédiction majoritaire sur l'ensemble des images. Grâce à un apprentissage réalisé par transfert, les poids initiaux de modèle ResNet ont été pré-entraînés avec ImageNet puis optimisés à notre tâche, c.-à-d., la classification pleurs vs non-pleurs, en réalisant un nouvel apprentissage. Pour adapter le modèle à nos données, les paramètres de durée des images d'entrées, de profondeur du réseau de neurones ainsi que le taux d'apprentissage initial ont été optimisés. Après une stratégie en deux étapes permettant d'abord de fixer le taux d'apprentissage, une évaluation de plusieurs combinaisons à l'aide d'une validation croisée a permis d'identifier le modèle avec la meilleure précision. Celui-ci correspond à des images d'entrée d'une durée de . s, une architecture ResNet et un taux d'apprentissage initial de -. Après avoir été à nouveau entraîné sur bébés ( sons), le modèle a obtenu de bonnes performances de classification sur un ensemble de trois nouveaux bébés ( 6 sons). Les résultats montrent que 86% des pleurs initialement annotés ont été détectés (sensibilité) et que % des sons classés comme pleurs sont effectivement des pleurs (précision).

-Pour l'estimation du suivi de la fréquence fondamentale F , nous avons proposé une nouvelle méthode de suivi de la fréquence fondamentale des pleurs de nourrissons dans le contexte d'un suivi en temps réel dans les unités de soins intensifs néonatales. Si les méthodes de la littérature fixent généralement la bande de fréquence dans laquelle effectuer le IV Détection et caractérisation des vocalisations chez des nouveau-nés prématurés suivi du F [ , 6, , ], nous avons proposé une étape initiale pour identifier automatiquement cette bande. Une fois calculé, le suivi de la fréquence fondamentale est effectué en utilisant une détection de contour dans le spectrogramme.

Pour valider notre méthode, nous avons comparé nos résultats d'estimation F à ceux calculés par le logiciel BioVoice que nous avons identifié comme le programme de référence pour l'analyse de pleurs de nouveau-nés. En effet, la méthode développée par Manfredi et al. a obtenu de bonnes performances sur des formes mélodiques synthétiques de cris de nouveau-nés : [ , ].

La comparaison qualitative des suivis de la fréquence fondamentale obtenus sur 8 6 pleurs a montré des estimations correctes dans 8 % des cas avec BioVoice et % des cas avec notre méthode.

Finalement, la chaîne automatique de traitement a été déployée sur une base de données de En conclusion, si ce travail de thèse apporte les outils pour l'évaluation de la maturation et des tendances d'évolution des paramètres des pleurs en fonction de l'âge dans un contexte de soin courant en unités de soins intensifs, il n'en reste pas moins que beaucoup d'améliorations sont à apporter. Les perspectives s'inscrivent naturellement dans la dynamique déjà étudiée et auront comme volonté de traiter le plus de données pour confirmer, renforcer les tendances observées et couvrir la plus large période d'hospitalisation possible dans l'objectif d'apprécier les déviations éventuelles liées à des infections ou des pathologies. Ces travaux seront alors le socle des développements futurs afin de progresser vers une solution entièrement automatique pour une nouvelle génération de systèmes non-invasifs de surveillance en temps réel des nouveau-nés prématurés par l'intermédiaire de l'analyse audio.

Pour continuer avec l'équipe, je souhaite transmettre un immense merci à Sandie. Tu as été une mentore exceptionnelle et je te remercie infiniment pour tout le temps et l'énergie que tu m'as accordés. Tu m'as donné confiance en moi et tu m'as toujours soutenue. Merci de ta présence bienveillante à mes côtés. Merci aussi à tous les membres de l'équipe Digi-NewB, je pense particulièrement à Édouard, Raphaël, Thomas, Crysthine et Gustavo, que ce soit pour le côté professionnel ou personnel, j'ai passé de bons moments en vos compagnies. Je remercie ensuite tous les membres de mon bureau, le 8 aura été le centre de discussions intenses en petit ou grand comité, mais il aura surtout toujours été un lieu de partage et de soutien.

Un lieu où je me suis sentie entourée d'amis sincères. Alors, merci Houda pour tes questions et tes compliments sur mes illustrations. Merci Valentin pour les discussions à n'en plus finir sur le deep-learning, tu m'as rassuré quand j'en avais besoin et ça m'a fait beaucoup de bien ! Je vous souhaite à tous les deux beaucoup de courage, il en faudra à certains moments, soyez-en X Détection et caractérisation des vocalisations chez des nouveau-nés prématurés sûr ! Merci ensuite Pablo pour toutes nos conversations où on a refait le monde tard le soir, ce fut des moments suspendus. Une petite parenthèse ici pour Karim, merci pour nos rendez-vous "réguliers" aux restos ou autour d'un verre, je suis contente qu'on ait réussi à garder le contact entre Rennes et Lyon. Ton sourire fait chaud au coeur tout comme les encouragements que tu m'as offerts. Merci Yannick pour ton énergie et ton positivisme permanent avec ta phrase fétiche du : "T'inquiètes, ça va aller !". Merci pour tout le temps passé à m'aider à mettre mon réseau de neurones en place, merci pour toutes tes réponses, ta patience, ta transmission, et tout le courage que tu m'as donné pendant toutes ces années. Enfin, merci Soumaya pour ta bienveillance, ton sourire, toutes les gourmandises que tu m'as données ;) Réaliser cette thèse en ta compagnie m'a rendu la vie bien plus douce à des moments parfois bien compliqués ... Tu as été un phare plein de lumière et de joie me guidant dans les nuits sombres et dans les chemins parfois tortueux que peut prendre une thèse. Je suis sincèrement heureuse d'avoir partagé tous ces moments avec toi et je te souhaite un bonheur éternel ma tendre amie.

Merci à toutes les personnes qui ont annoté des extraits sonores ce qui m'a permis de constituer, ce que je pense être la première grande base de données de pleurs de prématurés. Cette constitution est un atout majeur de ma thèse et je tiens à remercier particulièrement Paul pour les très nombreuses écoutes et ta grosse contribution à ce travail si minutieux. Merci, aussi, de ta présence et ton soutien au laboratoire, tu es un acharné du travail et je te souhaite le meilleur pour la suite, n'oublies pas de vivre un peu pour toi quand même ;)

Je remercie ensuite les différents maîtres sportifs qui m'ont accompagné tout au long de ce périple. Comme l'a dit Juvénal "Mens sana in corpore sano" ! Je sais que sans ces efforts physiques, je n'aurai jamais tenu psychologiquement, les séances m'ont permis de me défouler et de me vider la tête là où nulle autre activité ne me le permettait. Pendant ces années de thèse, je me suis donnée à fond sur les terrains de badminton et j'ai progressé comme jamais auparavant.

Merci Marine pour tes conseils avisés, je n'oublierai pas ces rendez-vous hebdomadaires qui m'ont fait tant de bien ! Merci aussi à Jaff et Hélène de m'avoir fait découvrir l'Aïkido, ai : l'harmonie, ki : Ils m'ont encouragé et m'ont fait beaucoup de bien. Merci pour toutes les fois où l'on a pratiqué ensemble et pour toutes les conversations qui s'en sont suivies. Ta présence dans le bâtiment fut un cadeau plus grand encore que tu ne l'imagines.

Merci à Lucille, avec qui j'ai partagé de chouettes moments à Rennes, que ce soit les boîtes de nuit improvisées à la maison ou les sorties au bar, on a bien rigolé ! Merci d'avoir été présente aux deux grandes étapes de la thèse à savoir le rendu du manuscrit et le repas de soutenance. Je suis contente d'avoir pu profiter de ton expérience et de tes mots pendant tout ce long parcours.

Je te souhaite beaucoup de bonheur pour la suite ! Merci à l'équipe Tout En Vélo qui m'a généreusement offert le café le midi quand je cherchais un peu de lien social. Merci notamment à Corentin pour tous nos échanges très constructifs, sur le café de spécialité, le bon vin et surtout merci de m'avoir offert une oreille attentive dans des moments parfois difficiles. Merci aussi de m'avoir convaincu, malgré toi, de placer ces remerciement après le résumé tel un groupe de Rock demandant à la foule "COMMENT ÇA VA CE SOIR ?" après avoir joué la première chanson. Puisse ton vélo te mener vers de beaux et joyeux chemins !

Je remercie aussi mes parents d'être restés curieux tout au long de ma thèse, de m'avoir posé tant de questions et de m'avoir aidé aussi à me remettre en question ! Merci d'être si présents dans ma vie et de me soutenir dans toutes les situations. Je me sens extrêmement fière et heureuse d'être votre fille. Merci aussi de m'avoir donné la plus chouette des petite soeur qui m'impressionne et qui me donne du courage dès que j'en ai besoin. Il nous en aura fallu du temps, mais je suis comblée par nos échanges, notre complicité et tout le reste. Merci pour ton illustration qui fait la première page de ce manuscrit et merci de tous les compliments dont tu m'abreuves à chacune de nos rencontres. Tu m'as rendue plus forte. Je tiens aussi à te remercier ma très chère Grand-Mère, pour ta présence et ton soutien, mais surtout pour l'amour incommensurable que tu m'offres.

Je mesure la chance de t'avoir dans ma vie et je te remercie pour tout. Merci aussi à vous mon Papé et ma Mamé de m'avoir enseigné l'art de la promenade en forêt ainsi que celui de la couture.

Grâce à vous, j'ai développé une curiosité de tout et j'ai pu créer mes propres vêtements pour ma soutenance de thèse. All newborns are vulnerable, but premature babies are even more fragile because early birth has prevented complete organ development. Thus, these babies have immature functions such as digestive, cardiorespiratory, immunological or neurological and require special care to stay alive.

Therefore, these babies are cared for in Neonatal Intensive Care Units (NICU), where high medical supervision is provided by the medical staff to ensure their optimal development.

Although each infant's development is unique, the journey of a very premature baby begins in an incubator where she/he is usually given various aids such as respiratory intubation, central intravenous infusion or feeding tubes. These invasive devices are removed as the newborn develops and becomes more independent. Unfortunately, infants born very early have a very immature immune system and are therefore more exposed to nosocomial infections from these invasive procedures [ , ].

However, scientific and clinical advances in perinatology and neonatology have improved the chances of survival of preterm infants. In order to detect markers of possible developmental deficits, clinical and ethical demands have emerged regarding the early assessment of these newborns. Hence, the evaluation of the development of the extreme (i.e., born before weeks)

and very preterm (i.e., born before weeks) newborns by the monitoring their unique behavioral communications was proven to be relevant to adapt the care and the caregiving environment [6, ].

In addition, the continuous monitoring of sleep stages, vocal, motor, or facial activities was shown to be relevant for the detection of various neurological disorders [8-]. Thus, nowadays, nurse observations are performed in the presence of the newborn as part of the Newborn Individualized Developmental Care and Assessment Program (NIDCAP) [ ]. However, several limitations hinder the generalization of these procedures since they are very time-consuming and only a small proportion of newborns can benefit from it. Furthermore, although it is performed by specially trained nurses, these observations remain subjective.

In light of this information, it seems obvious that new solutions for monitoring neurobehavioral development could improve the care of newborns. In regard to the already very intrusive care machines, it is important to consider non-invasive monitoring methods.

Among the non-invasive techniques, the use of cameras associated with microphones seems to be one of the most relevant to provide a behavioral characterization close to the observations made by nurses. Indeed, that way, vocal, motion or facial activities can be captured. In addition, their set-up requires no interaction and no contact with the newborn. The analysis of acoustic parameters development of infant cries might offer a non-invasive tool since these characteristics reflect the development and possibly the integrity of the central nervous system. Indeed, crying is a functional expression of basic biological needs, and emotional or psychological conditions and requires a coordinated effort of several brain regions, mainly brainstem and limbic system and is linked to the breath and the lung mechanisms. Thus, acoustic analysis of newborn infant cry appeared to be a good indicator to assess neurophysiological parameters. Moreover, being easy to perform, cheap and completely non-invasive, it can be easily applied in many circumstances.

This thesis was conducted in the context of the European Project Digi-NewB started in March From an audio perspective, this is the first time that such a device has been implemented and so much data has been recorded. This is why the objective of this work was to develop an automatic processing chain for the detection and characterization of premature newborns spontaneous cries recorded in routine care environments. The resulting manuscript is divided in six chapters.

-We review the basic concepts and terms related to prematurity used throughout the manuscript. Next, we present the anatomy and physiological phenomena involved in the production of crying as well as the definitions of the acoustic parameters of crying. Finally, we review the literature of different clinical and methodological studies on the topic.

-We present the neonatal intensive care units where the recordings are performed as well as the usual cares provided to preterm newborns. Then, we describe in more details the Digi-NewB project, in which all subsequent studies presented in the thesis are framed. Finally, we describe the complex noise environment we have to face and prove the interest of our strategy.

-We propose a segmentation step used to separate the useful sound segments containing audio information from the background noise. Originally based on the study proposed by [ ], it was then improved to better process our data. In addition, we propose to use video signal processing to extract only sounds occurring in infant motion periods.

Detection and characterization of vocalizations in preterm newborns

-We propose a recurrent neural network model, which uses sound segment spectrograms to detect whether it contains crying. This method was developed with a population of neonates and sounds that were annotated by SoundAnnoT software that was designed for this.

-We propose a new method for fundamental frequency characterization based on contouring techniques on spectrogram after an automatic frequency band of analysis detection step.

Introduction

[ ] R J. Prevention and treatment of neonatal nosocomial infections. Maternal health, neonatology and perinatology, vol. , -(

).

[6] B T.B. N J.K. Neonatal behavioral assessment scale.

. Cambridge University Press (

).

[ ] P H. The behavioural states of the newborn infant (a review). Brain Research, vol. 6, 8 -(

).

[8] P H.F. Qualitative changes of spontaneous movements in fetus and preterm infant are a marker of neurological dysfunction. Early Human Development, vol. , -

[ ] P H.F., E C., C G., B A.F., F F., S D. An early marker for neurological deficits after perinatal brain lesions. Lancet,vol. ,.

[ ] B A.F., M A., A R.M., H -A M., O A., P H.F. Qualitative assessment of general movements in high-risk preterm infants with chronic lung disease requiring dexamethasone therapy. The Journal of Pediatrics,vol. ,.

[ ] A H., L G., D F., M G., G -G R., B J. Individualized developmental care for the very low-birth-weight preterm infant. medical and neurofunctional effects. JAMA : the journal of the American Medical Association,vol. ,.

Introduction

The purpose of this chapter is to present the background and interest in the analysis of infant crying. After an overview of prematurity in terms of definitions and care in real-life conditions, we give a brief description of the infant's mechanical and neural activities responsible for the production of crying. Next, we define key terms and features used in this work. Then, through a state of the art, we review the clinical analyses of crying in term and preterm newborns, as well as the methods commonly used in acoustic signal processing. Finally, we present our strategy.

. Prematurity

This section provides several definitions that will be used throughout this document. Therefore it is important to introduce these terms first.

Pregnancy is the term used to describe the period in which a fetus develops inside a woman's uterus. From a medical perspective, it is defined as the period measured from the first day of the last normal menstrual period until delivery, it is measured in weeks of amenorrhea. When lasting about weeks, or months, infants are considered full-term (FT). However, pregnancy can be shortened for various reasons. In the case of birth occurring before weeks of gestation, it is defined by the World Health Organization as preterm birth (PT) [ ].

-In this work, we use the standard terminology proposed by the American Academy of Pediatric [ ], which we define in the following and in Figure . . • Gestational Age (GA): the duration (or term) of pregnancy measured from the first day of the last normal menstrual period until the birth date (in weeks of amenorrhea). It is a valuable definition since it proposes a fixed age to refer to and to identify premature babies independently from their current age at the assessment.

• Post-Natal Age (PNA): the duration elapsed since birth (in days, weeks, or months).

• Post-Menstrual Age (PMA): the duration between the first day of the last normal menstrual period and the date of assessment (usually in weeks + days). It can be seen as the summation of the GA and PNA.

• Corrected age: the duration elapsed between the expected date of birth and the date of assessment (in days, weeks, or months). In fact, this term only exists in the case of premature birth, otherwise corrected age and postnatal age are identical.

-According to severity, prematurity is subdivided into three categories. They are defined according to the pregnancy duration (in weeks) and represent respectively , , and 8 % of the total premature births [ ]:

• Extremely Preterm (EP), newborns born before 8 weeks;

• Very Preterm (VP), newborns born between 8 and weeks;

• Moderate to Late Preterm (MLP) newborns born between and weeks;

-A birth can be premature for many causes that can be classified into two main triggers:

• provider-initiated, defined as the induction of labor or elective cesarean due to maternal or fetal indications or other non-medical reasons,

• spontaneous, with spontaneous onset of labor or premature rupture of membranes. The main factors, in that case, are multiple pregnancies, infections, chronic maternal conditions (diabetes, hypertension, anemia, asthma, thyroid disease), nutrition, lifestyle, maternal psychological state, genetics, or even age at pregnancy.

However, in up to half of all cases the cause remains unidentified [ ].

-Premature birth interrupts the newborn's in utero development resulting mainly in the immaturity of four essential organs: the brain and brainstem, the lungs, the digestive tract, and the ductus arteriosus [ ].

Thus, preterm infants are exposed to severe cardiorespiratory events (associating apnea with bradycardia and oxygen desaturation) and to an excessive risk of unexpected sudden infant death. The more severe the prematurity, the greater the risk of health problems or sequelae. For instance, the chances of survival of an extremely preterm newborn vary greatly (i.e., between and %), whereas an infant born after weeks GA has a much better chance of survival (i.e.,

%) [ ].

Chapter • About infant crying

In addition, prematurity sequelae can also have several long-term effects on both physical and neurological developments such as visual, hearing, or learning impairments, cardiovascular or respiratory disorders, and, global developmental delay.

-However, most of these vulnerabilities are resolved with a good maturation (i.e., development during hospitalization). This is why, since the s, infants are cared for in specialized units called Neonatal Intensive Care Units (NICU). There, they can benefit from thermal, respiratory, and nutritional assistance. Tracking physiological conditions in the perinatal period is of utmost importance to provide the appropriate care and clinical setting to each newborn. Therefore, during their complete hospitalization, infants benefit from careful monitoring of their vital and physiological constants [6].

In addition, in some cases, human observations are performed in the presence of the newborn .

About crying

. .

Anatomy and physiology

Contrary to most mammals, newborns remain dependent on adults for a while to eat, move, care, ... This is why babies produce distress signals in order to warn their caregivers. Crying is their primary mode of communication. Shortly after birth, this innate survival mechanism should not be interpreted as a demand for emotional attention but rather for someone to meet their basic needs (absence of caregivers, weariness, colic, fear, fatigue, hunger) [8, ]. The following paragraphs address the crying mechanical and neurological production which are topics explained in more detail in [ ] and [ ].

Mechanical production

Breathing is the first step of crying. During the expiration, air comes out of the lungs and travels through the trachea into the larynx located in the throat (see the anatomy of cry in Figure . ). This organ, composed of the vocal folds and the glottis, is involved in the swallowing, breathing, and voice production functions. Vocal folds are muscular organs composed of two membranes that can be completely relaxed (as for free-breathing), totally blocked, or in an intermediate position (see illustration in Figure . ). In this case, the increase in air velocity in the narrow passage between the folds results in a drop in air pressure causing them to open and close rapidly. This vibration is responsible for phonation and has a fundamental frequency defined as F .

. . About crying Then the sound is shaped by the different areas it crosses. Thus, after having met the aerodigestive crossroads between the airways and the digestive tract at the level of the pharynx, it reaches the sub-glottal or vocal tract area. The latter is divided into the oral and nasal tracts and it is their size and contour that carve the sound to produce resonant frequencies or formants.

Neurological production

Neonatal crying is triggered by internal or external stimulation and is produced by the coordination of several brain regions (see Figure . for part of the brain's anatomy).

While cry initiation has been associated with the limbic system, hypothalamus, and sympathetic arousal, the crying configuration is controlled by the midbrain.

Indeed, the lower brainstem controls the muscles involved in sound production through the network of neurons called the reticular activating system. It is the tension's variation of these various muscles (i.e., larynx, pharynx, chest) that is responsible for the fundamental frequency and crying modes. The brainstem also controls the size and shape of the supraglottal system (upper vocal tract) which carves formant frequencies.

Finally, a cry can occur thanks to the nervous system responsible for processing sensation and controlling movement, action, and response to the environment (see the illustration in Fig-

ure .

). In particular, crying is controlled by the autonomic nervous system which manages the coordination between the vagal innervation and the central nervous system.

Furthermore, modulation of the overall contour of F as well as the amplitude or intensity of the cry reflects autonomic mechanisms. Thus, atypical F patterns, rapid changes, or high variability suggest neural control system instability or cranial vagal nerve complex lesions (carrying information for the parasympathetic system that help to calm the body).

A more exhaustive review of the crying characteristics with the associated biological mechanisms is presented in [ ].

. . About crying . .

Prosodic feature definitions

This section is intended for the definition of terms and crying prosodic features with which the reader is likely to be unfamiliar and for common terms that have a specific definition in the context of this manuscript.

• Cry unit: sound resulting from the passage of air through the vocal folds during a single inspiratory/expiratory cycle.

• Cry: total sound response, which may contain many cry units.

• Fundamental frequency (F ): a physical characteristic of all periodic waveforms. It is measured in hertz (Hz) and refers to the number of times a complex waveform repeats itself in one second.

• Pitch: the hearing subjective tone perception of highness or lowness that depends on the number of vibrations per second produced by the vocal cords. Its unit is the Mel.

• Cry type or mode: identifiable acoustic output an infant can produce, based on the vibration of the vocal cords. There are three expiratory and one inspiratory modes which are illustrated in

Figure . :

The four crying modes illustrated using spectrograms.

8 Chapter • About infant crying -Phonation or basic cry: resulting from periodic vocal fold vibration occurring with a F between -Hz and produced thanks to neural control of muscular tension and airflow.

-Hyperphonation or high-pitched cry: caused by a sudden upward shift with F greater than Hz due to a neural constriction of the vocal tract.

-Dysphonation or turbulent cry: caused by noisy or inharmonic vibration of the vocal folds due to unstable respiratory control. Such cry unit is not periodic.

-Inspiratory phonation: any sound produced during inspiration.

Phonation and hyperphonation cries have additional frequency characteristics related to their periodic acoustic content, we can mention the essential ones:

• Harmonics: multiples of the fundamental frequency. For example, if the fundamental frequency is Hz, then the first harmonic would be Hz, the second Hz, etc.

• Formant frequencies: the resonance frequencies of the vocal tract. Formant frequencies are usually independent of the fundamental frequency and its harmonics. Only the first two formants are typically measured.

• Melody: identifiable variation of the fundamental frequency along with a cry unit. • First formant (F ) between and Hz;

• Second formant (F ) between and Hz. both, for children and full-term or premature newborns (see [6] for a review).

Full-term newborns and infants

Infant cries were studied for the differentiation between normal and pathological cries. For instance, the similarity between the cry of a malnourished infant and the cry of a brain-damaged one suggested that malnutrition might affect the regulatory function of the central nervous system [ ]. Moreover, equivalent results suggested that heavy marijuana use also affects the neurophysiological integrity of the infant [ ].

In [ 6] and [ ], cries of newborns with prenatal and perinatal complications (such as low birth weight, respiratory symptoms, jaundice, apnea, ...) were detected and acoustical properties presented differences when compared to that of healthy newborns. Furthermore, a comparison was performed between normal and high-risk subjects to find possible early signs of autism [ 8].

Differences were seen in the fundamental frequency value, the number and the length of episodes, and in their melody.

Then, cries were evaluated either to discriminate, with facial expressions, behavioral reactions between invasive and non-invasive procedures [ ] or to measure pain after a heel-prick stimulus.

In the latter case, the conclusion was that crying can be used to measure pain in newborn infants only when the cause of crying is known [ ].

Finally, it is worthwhile to notice that most of the previously mentioned studies were based on the analysis of pain-induced cries, which were easier to analyze because no processing to detect them was needed. Therefore, the investigation of infants' spontaneous cries was only recently studied in several contexts, such as profound hearing loss and/or perinatal asphyxia [ -],

early detection of autistic signs [ ], monitoring [ ] or comprehension of vocal development and early communication [ 6].

Chapter • About infant crying

Premature newborns

Characterization of crying episodes in preterm infants was also largely explored either solely or in comparison with full-term newborns where neurophysiological maturity differences were observed as well as a later impact on speech development.

Once again, early studies focused on the analysis of pain-induced cries. Although differences were shown between the cries of premature and full-term infants at the time of birth, it was also shown that as preterm newborns grew, their cries became more like those of full-term infants [ ].

The same kind of conclusion was reached in a pain evaluation study, based on facial expressions and crying, when comparing newborns to -and -month-old infants [ 8].

As for full-term, analysis of spontaneous cries of preterm infants has been less investigated and is recent. The comparison between spontaneous cries of six premature children (three pairs of twins) recorded at different ages showed essential changes in the cries from the 8th-th week of life up to the rd-th week of life, and were interpreted as an intentional articulatory activity [ ].

In a study, Orlandi et al. presented a correlation between central blood oxygenation and the distress occurring when crying [ ]. For a similar decrease in oxygenation levels in both groups, results showed that, after the crying episode, full-term had a faster and more stable recovery time than preterm newborns.

Eventually, effects of gestational age, body size at the recording, and intrauterine growth retardation (IUGR) were investigated in [ ]. Cries were recorded before feeding in both healthy preterm and full-term newborns at term-equivalent ages and showed that shorter gestational age was significantly associated with higher F regardless of the smaller body size at recording or IUGR.

. . Methods for acoustic signal processing

Crying data acquisition and databases

Collecting infant cries is a challenging task since it is difficult to create and implement an audio acquisition protocol as well as to find infants to record since it requires parental consent. In addition, crying analysis requires data annotation to assess the automatic methods. Therefore, in this paragraph, we review several procedures and databases cited in the literature.

So far, most of the cry analysis studies have been conducted on real audio signals recorded in a hospital or at home with microphones placed near infants. While recordings used to be performed occasionally (i.e., to capture crying events one by one), they are now used for long time to record all sound events, this is called monitoring.

Then, these recordings are usually split into small cry/sound signals (a few seconds) that are then annotated by the doctors, nurses, or parents. However, this task is very time-consuming and remains annotator-dependent due to the perceptual aspect and the lack of crying type definitions.

Hence, every author constructed their own specific clinical annotated database.

Moreover, due to resource limitations and the sensitivity of the infant data, which have to be anonymized during the collection process, there are few available databases. A review of those

is proposed in [ ].
To . .

Finally, synthetic signals have also been used, and sometimes compared with a real dataset, to increase the number of processed data [ 6, -].

Audio signal processing

Once the recordings have been made, the signal processing step consists in extracting the cries from the recorded signal. Initially called cry segmentation, because the cries were manually extracted from recordings made in quiet controlled environments, this step was then enhanced as the recordings became more complex. With the development of technologies, the duration of recordings increased, with a consequent increase in the amount of data to be handled. Therefore automated processing methods emerged with the objective to extract the cries from the background noise. They are usually based on energy computation techniques derived from speech processing.

Then, more recent studies investigated spontaneous cries of infants in real-life monitoring context at home or in the NICU. This uncontrolled environment led to new issues, such as cries occurring at unknown times, as well as unpredictable sounds occurring in the recordings (e.g., voices, doors, ...).

To date, two strategies are commonly used to process such data (Figure . ):

• Based on cry segmentation methods, the first strategy extracts in the audio signal all sound events from background noise; then classifiers are used to detect sound segments containing cries [ -6].

• The second strategy only relies on classifiers that run through the windowed signal and detect the windows containing crying [ 6, -].

Crying feature extraction

Feature extraction is the stage to extract discriminative components from audio signals to perform cry analysis. It can be local features extracted from short frame intervals of cry signal or global features computed over the whole cry unit. Due to the high instability of a cry, it is better to use local features to be robust enough to cover variation within the signal. Although the human voice is a subject that has been widely studied, specific cry acoustic and prosodic features need to be defined since infant and adult voice productions differ in terms of energy, intensity, and formants. T -Duration is the most common time feature which has been investigated. It has been derived in several definitions such as cry unit duration [ ] with its mean [ , ], total cry duration (including one or more cry units) [ , , ] and, the ratio of cry duration within audio signal [ , ]. Pauses between cry units have also been examined through similar duration metrics [ , ].

Another commonly used parameter is the latency time, described as the time from known stimulus [ ] or pain stimulus [ , , 6, ] (in case of pain-induced cry) to the first cry. In addition intensity, zero-crossing rate, amplitude, and energy-based features have also been proposed in [ , ].

However, even if time-domain features are easy and straightforward to compute, they are not robust enough to cover the variations within infant cry signals because of their sensitivity to background noises.

F

-On the contrary, frequency-domain features have a strong ability to model the characteristics within infant cry signals. The spectral energy features have been computed through different approaches such as the overall spectral energy of the signal [ 8,8] or the energy only induced by low or high frequencies [ ].

Yet, the most relevant clinical parameter to date is the fundamental frequency (F ) which was investigated in virtually all the mentioned works. Indeed, as explained before, this prosodic feature offers a direct measurement of vocal development since it corresponds to the rate of glottal opening and closing in the vocal tract (see Section . . ). The fundamental frequency is usually studied through statistical parameters calculated on several cries, such as mean [ , ], maximum and minimum [ ], standard deviation [ ] or variation coefficient [ ]. Moreover, resonance frequencies were usually investigated through the first two formants F and F in [ , , ], but some authors also proposed to assess the third one (F ) [ , , ].

Chapter • About infant crying
Moreover, the common and well-known acoustic features Mel-frequency cepstral coefficients (MFCCs) [ 8, 6 -6 ] and Linear Prediction Cepstral Coefficients (LPCCs) [ ] have proven to be efficient to detect cry within the signal [ ]. MFCCs are obtained through a signal projection on the Mel-scale with frequency bands equally spaced inspired by the human auditory system, whereas LPCCs are based on the vocal tract modelization.

T -

-Actually, due to the highly non-stationary cry signal characteristics, it is better to represent the energy contents of a signal in a joint time-frequency domain.

In practice, it means that frequency features are extracted locally (from short frame intervals of the cry signal) and displayed with respect to time.

Thanks to this representation, one can see the melodic shape of a cry which describes the pattern of F as it varies with time (see Figure . ). It is the most common time-varying frequency descriptor and four main melodic shapes were firstly defined in [6 ]: falling, rising, falling-rising (or rising-falling) and flat.

In [6 ], Várallyay reduced these shapes to three fundamental units (i.e., falling, rising, and flat) that were further used as the basis for the definition of melodic shapes. Later, the "complex" shape was introduced in [ , , 6 ] to cover all melodic patterns composed of more than two In addition, several other features were defined to assess variations in F along a cry unit or during a cry event (succession of cry units). To mention a few, there is jitter (cycle-to-cycle variations of F ) [ , 66, 6 ], shift (sudden change in F ) [ , , ] and glide (rapid variations in F ) [ , , 68].

. . State of the art .

Our strategy

In this chapter, we addressed the subject of prematurity as well as the relevance of acoustic monitoring in the infant's maturation assessment. We showed that the coordination between mechanical and neurological systems is necessary for cry production. Moreover, we saw that the crying analysis has been widely reported in the literature in both term and preterm infants.

However, spontaneous crying analysis, especially recorded in NICU, is quite recent and we reported a few studies.

The lack of spontaneous cry analysis is due to several major obstacles related to data. First, we mentioned the sensitivity of human data, where anonymization is crucial, making it difficult to create large databases. Secondly, the long recordings with unpredictable cry onsets led to the use of automated methods. Finally, recording real data in a clinical environment, such as the NICU, remains a real challenge and requires robust signal processing since transitory random noises can occur in the signal (doors, voices, machines, ...).

Although some teams have already proposed methods [ , , 6, , 6 ], to date and to our knowledge, no one has realized a continuous processing chain for long recordings performed in such a noisy environment. Therefore, the thesis's objective is to address this topic by proposing a workflow for automated cry analysis and the evaluation of maturation from long recordings made in the NICU.

Due to the many issues mentioned above, we developed a three-step strategy, detailed step by step in the following chapters, illustrated in Figure . and briefly described hereafter:

. sound segment extraction;

. cry detection among the extracted sound segments;

. fundamental frequency characterization. In order to achieve efficient automatic sound processing, it is essential to have a good knowledge of the acoustic environment. Therefore, in the following chapter, we describe the intensive care given in NICU through a review of the assistance and monitoring generally performed during sick or premature infants' hospitalization. Then, we present the European project Digi-NewB in which the acquisition system and the data collection protocols were designed. Finally, we present the acoustic environment that is valuable to understanding our choice of processing strategy. 8).

[ ] O S., R G C.A., B A., D G., M C.
Application of pattern recognition techniques to the classification of full-term and preterm infant cry. Journal of Voice, vol. , 6 6-66 ( 6).

[ 8] B M.V. S

V. An automatic infant's cry detection using linear frequency cepstrum coefficients (lfcc [ ] N G., K J., V T., T O., P M.J., L J.M. Automatic segmentation of infant cry signals using hidden Markov models. EURASIP Journal on Audio,Speech,and Music Processing,vol. 8,.

Bibliography Introduction

Developing an automated cry analysis processing chain from audio monitoring requires the knowledge of the neonatal intensive care unit. These units, designed to provide specialized medical care for premature newborns or sick infants, have different configurations that depend on the newborn's developmental status. Therefore, in this chapter, we discuss the medical equipment available in NICU and its usefulness during the infant's maturation process. Then, we present the audio-video acquisition system designed for the newborns' contactless monitoring as well as the database created thanks to the European project Digi-NewB. Finally, we address the subject of the acoustic environment in NICU that makes the data automatic processing particularly challenging.

After identifying the sound sources heard in the bedrooms, we present two analyses performed on manually annotated recordings. The first one is the quantification of voices, and alarms in a -hour recording performed on a premature baby staying in an incubator. The second one shows the sound content's great variability within the recordings. This context is essential for understanding the strategies and methods developed in this work.

. Thus, despite the technological progress brought by the Industrial Revolution, it was necessary to wait for Dr. Hess's intervention in so that these techniques are finally recognized. With his chief nurse's help, Evelyn Lundeen, he initiated the establishment of trained nurse teams following specific protocols. Finally, it was in the s that neonatal intensive care units flourished and became hospitals' integral part of the developed world, which helped to drastically decrease neonatal mortality (see [ ] for a historical review).

Description of the NICU

Nowadays these units are designed and used to provide specialized medical care for sick and premature newborns to ensure their development. During hospitalization, newborns go through different configurations depending on their physiological state. To be discharged home, infants must meet the following criteria:

• be thermally independent to maintain the body at a normal temperature;

• have self-sufficient respiratory control;

• be able to feed by mouth to support appropriate growth.

When infants do not yet meet these criteria, and according to their disabilities and development, they may benefit from thermal, respiratory, and nutritional assistance in different bedroom configurations. Depending on their functional immaturity, at birth but also throughout the hospitalization, they join the care process at the appropriate step. These steps are detailed below and then the physiological and neuro-behavioral monitoring in NICU are presented.

-A baby born extremely premature (i.e., before 8 GA) is not ready to face the extra-uterine life. Her or his thermal system is not able to regulate body temperature properly and her or his skin is still too thin to ensure its protective function. Therefore, at this stage, the baby is always placed in an incubator in the NICU for several weeks. An incubator is a bed enclosed by a plastic shield in which the environment is controlled to keep the baby at the right temperature and humidity level (see -Several respiratory support techniques are used to meet the oxygen needs. Very premature infants may have respiratory distress, such as apnea or bradycardia (slow heart rate), that requires immediate intervention. For the most dependent, an invasive procedure such as intubation supplemented by a ventilator assistance device may be used. Then, depending on respiratory autonomy level, intubation is gradually replaced by less invasive devices such as nasal masks or cannulas (see Figure . ). Throughout the hospitalization, clinicians daily assess the newborn's needs by various means, such as evolution analysis of respiratory distress or blood tests (twice a day).

A similar strategy is applied to feeding. Initially, premature infants are not able to digest food. They are therefore fed (i.e., given essential nutrients) through a central venous catheter, which is connected to the heart through the arm or leg. Next, thanks to a naso-gastric tube (i.e., a tube connecting the nose to the stomach, see Figure . ), they are fed with very small milk quantities. As they develop, infants consume more milk and less infusion.

Then, when they are sufficiently developed and after they began to suckle, food administration is gradually replaced with breastfeeding. In general, the feeding capacity remains the last step before discharge. As mentioned earlier, the newborn's skin temperature can be measured to regulate the temperature of the incubator or the radiant warmer. Heart and respiratory rates are continuously acquired through electrodes set on the infant's body, while arterial blood pressure is measured with an armband. In case of cardiorespiratory distress, alarms are triggered for nurses who can perform immediate interventions.

Non-invasive pulse oximetry is also used to measure blood oxygen saturation and pulse rate using a photodetector. It is used in most neonatal intensive care units as a detector of de-saturation (sudden loss of oxygen in the blood).

These sensors are kept for most newborns throughout their hospitalization. An example of a radiant warmer infant bedroom configuration is shown and described in Figure .6. Sometimes at the end of the stay, when the infant goes into the parents' arms or in the parents' presence, the sensor's wires can be disconnected from the scope. In practice, a one-hour examination is performed by a trained nurse who visually annotates, within -minute steps, several components such as sleep stages, vocal, motor, or facial activities.

These components have also been shown to be relevant for the detection of various neurological disorders [6][7][8].

However, several limitations hinder the generalization of these procedures. Indeed, this operation is very time-consuming and only a small proportion of newborns can benefit from this monitoring. Furthermore, although it is performed by specially trained nurses, these observations remain subjective.

. The Digi-NewB proposal

The -Audio and video data were stored independently in -minute files, respectively in "WAV" and "MP " formats. It is worthwhile to remind that the prototype of the recording device was created during the initial phase of the project and some problems were encountered on the first recordings. Thus, despite the availability of two microphones, only one of the two channels is used in this work. In addition, several video modalities were explored at first to select the best ones for the sepsis and maturation purposes.

-Concerning the recording protocol, nurses from the six partner hospitals received training to place the devices on either side of the infants. Due to the room layout diversity and the available equipment, it has not been possible to establish a strict protocol for the system position and distance. Moreover, microphones were placed differently depending on the type of bed: in a closed bed, they were placed inside the incubator at the newborn's feet,

while in an open bed, they were set near the head in a perimeter ranging from cm to meter.

Examples of the Digi-NewB devices used to collect data in real context are shown in Figure .

that presents the system installed in the NICU at the Rennes CHU. In the pictures, babies are connected to the traditional physiological monitoring systems while cameras and microphones are set up to record the infants' movement and sound without any contact. This continuous real-time monitoring has the advantage of not affecting the infants' environment in the NICU, which could be detrimental to their maturation, neither imposing additional difficulties for health care staff or parents to interact with the newborns.

8 Chapter • Work context
-In this work, we focused on the second objective of the Digi-NewB project, i.e., the evaluation of maturation therefore we assess only healthy infants. The protocols established for these newborns consisted in recording for several consecutive days, between birth and the date of central line removal, and then every days for approximately hours until discharge (Figure . ). Although during the Digi-NewB project babies were recorded, only a small cohort is analyzed in this work. This is because a careful and time-consuming selection was made by the medical team to identify healthy newborns who had no complications during their entire hospitalization.

As a result, a base of healthy babies including girls and boys born between + and +6 GA recorded between + and + PMA were involved in this thesis work.

. . The Digi-NewB proposal .

Acoustic environment in the NICU

To ensure the proper newborn development, the NICU environment should be similar as far as possible to what it would have been in an intrauterine pregnancy. However, as mentioned before, a lot of medical devices are needed to take care of the newborn's health. Alarms are activated when the infants' states are unstable (i.e., cardiac, respiratory distress, ...) or when the machines require human intervention (i.e., warnings for empty syringes, empty ventilation water tanks, missing equipment connection, ...). In addition, several adults are also present around the baby, mainly the medical staff and the parents. As a consequence, the acoustic environment surrounding newborns in the NICU is quite noisy and has already been investigated [ -].

Hence, the sound environment contains many disturbing noises of short duration and at irregular intervals that deeply corrupt the audio recordings. Furthermore, several of these sound sources -During hospitalization, newborns are monitored through sensors measuring their heartbeat, oxygen saturation, and respiratory frequency signals. When an abnormality is detected, such as an irregular heartbeat (bradycardia) or lack of oxygen (de-saturation), the monitor starts ringing through different alarm levels to inform of either an important or a critical situation.

-The medical devices surrounding the baby's cradle are designed to recreate the intrauterine environment and provide all the care needed. Furthermore, the more premature is the baby, the more machines are needed for her/his good development and the noisier is the room.

The machines aim to warm up the water infusion, warm up the bed, help the infant's respiratory system, feed, etc.

In addition to the fact that some of these machines produce noise during their use, they all have one or more specific alarms to inform nurses of their status when they require human action (i.e., problems or maintenance such as filling the water infusion or the feeding syringe driver). While noises are usually wide frequency bands, alarms are narrow-frequency bands with each of them having a different tone, duration, and time of repetition.

-There are several types of ventilation devices in the NICU that are chosen according to the preterm infant's particular needs. Ventilation produces noise that strongly interferes with the acoustic environment. This noise is usually spread over a wide frequency and is not constant over a recording since it can be turned on or off at irregular intervals. As a result, ventilation introduces a lot of variability to the data.

-Mainly sliding glass doors that make noise when they open and close and have poor sound insulation properties. A door sound is short in time and has a wide frequency range.

-Parents and nurses can be talking or whispering when being in the bedroom. The human voice is a harmonic signal located in the low-frequency range (i.e., up to Hz).

. . Acoustic environment in the NICU -Several times a day, care is given to the newborn, for change, wash, feed, ... during these moments noises can be emitted by medical plastic bags, drawers, hand washing, ... Usually very short in time, these noises have a wide frequency band.

For readers who may not be familiar with the NICU environment, a typical neonatal health care unit bedroom is depicted in Figure . . It is worthwhile to mention that during his/her shift, a nurse is in charge of several newborns.

To ensure an intervention when necessary, all the monitors corresponding to those babies are related one to another. Thus, when caring for one baby, the nurse can listen to alarms related to the other newborns she/he is responsible for through the monitor located next to her/him. Therefore, besides the alarms related to the monitored baby, slightly different alarms can occur in the acoustic background coming either from other close bedrooms or from the central reminder (where nurses monitor all newborns). Moreover, although nurses are paying close attention to all the assigned babies, they cannot be everywhere whenever an alarm occurs, therefore, it can lead to long, noisy periods. It is the most harmful and common sound source in the bedroom.

In addition, it is worthwhile to mention that even more complex environments were recorded with co-bedding for twins or shared bedrooms with several babies. Particular attention will be paid to these recordings since we cannot know if the recorded cries are actually produced by the monitored baby.

. . Noise quantification in a -hour annotated recording

To show how noisy the acoustical environment can be in NICU, we studied a -hour recording performed in the incubator of a very premature infant .

After listening to the recording (i.e., hours corresponds to WAV files in the Digi-NewB . i.e., baby recorded between : PM and 8: AM the --.

Chapter • Work context database), we decided to manually annotate two noise categories, i.e., alarms and adult voices which are the most representative. These annotations were performed through a careful subjective listening recognition according to the sound level and spectral content.

Alarms also called beeps are short and often repetitive noises that alert nurses about the status of either the machines or the infant's health. All alarms have different tones allowing the nurses to quickly detect the problem source and fix it. Since the physiological monitor can relay alarms unrelated to the observed baby (see the previous section Sounds in Digi-NewB recordings), we discriminate the alarms into two subcategories:

• baby alarm: alarms that directly concern the infant's health status (high intensity level);

• reminder alarm: alarms coming from the physiological monitor that correspond to the other neonates under the nurse's care (low intensity level).

We inspected the recording through -minute windows. In practice, it means that for each of the minutes, we annotated whether an adult voice, a baby alarm, and/or a reminder alarm occurred. Therefore, when at least one of these sounds occurs in the analyzed minute, we consider it "noisy", whereas when no sound is found, minutes are considered "clear". A resulting timeline is illustrated in Figure . for a -minute WAV file.

Figure . :

Illustration of the annotations performed on the one-minute windowed timeline for a -minute audio file. A minute is labeled in red when at least one adult voice or one alarm occurs, otherwise, the minute is labeled in blue. The "reminder" and "baby alarms" lines are combined in the "alarms" line, itself combined with the "adult voice" to form the "total" line.

The annotations' distribution over the hours is presented in Figure . . From a global point of view, results show that 6 % of the minutes are considered "noisy", meaning that from the analyzed minutes, 6 of them contain at least an adult voice or an alarm. Among these noisy minutes, 6 % of them correspond to alarms, % correspond to adult voices, while % are a combination of these two sources.

Since alarms were discriminated through two subcategories, one can see the unexpected distribution. Indeed, only 6 % of the minutes are related to the monitored infant, whereas 6% of them come from reminder alarms corresponding to the other neonates under the nurse's care. The remaining % corresponds to a combination of both sources.

To go further, we also studied the alarms and identified ten different types. For each of them, the duration between two occurrences is different, as well as the frequency spectrum, which is . . Acoustic environment in the NICU composed of a fundamental frequency that ranges from Hz to . kHz and may contain several higher components. Furthermore, it is worthwhile to mention that most alarms do not consist of a single beep, but rather of several beeps. Also, this list may not be exhaustive since the study was conducted during the night ( : PM -8: AM) for a preterm infant in an incubator.

Other alarm types may be encountered during the day and/or in other NICU rooms, depending on the bed and care equipment.

In addition, the author would like to point out the risk of such a noisy environment on newborns.

Indeed, it has been recognized that babies are very sensitive to high surrounding variations such as light [ 8], odors [ ], or noise [ ] and high exposure can lead to potential neuronal circuits wire damage of the newborn brain. Moreover, premature infants have an underdeveloped auditory system that is not able to adapt to an extrauterine acoustic environment in the same manner as a full-term infant. While the fetus begins to respond to low-frequency sounds after weeks of gestational age [ ], the cochlea's response to sound continues to mature between and weeks. As a result, loud noise can create neonate stress responses [ ] that may lead to hair-cell damage and subsequent auditory impairments [ -6]. Nevertheless, the problem of the noise level in NICU is well known by the medical staff who suffer from it as well. Hence, improvements are required and the internal review at the Rennes CHU suggested for the future the use of portable alarm systems, the development of new alarm algorithms, or the development of new devices. To date, the neonatology health service in Rennes suggests improving acoustic conditions through alarm management protocols and wishes to consider acoustic improvements during the reconstruction of its site.

In this section, we proposed to quantify the noise that can be present in a NICU bedroom. This characterization, although considering only voices and alarms, showed once again the difficulty of automating crying detection treatments in such an environment.

Chapter • Work context . .

Sounds variability in recordings

Through this study, we want to show the great variability of the sound content encountered in the different WAV files that constitute a recording. Therefore, three -minute sound files were selected from a -hour recording made for one baby . Their content is briefly described hereafter:

• h : some sound events with few cries;

• h : a lot of cries with few sound events;

• h : very few sounds, no cry.

Thanks to the knowledge of acoustical environment in the NICU, five labels were chosen according to the possible sound activities and are defined as follows:

• cry: infant crying;

• baby other: infant vocalization (e.g., cooing), coughing or hiccups;

• alarm: alarms produce by medical devices;

• voice: adult voices, whispering;

• other: background, footsteps, doors, cares, any other noise.

The recordings were manually labeled using the free and open-source digital audio editor and recording software Audacity through start-and end-points identification of all audible sound events in the soundtrack. In practice, it means that even when some sounds occur at the same . . Acoustic environment in the NICU From these distributions, it appears that sounds do not last longer than seconds and that cries are mostly longer than . s. Although the sounds Baby other and Other are slightly shorter, all labels have a similar range of duration. Therefore, it is not possible to distinguish crying from other sounds by their duration. Moreover, we remind that we have considered the sounds individually.

Therefore, longer durations are to be expected when considering mixed sounds, i.e., when we place limits on the points where all the sounds together can no longer be heard.

During the process, we decided to ignore short sounds with a duration of less than . s which are more difficult to identify. The resulting selected sounds are presented by labels in terms of total segment quantities and duration in Table . : Labeled sound events for three -minute files.

Based on these results one can see the file diversity and the great variability of the sound contained within the same recording. Naturally, the sound environment is not always noisy and if sometimes a -minute recording contains almost no sound (i.e., : a.m. -sounds, . 6 s), it can also contain a lot (i.e., : p.m. -6 sounds, . s). These distributions show once again the complexity of the sound environment and the difficulty to set up a completely automatic processing chain which, however, would be absolutely necessary to perform cry analysis in the NICU.

6 Chapter • Work context .

Conclusion

The main objective of this chapter was to learn about the neonatal intensive care unit. Indeed, to perform cry analysis, it is necessary to understand this environment, which is designed and used to provide specialized medical care to sick and premature newborns. After a review of the medical equipment that may be used to assist the newborns, we also presented the physiological and neuro-behavioral monitoring usually performed there.

Next, we introduced the European project Digi-NewB in which an audio-video acquisition system was designed and used to record more than infants providing a very large multi-signal database (i.e., heart rate, respiratory rate, video, and audio signals). We also detailed the recording protocols used and illustrated the system's set-up in two NICU configurations.

Then, we presented the acoustic environment in the NICU, which was recorded during the Digi-NewB project, and we reviewed the main sound sources occurring in the audio signal. Through the quantification of the noises occurring within a -hour recording on a very premature infant staying in an incubator, we showed the acoustical complexity of the environment. Moreover, to survive the infant needs medical equipment that produces alarms in 6 % of the minutes for this recording, which shows that the files can sometimes be very noisy. Finally, with the annotation of three WAV files from a -hour recording, we showed the great variability of the sound content between the different minutes files. Indeed, while some contain a few sounds, some others can contain a lot.

With knowledge of the environmental context and the sounds occurring within the recording, it appears that using the three-step processing chain proposed in the first chapter (see Section . ) is a good approach. Indeed, using a sound event segmentation step seems to be a relevant strategy considering the large amount of data to be processed. Actually, such a procedure is essential to reduce the quantity of data to analyze in order to detect crying. Therefore, in the following chapters, we present the different steps implemented in this work, including the segmentation, classification, and fundamental frequency estimation methods.

[ ] Z E. L A. Ototoxicity in preterm infants: Effects of genetics, aminoglycosides, and loud environmental noise. Journal of perinatology : official journal of the California Perinatal Association, vol. ( 8 ). Introduction

Traditional cry segmentation methods are based on energy thresholding. When applied in a controlled and non-noisy environment these methods lead to accurate cry detection. However, in the context of this work we are dealing with long audio recordings and a large database. In addition, recordings are performed in NICU where the sound environment is very noisy and many sounds occur besides the infant's cries (see Chapter ). In this case, it is important to understand that using traditional techniques, all the sounds will be extracted and we should therefore call this approach sound segmentation. Hence, to detect the cries in such signals, a two-step strategy was adopted during the Digi-NewB project, including a sound segmentation step and a subsequent classification step.

This chapter introduces the first step of this strategy. After a review of the state of the art, we describe the method proposed by Orlandi et al. [ ] that inspired our own. Then, we introduce the improvements, which have been done, to adapt the method to our data. Finally, we also propose to use the motion information computed by another team of our laboratory (LTSI) during the Digi-NewB project [ , ]. We suggest collecting only the sounds appearing in newborns' motion intervals to reduce the data quantity to be further processed.

.

State of the art

If in the literature, cry segments used to be manually recorded or selected, some recent studies, proposed automated solutions. Indeed, when working with long recordings it is necessary to apply appropriate processing to extract cries from the signal. The preliminary step, defined as the segmentation step, is a must to separate cries from the background soundtrack.

In the context of speech processing, it is common to perform audio segmentation tasks. Indeed, it is widely used in speech detection in the audio signal or in voiced/unvoiced part detection, resulting in both case in the extraction of relevant parts of the acoustic signal. The same principle is used for infants, it is called the detection of cry units (CU). As in speech, where the initial and final points of a word are located, the objective in cry unit detection is to find the initial and final points of a cry unit. As words in speech, the cry units have higher energy than unvoiced segments.

In the following, we describe traditional segmentation methods and then discuss the new strategies that have emerged to process long-term audio monitoring which has recently grown in popularity.

. .

Methods for cry segmentation in short recordings

At first, cries used to be manually recorded in controlled quiet environments. Thus, most of the traditional methods were based on the computation of Short Time Energy (STE) and Zero

Crossing Rate (ZCR). While the first one provides an energy envelope of the sound signal, which helps to distinguish audible sounds from silence, the second one allows to detect the voiced parts. However, to reach good performances, these processes require large amounts of data, and some authors proposed to introduce normalization and regularization to adapt CNN to a limited data set [ 8], or to enhance the data set with simulated data [ ].

Methods

As a result, automatic cry classification in domestic environment led to AUC over % in [ 8] and .

an

. Discussion

Therefore, when processing recordings performed in controlled quiet environments, an automatic cry segmentation can be easily computed based on energy thresholding methods. However, these techniques are no longer sufficient when it comes to recordings in a routine hospital environment.

In fact, in such a noisy environment, all occurring sounds are segmented and must be sorted to find those that contain crying.

To date, only a few studies have achieved cry segmentation methods in a monitoring context, and none have involved long recordings in a routine hospital care setting. Moreover, while frameby-frame classification methods work well, they are computationally intensive to process long records. As this work focuses on the large Digi-NewB database, we proposed a two-step strategy including an audio segmentation step and a classification step to reduce the data quantity to be processed.

. . State of the art .

Audio segmentation method

The first step of our strategy is therefore to segment the sound events that occur in the recordings.

We choose to exploit the method proposed by Orlandi et al [ ] which is based on energy and threshold calculations, the process is described and discussed below. Subsequently, we propose improvements due to the issues encountered during the application to our data.

. .

Orlandi's method

This traditional segmentation method based on energy thresholding is described in details in [ ] where a long term audio analyzer was proposed. The process was compared to existing software tools commonly used in biomedical applications using two synthetic signals sets: the first one was based on adult voice excerpts and the second one was obtained from newborn cries. This method is now deployed in the user-friendly voice analysis software BioVoice also developed by the Italian team [6, 8, ].

This method, illustrated in an exhaustive workflow in Figure . , is described in the following and can be decomposed into three main steps:

• pre-processing: band pass filtering and down-sampling of the recording;

• automatic segmentation: detection of sound intervals in the signal;

• duration filtering: exclusion of short sounds. --First, the recorded signal is band-pass filtered by a th-order Butterworth filter and cut-off frequencies set between and Hz. Then, the resulting signal is also down-sampled to . kHz to speed up processing.

--( ) The pre-processed signal is then divided into -millisecond windows with % overlap between adjacent windows. On each window the short time energy is evaluated as:

ste= log      n i= s(i) n + ε      ( . )
where n is the number of samples in the window, s is the signal and ε is a small constant to avoid log( ). The resulting values of all windows are stored in an energy vector named ste. An example of this vector is given in Figure . for a -second signal containing two cry units. We can see that STE values increase during the cries and decreases during silences. -' -Suitable thresholds are required to determine the boundaries, i.e., sound events start and stop points. They are obtained using a modified version of Otsu's method [ 6] applied on the STE value histogram. While the original method is described in the of this chapter, Orlandi's proposal [ ] is detailed hereafter.

. First, the histogram of the STE values is calculated through levels for a reasonable compromise between sufficient detail and computing speed. . Finally the two thresholds are multiplied by a factor d given by the ratio of the differences between the maximum and minimum values of energy and the number of levels of the histogram. This gives the upper T U and the lower T L thresholds required to determine if a frame is voiced or not. Then, minimum value of the signal energy is added to T U to guarantee that T U is above the minimum.

6 Chapter • Audio-Video segmentation T U , T L and STE histogram (for values from zero to T U ).

(c)

UP -short time energy vector with upper and lower thresholds, BOTTOM -segmented audio signal through both thresholds. Using a double threshold prevents sound split and improves the segmentation by finding better boundaries and extracting the complete sound event.

D fi

All detected sounds with a duration of less than ms are removed, so that inspiratory sounds are not taken into account [6]. Moreover, this duration was chosen for its hours recordings = 8 wav files). Hence, there are several ways to apply the automatic segmentation on our data with the thresholds that can be calculated:

• locally -with new computation for each -minute file;

• longitudinally -with a single computation for each recording (i.e., several hours);

• generally -with a single computation for all recordings.

We decided to use the first strategy with local thresholds because the sound contents are very variable within a single long recording (several hours) and especially between different recordings/babies with distinct configurations. For example, newborns in incubators require a lot of assistance and the sound environment can be disturbed by noisy machines (such as ventilators, heaters, ...) while the surroundings of infants in a regular cradle can be very quiet.

Therefore, since Otsu's method is sensitive to the environmental audio content (based on the fact that thresholds are computed over the signal energy values distribution), it was decided to compute local thresholds to ensure crying detection in any acoustical context. In practice, the method applied to our data consists in computing the short-time energy vector and Otsu's thresholds for each -minutes file.

The signal resulting from the segmentation process for a -second noisy audio signal is illustrated in Figure . with the different sound sources occurring in the recording. We can observe that some extracted segments are very long (e.g., more than seconds).

Issues related to poor boundaries detection

After applying the segmentation method on our data, we noticed several errors in the detection of the initial and final points in the following three cases:

• First, in the case of crying bout (several consecutive cries) and when the pause between two cries is very short only one audio segment might be extracted. This is due to the fact that the energy signal is calculated using sliding windows and that the STE values cannot

decrease. An example is given in Figure .6a where the resulting audio segment is a signal containing a crying bout composed of six cries.

• Then, once again it is worthwhile to remember that the recordings are performed in the NICU, a routine care environment where many sounds can occur besides infants' cries (i.e. • At last, some noise sources are activated for long period and considerably influence the energy value distribution. This case can occur when a ventilation system is used to assist a newborn in an incubator. The machine produces a constant background noise that is randomly turned on and off during the whole recording. Impacting the energy value distribution can lead to the detection of very long sound events matching the ventilator activation. In that case, the extracted sound events can last several seconds or more.

. . Audio segmentation method

Issues with replication

In addition, we faced a problem when reproducing step of Orlandi's method (see Section . . ).

Indeed, in this step, the authors propose to adjust the thresholds in particular with the help of a ratio using the minimum of the energy. According to them, this step is used to guarantee that T U is above the minimum. However, in our case, the minimum of energy is very small, with values close to zero. Therefore, we have not implemented this step in the replication.

In fact, this type of step is necessary when processing signals containing very few sound events, because the majority of the energy values are located in the low amplitudes. Thus, the thresholds, which are computed on the data distribution, do not allow to separate the few sounds from the background noise.

The case of a signal extracted from a recording containing very few sound events is illustrated in Therefore, in the next section we propose improvements so that the method correctly handles i) the resulting segments of long duration that are likely to be part of the three cases described above and ii) the recordings that do not contain many sounds. . .

Improvement of the method

Regarding the extracted segments of long duration, we propose a re-segmentation step. Then, in order to manage sound detection in recordings with little sound content, we introduce a modified workflow that better takes into account the sound environment thanks to new frequency filters.

Finally, to reduce the amount of data to be processed, we suggest a supplementary step to ignore recordings containing very few sounds. These three improvements are presented below.

Re-segmentation (RS)

As we showed, the segmentation sometimes results in the extraction of long segments containing several cry units (see ).

After applying the pre-processing and automatic segmentation steps, the segments of duration longer than five seconds are identified. For each of them, the corresponding pre-processed signal is extracted and new local thresholds are computed. These local thresholds are then applied to the extracted signal to find new initial and final points. Eventually, among the resulting segments, only those with a duration between . and seconds are retained. This re-segmentation step is illustrated in Figure . .

Narrowing STE frequency band (NFB)

To better handle the recordings that do not contain many sounds, we propose to modify the pre-processing and automatic segmentation steps of the initial workflow by using a double frequency filter.

. . Audio segmentation method • the second band-pass is set between and Hz, which is a reasonable frequency band where an infant can be expected to cry [ 8].

Once both signals are pre-processed, the two thresholds can be computed between and

Hz to be more sensitive to the acoustical environment. However, the short-time energy on which the thresholds are applied is computed on the signal filtered between and Hz.

This strategy allows a better segmentation in recordings that do not contain many sounds and has the advantage to reduce the segment detection to sounds with energy located within the infant crying frequency band. Therefore, the final workflow is illustrated in Figure . .

Long-term threshold (LTT)

First, to detect variations in sound content in the recordings, we suggest using a threshold T computed on sliding window of two hours. To do so, T is computed like T U over the concatenation of ste vectors calculated on the pre-processed signal filtered between and Hz. Then, the files with less than intervals detected above this threshold are discarded. This step allows to reduce the number of resulting segments by ignoring -minutes files with very little detected sound content.

6

Chapter • Audio-Video segmentation .

The use of motion for audio segmentation

It is quite natural to consider that a baby is moving when she/he cries. Indeed crying, which Therefore, information about the infant's movements could be used to reduce the amount of data to be processed during automated crying analysis.

In addition, Orlandi et al. proposed a contactless system for audio-video infant monitoring (AVIM) in which both modalities are considered separately [6]. In their study, the motion analysis is semi-automatic since the user needs to select points to track on the video frame, and automatic crying analysis is performed after the manual removal of interfering sounds.

In this work, the automatic sound segmentation allows extracting the sounds with energy located between and Hz (i.e., crying, adult voices, monitor beeps...) and thanks to the work of another team of the laboratory, the automatic motion segmentation allows identifying the intervals of movement and non-movement.

. . The use of motion for audio segmentation Therefore, after presenting the video segmentation method in the following section, we propose to use joint audio and video processing to reduce the number of segments to be further processed in . Then, the intervals when the baby is not present in the bed, as well as those including the presence of adults (parents or caregivers) in the field are automatically excluded. This essential step is performed thanks to a Deep Learning approach (see [ ] for more details).

The different configurations are presented in Figure . . . . The use of motion for audio segmentation . .

Database

We selected a set of data representing a large part of the diversity encountered in the project.

Hence, 6 recordings were selected from the Digi-NewB database. They involve girls and boys born between +6 and + GA and recorded between 8+ and + PMA.

In total, hours (i.e. 8 audio and 8 video files of minutes) were processed. Recordings were generally performed overnight periods, between : PM and 6: AM, and lasted about 8 hours each.

Using the audio segmentation method (see Section . . ), sound segments were automatically extracted corresponding to day, hours, 6 minutes and seconds duration.

To check the cries distribution within movement, a part of these sound segments was annotated manually. Finally, a total of cry segments were identified through human listening, corresponding to a duration of hour, 8 minutes and 6 seconds.

. .

Motion quantification

First, we quantify the percentage of motion in each recording. To do so, we apply motion segmentation to all 6 video recordings of the infants. . .

Motion distribution averaged over all recordings is presented in

Sounds within infants' movement

In this section we consider all the automatically extracted sound segments resulting from the audio segmentation step and we quantify their distribution within motion segmentation. In practice, it means that for each sound, we observe the corresponding motion segmentation signal and consider the sound with the following conditions: . . The use of motion for audio segmentation

sound is         
• in non-motion phase during % of the time and should correspond mostly to machine noises;

• in NaN intervals during % of the time which correspond to the baby's absence and/or the presence of adults in the image field. It is important to note that in the latter case, it is normal to observe a significant sound contribution since adults are usually there to take care of the newborn. Hence sounds occurring in this interval should mostly correspond to adult voice, cares and newborn cries;

• in motion intervals during % of the time and should contains infant cries.

Thus, the purpose of the following section is to assess the cries distribution within motion segmentation.

. .

Cries within infants' movement

In this section we consider the cries manually identified. In the same way as before we observe, for each cry, the corresponding motion segmentation signal. Results are presented in

Figures . 8:

Cries distribution in motion segmentation (including babies, recordings, cries).

These new results showed that only few cries occur when the baby is not moving (i.e. less than % within the non-movement intervals). After a new investigation, we found that it correspond to video segmentation errors especially with the detection of non-movement intervals instead of baby absence intervals.

.

.6 Discussion

In light of the preliminary study results, we can conclude that cries never occur in the non-motion intervals. Hence, video segmentation can be used to reduce the amount of signal to be processed.

Indeed, limiting the sound segmentation within motion intervals reduces by 8 % the sounds to be classified afterwards (see Figure . 6b). However, if it is easy to ignore the non-movement intervals, special attention should be paid to the NaN intervals.

On the one hand, these data must be analyzed differently depending on the application. When the goal is to detect as much crying as possible, they must be kept, however, when motion information is required, they can be removed. This is the case in sleep stage estimation [ ], where signal information are combined to define the infants' sleep states. In this case, the NaN intervals are unusable anyway for motion analysis, hence, audio processing is not performed during these intervals either. On the other hand, these intervals correspond to complex data, especially in the presence of adults. Indeed caring can produce a lot of sounds that may be mixed with the crying.

Hence, in the framework of this work although many cries occur during NaN intervals, we decided to collect the sounds occurring within motion intervals only. In addition, this strategy can be . . The use of motion for audio segmentation useful to process recordings made in shared-bedrooms since it should limit the detected crying amount that does not belong to the monitored infant.

In regards to the studied database, considering only infant's movement intervals would reduce to % of the total duration of the sounds automatically segmented. In terms of duration, it means processing hours instead of the hours initially segmented. Therefore the video segmentation can be a valuable strategy especially when processing very large database such as the Digi-NewB.

. Evaluation strategy

To assess our segmentation method, three -minute files were manually annotated by identifying the start and end points of all audible sounds and their type (crying or not crying). The proposed strategy is to perform a comparison between cries manually identified and the sound segments derived from the automatic segmentation methods.

To perform such a comparison, two detection signals are created for both cases: manual and automatic. Designed with the audio recording sampling rate (i.e., kHz), these signals are filled with and values such as:

detection signal =       
within manual annotations (cries) or audio automatic segmentation (sounds), otherwise (silence).

( . )

The annotated cries and segmented sounds are compared in terms of segment quantities and durations. Moreover, for consistency with the method, annotated segments with a duration lower than . second and greater than seconds are not taken into account. The parameters used to evaluate the segmentation are described hereafter.

. .

Segment comparison

To compare the manual annotation and the automatic segmentation in terms of segment quantity we define four segment parameters, which are illustrated in Figure . . For each manually annotated segment, we evaluate the corresponding automatic detection signal. When at least % of the manual annotation samples are equal to within the automatic segmentation signal, the audio segment is considered detected, otherwise it is not.

Chapter • Audio-Video segmentation
Narrowing Frequency Band (NFB) improvement

As mentioned before, this step improves the threshold computation and helps to better take into account the acoustical environment. Its impact on the three files is detailed below.

• In : the number of false detections (which are mostly noise) particularly decreased.

Indeed, this step allows extracting segments instead of the ones extracted with the reproduction method. In terms of duration, it is a matter of extracting .6 s instead of the 8 .8 s previously extracted.

• In : the number of detected segments that are not cries diminished (i.e., FP decreased from to segments). Moreover, we can also notice that the number of detected cries has slightly decreased without impacting the total duration of the extracted cries (i.e., ∆ S decreased from % to 6 %).

• In : the number of extracted segment increased (i.e., TP A increased from to 6 segments) without impacting the sensibility).

Re-Segmentation (RS) improvement

The re-segmentation step is applied to the resulting segments obtained through the NFB method.

First, we saw in section Section . . that the annotated cries did not last more than a few seconds (see Figure . ). Then, we showed some issues related to the poor segmentation of cry bouts (several consecutive cries) and mixed sounds. Therefore, this is why the re-segmentation step proposes to re-cut the detected segments whose duration is longer than five seconds.

This step allows both to normalize the extracted segment duration and to better segment the sound events. This can be noticed with the file for which the amount of automatically extracted crying segments increased (i.e., TP A increased from 6 to 6 8 segments) without affecting the total duration of the extracted cries (i.e., ∆ S decreased from 8 % to 86%).

Long-term threshold (LTT)

As a reminder, we proposed to use this step to reduce the amount of data to be processed by ignoring recordings containing very few sounds.

In order to illustrate this procedure, we computed the thresholds for all -minute audio files of the recording performed on baby (i.e., files). They are presented in Figure . a.

First, one can see the considerable variability with T U , represented in grey, and T L represented in red. These variations are due to the signal energy value distribution related to the acoustic environment. Thus, files containing minimal audio content have low thresholds (e.g., ) while files containing many sounds have higher thresholds (e.g., & ).

.

Results

Then, to be able to detect WAV files with poor acoustic content, we suggested to use a sliding It should be mentioned that this step is presented in this results section after the NFB and RS steps to show the interest of these improvements on the three annotated files. However, in the processing chain it is applied before these steps.

6 Chapter • Audio-Video segmentation

Audio-Video segmentation

Regarding the results derived from the complete audio-video segmentation by collecting only the sounds occurring in infants' motion intervals, we can see that it reduced the number of extracted segments to be further processed without seriously affecting the total duration of the extracted cries. Indeed, when comparing the REP step with the AV step, we can see that the sensibility decreased in both files (i.e., ∆ S diminished from % to 6 % for and from 8 % to 6 % for ) while the precision increased (i.e., ∆ P increased from % to % for and from % to % for ).

We remind that it is normal that the sensitivity (i.e., the number of extracted cries) decreases since we do not collect sounds contained in NaN intervals (i.e., adult presence and/or baby absence) in which cries can occur.

. Conclusion

We have seen that most of the studies conducted in the literature concern recordings made in specific, non-noisy environments. Thus, the usual pre-processing step called "cry segmentation" cannot be used in our case since the recordings studied contain many other sounds than infant cries. This is due to the NICU environment which hosts care activities and machines required to help premature or sick newborns. Therefore, we proposed a two-step crying extraction method.

While the first step segment all the sounds occurring in the signal, the second step will classify the extracted sounds and detect those containing crying.

In this chapter, we have presented the first step, which allows extracting sounds from background noise. This approach is based on the one originally proposed by Orlandi et al. in [ ] and improvements were proposed to better process our database. Thus, after removing -minute audio file containing a poor audio content, we included a frequency filtering as well as a re-segmentation steps. These enhancements were applied consecutively to three different -minute annotated files and we compared the results in terms of segment and duration. We showed that the method is relevant for cry extraction and also helps to reduce the amount of data to be further processed (i.e., sensibility greater than 6 % and precision greater than %).

Finally, since motion segmentation was also performed during the European Digi-NewB project, we proposed to extract exclusively the sounds occurring within infant's movements intervals.

The relevance of this strategy, which naturally suggests that a baby is moving when crying, was confirmed in a preliminary study with an evaluation on a large database including hours from 6 recordings of newborns (see Section . ). First, we showed that infants are most of the time immobile (i.e., % on average) and that they don't cry during those periods. However, the cries may be produced in motion intervals or in intervals with adult presence and/or baby absence. However, in the latter case, it is more complicated to process the audio recordings since . . Conclusion it correspond to care periods in which many other sounds are produced. Yet, to minimize and facilitate at most the process, we decided limit the audio segmentation within motion intervals.

Considering the studied database, this strategy led to discarding 8 % of the sound segment total duration initially extracted.

The use of motion for audio segmentation is however not a mandatory step and has some limitations due to interval detection errors. Nevertheless, this strategy was never performed before and seems relevant for our data processing.

At this stage, all sounds whose energy is included in the newborn's fundamental frequency band are extracted. Thus, the next step consists in classifying these sounds to detect those containing cries. This is the purpose of the following chapter which proposes a binary classifier by a Deep-Learning approach using spectrograms.

-' Otsu's thresholding concept is coming from image processing and is used to binarize an image based on pixel intensities. In other words, it manages to convert an image composed of several gray levels into black and white, such as illustrated in Figure . . To do so, the algorithm assumes that the picture is composed of two classes and tries separating the foreground pixels from the background ones. The optimal threshold is determined by minimizing intraclass intensity variance (defined as a weighted sum of the two classes' variances) or equivalently, by maximizing inter-class variance σ B defined as:

σ B (t) = ω (t)ω (t) [µ (t) -µ (t)]
where weights ω and ω are the two class probabilities and µ and µ are the means of these classes separated by the threshold t.

Therefore, by computing iteratively inter-class variance through all possible thresholds based on the image pixel distribution (i.e., histogram), it is possible to determine the optimal threshold located where the inter-class variance is maximum, see Figure . . Computed with a recursion relation it permits fast calculation and gives an effective algorithm with the advantage of reduced processing time.

Since the procedure to determine an optimal threshold based on the global histogram properties is simple, automatic, and stable, the method was implemented for mono sound signals in [ ].

Appendix Introduction

The previous chapter has demonstrated that the audio segmentation is not sufficient to extract cries in recordings performed in a noisy environment such as the NICU. Whatever the efficiency of the segmentation step, due to the real context with many sound sources, the resulting audio segments are not just crying but also voices, alarms, etc. Therefore, it is of great interest to classify the segments derived from the previous chapter to detect the ones containing cries.

In this chapter, we propose a framework based on a feature learning scheme powered by a pre-trained discriminative Convolution Neural Network (CNN) using spectrograms. After a review of the state of the art of methods investigated in the literature, we present the main components of the proposed framework. Then, we introduce our two-step training strategy to fine-tune some of the model hyperparameters. Finally, since supervised neural network approaches expect dedicated training and testing sets of annotated data, we introduce the annotation software created to design such a database.

.

State of the art

As mentioned in Chapter , studies have shown that important information related to infant health status, emotions, and needs can be interpreted by analyzing the acoustics of infant crying.

Therefore, in the last decades, many studies have focused on the detection or classification of infant vocalizations. In fact, the different approaches can be divided into four categories which are described below.

• Pathology detection which is a binary classification task where a cry is classified as normal or pathological [ , ].

• Pathology identification which aims to determine the type of pathology the infant is suffering from. It has been used, for example to detect deaf newborns [ , , ], those who have suffered from perinatal asphyxia [ ], both conditions [6, ], hypothyroidism [8, ] or even cleft palate [ ].

• Crying cause identification which aims to discover the reason that triggered the cry, for example, hunger, pain, sleep, or many other causes [ -].

• Crying detection which consists in identifying crying in the signal, either by determining the temporal limits of vocalization when processing the entire audio signal or by determining the presence of crying in segmented sounds.

In this work, we are interested in this last application where the goal is to detect the infant cry signal efficiently and accurately in a noisy environment. Studies that address this topic investigated data recorded over a long time, either at home to develop systems to detect crying and alert parents [ 6-] or in hospital. In the latter case, crying detection is performed to investigate infants' reaction to auditory stimuli of the NICU environment [ ], to quantify the amount of time an infant cries [ ] or to serve as a pre-processing stage for deeper analysis (i.e., related to pathology or cause identification) [ -].

A cry detection system is usually composed of two steps: i) a pre-processing step that extracts the most suitable features from sound signals and ii) a classifier to recognize the cry features in an audio signal. This section provides an overview of existing feature extraction methods and classification strategies (see [ 8] for an extensive review).

. .

Feature extraction

The challenge in cry detection systems is to select acoustic features that allow clear discrimination between a cry and other sounds. As mentioned in Chapter , the acoustic and prosodic characteristics of crying signals are often studied in time and frequency domains. However, it is the combination of the two, i.e., the time-frequency, that is most interesting. This domain involves dividing the sound signals into several small chunks called frames and constructing a feature vector for each frame. Thus, it allows following the variations of the frequency characteristics of the signal as a function of time. The most commonly used features are cepstral coefficients (MFCCs and LPCCs), wavelet transforms and Fourier transforms (illustrated in Figure . ).

Cepstral coefficients are widely used in the literature for audio signal processing and are recognized as performing well for tasks such as speech recognition or music genre classification.

In particular, MFCCs represent the short-term power spectrum of an audio clip based on the discrete cosine transform of the logarithmic power spectrum on a non-linear Mel scale. Thus, a method that takes into account all NICU challenges has yet to be proposed. Such a solution will provide a robust continuous monitoring tool to improve newborn health care through crying analysis.

.

. Evaluation metrics

Performance evaluation is an important aspect of the machine learning process. Metrics are mandatory to compare the results of the different trained models. Moreover, depending on the classification objectives, attention may be focused on different metrics. Therefore, in this section, we review the ones used later in this chapter to assess the model performance. While some of the definitions have already been described in Chapter , they are reported here in the context of a binary classification for the purpose of cry detection (i.e., the two classes being cry and non-cry).

is a very practical tool used to present the performance of a supervised learning algorithm. It quantifies the number of correct and incorrect classifications by comparing predictions to actual labels. A confusion matrix for a binary classifier is reported in Table . . It 86 Chapter • Classification for cry detection is composed of four numbers described hereafter.

• True Positive ( ): number of samples accurately predicted as cry;

• True Negative ( ): number of samples accurately predicted as non-cry;

• False Positive ( ): number of samples predicted as cry instead of non-cry;

• False Negative ( ): number of samples predicted as non-cry instead of cry.

All of the following evaluation parameters are calculated based on the confusion matrix and their values are between (i.e., worst) and (i.e., perfect).

Proposed method

In this work, we propose an infant cry detector using a framework based on a feature learning scheme powered by a pre-trained discriminative CNN using spectrograms. The latter are computed on sounds collected in the NICU and derived from the segmentation method (see Chapter ). Thus to identify cries in all the resulting audio segments we chose the spectrogram feature for its efficiency to represent a wide spectral decomposition in time. The final framework is a binary classifier composed of the two classes: cry and non-cry. The classification is performed in four steps illustrated in Figure . and described hereafter:

• for each extracted sound, the spectrogram is computed by a short term Fourier transform;

• since the extracted sounds have variable durations, the resulting spectrogram is cut into frames of the same duration;

• these images are used in the input of a convolutional neural network;

• for each initial extracted sound, the decision taken is the majority prediction on all the images. . .

Spectrogram computation

First, spectrograms were computed for each sound file using Short-Time Fourier Transform (STFT) of successive . ms long ( samples) Hamming-windowed frames with % overlap.

Since signals have a sample rate of kHz, the configuration provides a frequency resolution of . Hz (ranging from to kHz) and a time resolution of . ms (illustrated in Figure . ).

To have a good image contrast, the magnitude of the spectrogram is converted to a logarithmic scale and an image quantization of 6 levels is performed on a fixed colormap.

Then, the spectrograms were divided into several smaller spectrograms of the same size with a % overlap such as proposed in speech emotion recognition in [ ]. We named a small spectrogram: frame and the set of small spectrograms extracted from the big one: frame group.

Therefore, the number of frames depends on the cry duration. Before the split, the frame group is centered on the spectrogram since the acoustic characteristics are generally more interesting in the middle of it. Finally, each frame is saved with a resolution of x pixels with three channels (RGB). Since there is no consensus in the literature about the Fourier Transform windowing parameters, we decided to explore the spectrogram division with two different frame durations:

• . s which corresponds to the most common value found in the literature (see Section . . ).

• . s which is the minimum duration of the sound segments resulting from the segmentation (see Chapter ).

. . Proposed method . .

Transfer learning using ResNet architectures

Image classification tasks have seen breakthroughs in terms of performance, thanks to the rise of CNN. These networks are composed of a sequence of filters on the raw pixel data of an image to extract and learn high-level features. The analysis of the visual field is done through a set of overlapping sub-regions, this is called convolutional processing. The model then uses the extracted features to perform classification. The three main components of a CNN are convolutional, pooling, and fully connected layers. These layers are usually arranged in the form of a hierarchy where one can use any number of convolutional layers followed by pooling layers and at the end fully-connected layers. This type of architecture is defined by the number of layers in each component as well as the connections between them. For its part, the ResNet architecture was firstly proposed in to overcome the issues encountered when using a large number of layers [ 6]. Indeed, it solved the problem of the vanishing gradient by introducing the concept called Residual Network (ResNet). This technique, which allows skipping connections of a few layers, showed convincing performance in many computer vision applications and is now widely used for image classification.

In light of the ResNet performance, we decided to perform learning by transfer. This principle consists in reusing convolutional neural networks previously trained on a large image database. However, it should be noted that this particular strategy was used for the identification of crying cause in [ ] and that the strategy itself was experimentally verified in [ 8]. The input data for the CNN are the spectrogram images resized to the shape x pixels.

To adapt the model to our data, some parameters were fixed (see Table . ) while the following parameters have been optimized:

• the spectrogram division with frame durations of . or . s ;

• the depth of the neural network through ResNet architectures (i.e., 8 or );

• the learning rate : from -to -. . . Proposed method . .

Model training

As mentioned above, the parameter optimization was done in two steps. For this purpose, we defined four combinations grouping the frame duration (i.e., . and . s) and network depth (i.e., ResNet 8 and ResNet ) parameters. The four combinations assessed during this model training steps are given in Table . with their designations, the step they have been optimized, as well as their parameter values.

The first training is used to identify the learning rates giving the the highest precision for each of the four combinations. Then, using these learning rates, the best combination, i.e. the best model, was identified by -folds cross-validation. The best combination is defined with the best average precision because we want to maximize the number of true positives in the classifier output (i.e., to ensure that the sounds predicted as cries are actual cries).

fi

- . R N W _RESNET 8 -, -, -, - . s 8 W _RESNET W _RESNET 8 . s 8 W _RESNET
Table . : Definition of the assessed combinations.

All models are trained for a maximum of epochs. However, to reduce the calculation time, three thresholds were set up and the training is automatically stopped when:

• the loss value is not improving for consecutive epochs;

• the differences of the consecutive epochs are less than -.

• the loss value is less than -.

Best candidate combinations selection

This step is used to limit the number of calculations. Therefore, we compare 6 models corresponding to the four defined combination (i.e., frame durations and network depths) trained with the following learning rates: -, -, -, and -. Through a simple train/validation strategy, we want to identify, the learning rates associated with the highest precision achieved on the validation set for each combination (i.e., W _RESNET 8, W _RESNET , W _RESNET 8, and W _RESNET ). For that purpose, the 6 models are trained with the same train/validation datasets which are presented later in this chapter.

Chapter • Classification for cry detection

Final combination selection

Once the learning rates are identified for each combination, we use -folds cross-validation to select the best model. Cross-validation is the most popular method used to detect problems such as under-or over-fitting and to ensure the robustness of the model. It is a resampling method that uses different parts of the data to test and train a model over different iterations.

In our case, the database is re-sampled into folds, which, during iterations, are successively placed in the train or test sets, the process is depicted in Figure . . .

SoundAnnoT: database creation

A convolutional neural network is a supervised type of Deep learning algorithm which means that annotated data are required to train and validate the model. In this section, we present the SoundAnnot software that we specifically designed for annotating sound segments to create a training database for our network.

Thanks to SoundAnnoT it is possible to carry out simple and fast annotation of sound segments derived from the segmentation method with predefined labels. Annotations are performed only one time for each sound segment by a human through hearing and visual inspection of audios.

Initially created to simplify the process of annotating sound events, SoundAnnot was then de- .

. Interface

The software is composed of two main panels with the left one related to the audio information while the right one is dedicated to the annotations. In particular, SoundAnnoT is composed of the following components:

. an audio player for listening to the current sound and navigating within the previous annotations (see Figure . );

. a spectrogram of the sound with a representation of the spectral components between and Hz as they vary over time, useful for visual support;

. an annotation panel for label selection of the sound currently played. .

Labels

As we have discussed in the previous chapter, the sound segments resulting from the segmentation step are not just cries but includes also many other sounds which occur in the NICU environment. Thus, seven labels were defined to annotate sounds including cries, cries with other sounds, other baby sounds, alarms, other human sounds, other sounds, and mixtures of non-cry sounds. Detailed descriptions of these categories are given in Table . . Although in this work we focused on infant crying, annotating subcategories can help to know better the sound environment and to analyze and understand what leads to incorrect detection.

Hence, the annotation panel is composed of two sub-panels corresponding to the two classes of the binary classifier:

• upper panel for sounds containing crying (cry) ;

• lower panel for sounds unrelated to crying (non-cry).

. . Annotations

One annotation consists in assigning a label to a sound segment. When clicking on one of the annotation buttons, the following actions occur:

• the selected label is automatically saved;

• a new sound segment is automatically played and its spectrogram displayed.

In case a mistake has been done in label selection, annotation can be corrected by going back to the previous audio segment using the previous button in the audio player panel. Corrections are automatically saved when clicking on the new label.

In addition, when the sound source is not clear and it is difficult to choose the label, it is possible to click on the "?" button next to the label that seems most relevant. This allows us to take into account the fact that the annotator doubted while still requiring a label choice.

. . SoundAnnoT: database creation . .

User procedure

To annotate a large number of sound segments, SoundAnnoT was provided to non-expert volunteers who signed a confidentiality agreement. As the users were not familiar with the sounds occurring in the NICU environment, a procedure was set up (Figure . ) to guide them in order to obtain homogeneous annotations.

The software is composed of a homepage allowing user identification. Thus, any new user needs to complete a mandatory training phase before starting an official annotation session used to build the database. These two sessions are described below.

Training session

The purpose of the training session is to familiarize the user with the software and the sound environment present in the NICU. When registering for the first time, the user goes through the following steps:

• a software tutorial, explaining the different components and how to annotate;

• sound samples to listen to for each of the defined label categories;

• the training session in which sounds have already been annotated by an expert.

Once the user has annotated all the sounds, a score page allowing to listen again to the sounds is displayed. Sounds whose annotations differ from the experts are highlighted in red, yellow, or blue depending on the severity of the error. An example of this page is provided in

Official session

In an official session, the user is invited to annotate consecutively sounds chosen randomly from the database of unannotated sound segments. Each sound segment is annotated only once, however, the user can indicate uncertainty by clicking on the "?" button corresponding to the label that seems the most relevant (see the interface in Figure .8.

At any time the user can leave the session which is automatically saved. Once the session is closed, a statistics page is displayed with the duration and amount of annotations made during the current session as well as the overview of all previous sessions.

The database annotated thanks to the SoundAnnot software is described with the number of volunteers, babies, and sounds in the following section.

. Results

This section presents the database annotated with SoundAnnoT software on a population of infants detailed in the first section. Then the learning strategies recently explained are evaluated.

The first step aims to identify the best combination candidates while the second one aims to select the best final combination, i.e., the best model according to the averaged sound precision performance. At last, the chosen model is also assessed on a new cohort of infants never seen before.

. . Results

. .

Annotated data

We selected a dataset representing a large part of the diversity encountered in the framework of the European project Digi-NewB. Hence, 8 recordings were selected, performed in four hospitals:

Rennes, Angers, Brest, and Tours, in both types of beds: open or close. They involve boys and girls born between +6 and + GA and recorded between + and + PMA. Some From each audio recording, sound segments were extracted using the segmentation method (see Chapter ). Among the resulting audio segments, a total of sounds were annotated by a cohort of volunteers using the SoundAnnoT software. Therefore sounds were classified according to the seven labels defined previously in Section . . . We mention here that sounds annotated with a doubt "?" were not taken into account in this database to avoid training the model with false labels.

In addition, for each annotated sound, spectrograms were computed and frames of . and . s duration were extracted and saved leading to the construction of two datasets including respectively and 8 frames. Each frame was given the same label as the sound 8 Chapter • Classification for cry detection it was originally extracted from. Thus, the sound and frame databases are detailed in terms of quantities and percentages according to the seven labels in Table . : Sound and frame databases annotated according to seven defined labels.

During the two training steps, we chose to ignore sounds with the cry+ label (i.e., sounds of cries mixed with other sounds) to derive a confident model trained on pure cries. We believe that this strategy helps the model to learn the intrinsic cry characteristics. Thus, the binary classifier is composed of the cry class containing the sounds labeled cry, while the non-cry class merges the remaining sounds, i.e., those with labels: baby others, voices, alarms, mixtures, and others.

We divided the database into two sets. A training set composed of babies is used to optimize the models through two training steps and a test set is used to ensure the generalization of the model. The data distribution of the two classes and the two sets is reported in Table .6. In this table, one can see that there are more sound segments labeled as non-cry than cry. Since the dataset is imbalanced, we chose to use a weighted argument in the calculation of the crossentropy loss with values corresponding to the data distribution, i.e., .66 for the non-cry and .

for the cry class (see Table . )..

At last, the best combination is trained on all the data in the training set. Then, the trained model is deployed on the test set composed of sounds from three babies never seen before. To assess the good generalization of the model the test is performed twice, first excluding cry+, then including cries mixed with other sounds. The detailed dataset is also reported in Table .6.

. . Best candidate combinations selection

The selection of the four best combinations is performed using a simple strategy with data divided into two sets: babies used for the training and baby for the validation. While the detailed training and validation sets are reported in , the metric performance (obtained based on the predictions on sounds resulting from the validation) are detailed for the 6 models in • W _RESNET 8 with a learning rate of -and a precision of 8 %,

• W _RESNET with a learning rate of -and a precision of 8 %,

• W _RESNET 8 with a learning rate of -and a precision of 8 %,

• W _RESNET with a learning rate of -and a precision of 86%. . .

Final combination selection

Once the four best candidate combinations are identified, the second step of training is performed on the cohort of babies divided into -folds of six babies. The distribution received particular attention so that in each set there is a variety of centers, types of beds, PMA, and GA so that there is a good balance in the amount of available data ( ). Results of the averaged -fold cross-validation are presented with numerical results in The model with the highest averaged precision score is the one using the database with spectrograms framed over . s, using a ResNet architecture and an initial learning rate of -.

From the results, one can see the high prediction score of 86% meaning most of the sounds predicted in the cry class were actual cries. Moreover, it can be noticed that the recall value is also quite good (sensitivity of 8 %), which proves that only a few cries will not be detected. Hence, this model meets our objectives for sound classification since it gives sufficient prediction of whether a sound belongs to the cry or non-cry class based on the values of balanced accuracy and f -score which reach 88% and 8 % respectively. . .

Deployment of the final combination

Finally, the selected model is trained on the babies and evaluated on the test set composed of three new babies. To assess the good generalization of the model, the test is performed twice. Firstly by excluding cry+ (Table .8a) and then by including cries mixed with other sounds (Table .8b). The detailed confusion matrix with all labels and the resulting binary confusion matrices excluding the cry+ labeled sounds and including them are reported in Section . . .

Thanks to a training achieved on the full training set (i.e., babies), the results are better than those of the cross-validation, with % precision and 88% recall when cry+ is not taken into account. These results demonstrate the good ability of the model to generalize when deployed on new data. As for the deployment with the cries mixed with other sounds, one can see that the precision increased reaching a score of % whereas the sensitivity decreased a little (86%).

These results indicate that the model is relatively good at detecting crying in sounds containing multiple sound sources.

Chapter • Classification for cry detection 1 - . 8 . . 8 8 .
(a) Excluding the sound labeled cry+ in the cry class.

1 -.

. 66 .8 8 .

(b) Including the sound labeled cry+ in the cry class.

.6 Conclusion

This chapter was the natural continuation of the previous one focused on segmentation in which we observed that several sound segments were extracted including cries but also other sounds.

Thus, we proposed in this chapter a classification approach based on deep learning.

To fulfill the objective, we presented the SoundAnnoT software that we created to annotate sound segments derived from the segmentation step. Thanks to this program, a database was designed to gather a total of sounds annotated according to seven labels. This database was very useful for the training of the CNN which requires a lot of data to achieve good performance.

This database constitutes the first and quite important result of this chapter as it contains a large variety of sound events recorded in the NICU in four different hospitals. It offers, to our knowledge, probably the first large annotated database of sounds and cries acquired in a real environment.

Moreover, thanks to this database, we assessed the designed cry classifier with different parameter combinations. Based on the ResNet architecture, known to have good performances in image classification, the final model was trained on spectrograms divided into . s frame duration and with an initial learning rate of -. This model gave good validation results. In addition, its robustness was evaluated through a deployment on a test set composed of three new babies. Once again, the results showed good performance either when excluding or including cries mixed with other sounds by reaching up to .6% of precision and 8 . % recall, with a balanced accuracy of . %.

Therefore, our model accuracy achieved equivalent performance to those reported in the literature ( . % accuracy in [ ], 8 .8% F -score in [ ], and 86.6% accuracy-precision score in [ ]) and it is worthwhile to mention that we overcome the given limitations regarding the representativeness of the training and evaluation datasets thanks to our annotated database. Then, although the deployments are different, we can notice that we also obtain a higher accuracy (i.e., %) than the previous study conducted by our team which achieved a score of . % [ ]. In addition, by selecting spectrograms with a full frequency band as input to the CNN, we overcame the problem encountered in the classification of some sounds whose spectral energy was not in the frequency band considered.

Furthermore, even if the two-step parameter optimization strategy is not very common, it has proven to be relevant since it avoided the long computation times associated with classical strategies (e.g., grid search) while obtaining a final model with very good performance. Of course, it is also thanks to the transfer learning approach that this strategy could be applied. Indeed, as the ResNet models were already finely optimized, a less greedy optimization could be undertaken.

Having identified that this architecture gives good results, we can consider going further in the optimization of hyper-parameters to obtain an even more efficient model. For now, ours meets the Table . : Detailed database used during the final combination selection using a cross-validation with folds including six babies each. Introduction

In Chapter , we saw that crying is produced by a complex biological phenomenon that is a combination of neural and physiological mechanisms. Thus, the vocal cords variation or the fundamental frequency analysis is particularly interesting since it is intimately related to the infant's neurological development. Hence, it is of great interest to develop methods to automatically track this acoustic parameter to perform infant cry analyses.

This chapter is therefore part of the continuity of our processing chain whose objective is the automatic analysis of infant crying. Indeed, we present a new fundamental frequency characterization method to track the main spectral component of infant cries which were automatically extracted thanks to the methods proposed previously.

After a review of the state of the art, we introduce the automated method based on contour detection in spectrograms for the tracking of the fundamental frequency. Furthermore, a validation of the method is carried out by comparing our results with those obtained with the BioVoice software whose performances were validated on synthetic basic melodic shapes of the newborn cry [ ].

.

State of the art

A cry signal is known to be a periodic signal, which is a signal that repeats itself at a specific time interval called the period. The fundamental frequency is defined as the inverse of this period, while the harmonic frequencies are defined as integer multiples of F .

When working with continuous, stable, periodic signals, acoustic analysis can be quite simple.

However, real-world signals, such as speech or infant cries, are not perfectly periodic, which makes their analysis more complex. Indeed, due to the intrinsic periodic variations in time, it is not relevant to characterize these signals with a single frequency parameter. However, according to the quasi-periodicity assumption, it is possible to assume that these signals are periodic in very small time frames. Therefore the most common cry characterization consists to track the frequency components along the cry unit by computing features for each of these frames.

. . Methods

There are several techniques to solve the problem of fundamental frequency estimation, such as temporal, spectral, wavelet, and image domain approaches. Below is an overview of the most common methods used in the field of infant crying analysis.

Time domain

Regarding the time domain, the auto-correlation function is the most largely used method to estimate the fundamental frequency in cries. It is a measure of self-similarity of a signal in the

Spectral domain

In early studies, frequency feature characterization used to be based on spectrographic analysis through visual inspection of sound spectrograms [ -]. Then, through the emergence of computer audio signal processing methods, it has become possible to use automatized estimation methods. The analysis of a signal in terms of frequency is done thanks to the conversion of the signal from the time to the frequency domain by using Fourier Transform.

Most of the time, energy features are directly computed from spectrum and peak-picking procedures were implemented to extract F or resonance frequencies [ 6]. Although simple to implement, these methods are not suitable for complex cases such as when there is background noise in the recording or when the harmonics of the cry have a much higher intensity than the fundamental frequency.

In speech analysis, Long Time Average Spectrum (LTAS) is used for the identification of pathological speech. The calculation of average spectra allows LTAS to eliminate short-term variations present in the human voice due to the filtering properties of the vocal tract. In particular, it has been proven that this method provides a good representation of the acoustic signal with minimal Chapter • Fundamental frequency characterization influence of the vocal tract in order to better distinguish between different types of vocal behavior in infants, as well as between healthy and unhealthy infants [ -].

For their part, Varallyay et al. used the smoothed spectrum method which is a very accurate algorithm for detecting the most probable value of the fundamental frequency. It is based on the spectral analysis and is usually combined with noise filtering and statistical processing [ , ].

Wavelet domain

The wavelet transform is another way to transform the audio signal from the time domain into a time-frequency (more precisely time-scale) representation. It calculates the inner product of the signal with a wavelet family. There are two types of wavelets: the continuous and the discrete wavelet transform. Both have the ability to extract information from non-stationary signals such as audio. In addition, thanks to their variable time-frequency resolutions these transforms can overcome the shortcomings of STFT which has a uniform resolution.

Although this approach has proven to be useful in the estimation of the fundamental frequency in adult voices [ , ], it is still very little used in infant cry analysis. Only the Italian team of Manfredi et al. proposes continuous wavelet transform approaches, known for their robustness to noise [ , ].

Image domain

Eventually, some methods are based directly on the extraction of parameters in the spectrogram image. This is for example the case of the Scale Invariant Feature Transform (SIFT) which is initially a feature extraction algorithm in computer vision used to detect local information in images. Manfredi et al. also proposed a tuned method of the algorithm [ 6] which gave better F estimation than the original method [ ] also used in [ 8].

. . Softwares

Nowadays, the most popular software in acoustic analysis is PRAAT [ ]. Initially designed for adult voice by Boesrma in , it was then used in [ , -8] for fundamental frequency estimation, in [ -] for frequency features (such as MFCCs) and in [ 8] for noise filtering and segmentation of the recordings into useful and non-useful categories. Next, the openSMILE tool also allows the extraction of acoustic parameters [ -]. Both of these programs perform the automatic calculation of a wide variety of features (e.g., F , formants, MFCC, LPCC, jitter, shimmer) but must be initialized manually to give a meaningful analysis, particularly when analyzing infant cries.

For their part, Manfredi et al. developed BioVoice [ ,6,6] and WInCA [ ], two programs developed for infant cry analysis, where different estimation methods of F (respectively, SIFT

. . State of the art and wavelet) and resonance frequencies (respectively, peak picking in the power spectral density and wavelet) were implemented.

. .

Limitations on the fundamental frequency estimations

Cries are quasi-periodic signals with high-energy harmonic components. Therefore it can be difficult to estimate the fundamental frequency with the reported methods which are based on signal energy computation. Indeed, methods such as auto-correlation sometimes detect peaks that may correspond to harmonic components. To prevent such jumps in the tracking, authors usually limit the F estimation within a fixed frequency band which is mostly set between and Hz [ , , 6] corresponding to phonation cries [ 8]. As a consequence, this frequency band in which the tracking is performed is of utmost importance and has a great influence on the results [ ]. The only method proposed to our knowledge without frequency limit is the YIN algorithm which has been developed for speech or musical sounds. It was used to extract F features for the purpose of automatic segmentation of infant cry signals [ ].

To illustrate this issue, we give two specific cry examples for which the fundamental frequency tracking was performed using the BioVoice software. This program is a user-friendly software tool for the acoustical analysis of the human voice and is described later in this manuscript (see Section . . ). In both cases, the F was computed within the fixed frequency band ranging from to Hz [ ]. In the first sample, one can see that around the edges the high energy harmonic components affect the tracking with a shift of the estimation up to these components (Figure . ). While, in the second sample, the F is impossible to track since the fundamental frequency of the hyperphonation cry is outside the analyzed frequency band (Figure . ). • Extraction of the spectrogram maximums For each frequency row of the spectrogram, the point with the maximum amplitude is extracted. Result is called the maximum curve (c max ) which is, therefore, of the same size n= than the frequency vector.

• Local maxima detection Local maxima are sought in the c max curve. In order to avoid detection of peaks that are not main frequency components, only peaks whose amplitude is greater than % of the maximum amplitude are detected. Furthermore only peaks separated by more than Hz are retained since infant cries are harmonic signals. The fundamental frequency is detected at the lowest peak based on the frequency scale.

• Local minima detection Local minima are also sought in the c max curve. In order to avoid detection of irrelevant minima, a threshold is chosen as % of the amplitude of the selected maximum peak. Local minima surrounding this peak are chosen to be the new limits for the chosen frequency band.

When the first maximum has been detected close to the initial bound ( or Hz), one surrounding minimum might be missing. In this case, the missing bound is set to the initial corresponding one (such as illustrated in Figure . where the lower bound is set to Hz).

• Spectrogram filtering Spectrogram values with corresponding frequency above or under the new limits are removed (i.e., set to NaN).

. .

Contour detection

Contours are detected using a low-level contour matrix computation where isolines are calculated over cross-sections of the spectrogram magnitude with respect to the time-frequency plane .

Evaluation strategy

To validate the proposed method, we chose to compare our results with those obtained with a method from the literature. While PRAAT is today the most common software used in speech analysis, we computed the frequency estimation with BioVoice. Indeed, this software has proven . .

BioVoice software

BioVoice is a user-friendly software tool for the acoustic analysis of various vocal emissions, from newborns to adults and singers [ ]. It is designed in MATLAB and distributed free of charge on GitHub. First, a sound file must be loaded. It can either be:

• a long recording in which case BioVoice applies a segmentation and then characterizes all the extracted sounds;

• sound segments which are directly characterized.

In both cases, an initialization step is necessary to inform the software of the type of signals to process (i.e., newborn, child, adult).

Software execution

In case of a long recording, the software automatically process a segmentation step to detect the voiced and unvoiced segments in the input signal [ ].

Then pressing "Start" button launch the estimation of more than acoustic temporal and frequency parameters based on advanced and robust analysis techniques. We can mention the following ones: detecting the number, length, and percentage of voiced and unvoiced segments and calculating the fundamental frequency, formant frequencies (F -F ) [ ], noise level, and jitter [ ]. Specifically, for newborn cry and child voice, it computes the melodic shape of F , automatically identifying up to melodic shapes [ , , ].

As for the F tracking it is performed by means of a two-step algorithm [ 6]. After applying the SIFT to time windows of short and fixed length, the fundamental frequency is then adaptively estimated on signal frames of variable length through the average magnitude difference function within the range provided by the SIFT. Therefore, the resulting estimation vectors all have different time steps varying, according to our observation on the processed cries, from to ms.

The computed acoustic parameters are saved in Excel files for each sound segment. The one containing the F estimation is saved in a file named {audioname}_F .xls.

8 Chapter • Fundamental frequency characterization

Cry characterization for comparison

To perform the method comparison, cries are directly loaded in the platform using with parameters adapted to infant crying analysis Age -Range: Newborn/Infant and Voice Emission: Cry (see

Figure .

). Then cries are characterized by the acoustic parameters.

Limitations

Despite the real interest of the BioVoice system, this platform is not adapted for real-time processing in the NICU environment since it requires manual interactions. In addition, the calculation of the numerous acoustic parameters proposed can be very long.

. .

Qualitative comparison with BioVoice

The proposed method is compared to the BioVoice through a visual annotation of both fundamental frequency trackings superimposed on the spectrogram.

Thanks to a graphical interface developed under Python, we annotated the F estimations considering two aspects:

• first, the accuracy of the estimates. In practice, it means that through a visualization of the signals, we judge if one, both or none of the resulting estimation are correctly tracking the fundamental frequency;

• in a second step we assess whether one method is better than the other or if both estimations are equivalent. . .

Statistic parameters comparison

Usually in cry analysis, fundamental frequency is described in terms of statistical parameters.

Therefore, for each cry, the F mean, median, standard deviation, maximum, and minimum values are calculated for both methods. Then the Pearson correlation coefficient is computed to measure the linear correlation between the two datasets.

. Results

This section presents the cry database annotated with the Python interface. Then the qualitative and statistic comparisons are performed. 6 %). While BioVoice appeared to give better results for 8 cries ( . %), our method was reported with a better estimation in 6 cases ( . %).

Since fundamental frequency tracking estimations vary greatly according to the acoustical cries characteristics (energy components, type, melody), the following section presents examples for the three comparison types: when i) both estimations are equivalent, ii) BioVoice is better and, iii) the proposed method is better. BioVoice method is better -The BioVoice method happens to be better in some cases where the proposed method failed to track the fundamental frequency along with the whole cry unit. The F tracking can be missing at the end (left), in the middle (center) or at the start (right) of the cry such as in illustrated in Figures . . -In some cases, the proposed method estimates the fundamental frequency with a small shift in favor of low frequencies Figures . . . . Results

The proposed method is better -Harmonics are defined as multiples of the fundamental frequency, they occur in phonation cries and can have high energy. Therefore, the estimation resulting from BioVoice, which is based on energy, tends to jump to these upper high energy frequencies whereas the proposed method provides continuity in the estimation. (Figures . ). -Subharmonics or double harmonic breaks are defined as a simultaneous parallel series of harmonics in-between the harmonics of the fundamental frequency [ , , ]. In the case of such cries, the proposed method provides continuity in the estimation whereas BioVoice tends to jump to these in-between frequencies (Figures . ). Chapter • Fundamental frequency characterization -Due to the segmentation method, some audio segments may contain several cries. In such case, the proposed method correctly detects and estimates the cry units whereas BioVoice tends to track the whole segment as a single cry unit (Figures . ). At this stage we computed Pearson's coefficient ( This time, Pearson's coefficient revealed a high positive linear correlation between the maximum, mean, and median values. Indeed, the overall comparison shows similar results for these three parameters in both methods. One can note that values resulting from the BioVoice method tend to be lower than ours and this result is exacerbated for the minimum parameter. According to the examples given when our method is better Section . . this tendency can be explained by the many cases where the BioVoice F estimation drop to low-frequency noise.

6 Chapter • Fundamental frequency characterization

.6 Conclusion

In this chapter, we introduced a new method for fundamental frequency tracking of infant cries in the context of real-time monitoring in the NICU. The particularity of the proposed method consists of an initial step performed to automatically find the relevant frequency band in which to perform the F tracking. Indeed, this band has been proven to be very important to achieve good estimations. Then, once this band is computed, the fundamental frequency tracking is performed using a contour detection in the spectrogram.

For validation of the proposed method, we compared our F estimation results to those computed by the BioVoice software which we identified as the reference program for cry analysis. Indeed, the method developed by Manfredi et al. achieved good performances on synthetic basic melodic forms of newborn cries [ , ].

With a selection of 8 6 cries recorded in a preterm infant, we evaluated results, both, in terms of a quantitative visual comparison as well as a common parameters comparison. First, our visual inspections of the fundamental frequency tracking superimposed on the cry spectrograms showed good results with correct estimation rates of 8 % with BioVoice and % with the proposed method. Then, comparing both methods, we reported that F estimations were equivalent for .6 % of the cries evaluated and that one of the two methods was better than the other in . % of the cases for BioVoice and in . % for our method.

In addition, the visual comparison of the common parameter distribution showed similar results

for both methods except for the minimum which was generally lower in BioVoice. We explain this difference by the fact that despite some jumps in the F estimations, cries were considered as correctly detected during the visual annotation. Indeed, for some of the BioVoice mis-detections, we considered the whole signal without taking into account the small jumps occurring in the cases of high energy harmonics, vibrations, or low-frequency noises for example. Nevertheless, through a calculation of the Pearson coefficient, we found a linear correlation for the maximum, mean, and median parameters when comparing the cries which were judged with a correct visual estimation.

Therefore, in this chapter, we presented a new automatic fundamental frequency tracking method for the purpose of long-time monitoring in the NICU. Thanks to the initial step, we ensure that we perform an estimation of the F in a relatively relevant frequency band selection, which has been proven by results consistent with the literature. Yet, if this step seems relevant for future calculations of harmonics, we also believe that the parameters defined here through experimentation will require optimization later.

.

Conclusion Introduction

In the last three chapters, we have presented the methods developed to build an automatic processing chain for the automatic analysis of infant crying. As we have seen, the automation of such a process is a real challenge considering the complexity of the sound environment where the recordings are made as well as the quantity of data to be processed. Therefore, we proposed a two-step crying segmentation method composed of a sound event extraction step followed by a classification of these events to detect crying. Once the cries are extracted, it remains to characterize them, thanks to the fundamental frequency tracking which is performed by a contour detection in the spectrogram.

The objective of this last chapter is therefore to present results from the deployment of the automatic processing chain for the purpose of crying analysis in a routine care environment. After a brief review of the literature on previous studies on the topic, we present the database used and the work that was done to manage the data. Then, in a first step, we propose to compare our results with those presented in the literature and performed by semi-automatic methods on preterm infants. These assessments are important because they compare the results of well-defined trials with our results obtained in the NICU environment. Thus, they show a real interest in using our strategy in a clinical context. Finally, we present new insights into the duration and fundamental frequency trends in time for the whole studied database. These results have also a very valuable clinical impact because this is the first time that such longitudinal trends of normal evolution cohorts are drawn.

6.

State of the art

In this review, we focus on the studies related to our objective which is the evaluation of the maturation through cry analysis in preterm newborns. The characterization of crying in preterm infants has been extensively explored for i) the assessment of the evolution, ii) the early detection of pathologies, and iii) the comparison between full-term newborns. For the latter case, studies attempted to explain the differences observed in their neurophysiological maturity and the subsequent impact on their language development. As mentioned in Chapter , while studies investigated pain-induced crying, studies in recent decades concern the analysis of spontaneous crying and both topics are described in the following sections.

. Pain induced cries

The first studies were not automatic and consisted of audio recordings performed at the induction of pain followed mostly by spectrographic analysis [ -]

Deployment of the proposed methods

In the following sections, we present the deployment of the overall processing chain for the purpose of crying analysis in a routine care environment. After presenting the data processed by the three methods, namely i) audio-video segmentation, ii) classification for cry detection, and iii) fundamental frequency characterization, we present our results. First, we propose to replicate existing studies, and then we give new perspectives for longitudinal tracking.

6. . Deployment of the proposed methods 6. .

Database

A part of the Digi-NewB database was selected to drive a maturation study. To perform this selection, a rigorous examination of clinical records was made by clinicians in order to identify a subset of newborns without pathological development. Such a medical inspection is timeconsuming as it requires observing the entire journey of the infants during their hospitalization.

As a result, newborns were selected and divided into five groups depending on the prematurity severity:

• ( ) -newborns with GA between weeks and weeks + 6 days;

• ( ) -newborns with GA between 8 weeks and weeks + 6 days;

• ( ) -6 newborns with GA between weeks and 6 weeks + 6 days;

• ( ) -8 newborns with GA between weeks and 8 weeks + 6 days;

• -( ) -newborns with GA greater than weeks.

For each newborn, we processed all available recordings ranging in duration from a few hours to consecutive days (related to the birth date, see the maturation recording protocol described in

Section .

). It corresponds to recordings with a total duration of days, hours, and minutes or 6 WAV files of -minute duration. The database is illustrated in Figure 6. with the detailed available recordings for each baby.

Since we work on a very large database, special attention was paid to data management. In the following sections, we give details of the data used through the whole processing chain.

. Audio-Video segmentation

The sound segmentation step applied to the recordings belonging to the babies returned 8 6 sound segments corresponding to days, hours, minutes, and seconds.

Thus, this step considerably reduces the amount of data to be further processed since the total duration of the sound segments corresponds to almost % of the total recording duration.

Then, when collecting only the sound segments occurring within motion periods, the number was reduced to 8, corresponding to a total duration of days, hours, minutes, and 8

seconds. This step also reduces the data since only % of the sound segments occurred within the periods detected with babies' motion.

. Classification for cry detection

To perform the automatic classification step, the selected sound segments are transformed into spectrograms and split in windows of . s with a % overlap, which gives a total of 8 images. After the classification of these spectrograms and the reconstruction of the predictions corresponding to a total duration of day, hours, minutes, and 6 seconds. It has to be noticed that no crying was detected for one baby .

. Data management

As the recordings vary in duration (from a few hours to consecutive days), we decided to arrange the data in, what we decided to call, periods of up to hours. Thus, the records whose duration exceeds this limit are split into several periods. Therefore, we now consider 8 periods recorded in 6 babies. For each period, the post-menstrual and postnatal ages are computed at the starting date of the corresponding periods.

From there, the detected cry distribution in a period varies from one cry to 68 cries. To ensure consistency in the cry analysis, we chose to remove periods in which fewer than cries were detected. In addition, after listening to some recordings with overlapping cries, we realized that some rooms had several infants. With the help of the clinicians, these recordings (usually in the case of twins) were identified and removed from this study. Removing the recordings guarantees that the cries studied correspond to the selected babies, which is very important for the analyses performed later.

Therefore, the resulting database is composed of periods recorded in babies with cry distribution in a period varying from to 68 cries with a total of 6 cries corresponding to a duration of hours , minutes and seconds.

. Fundamental frequency characterization

The proposed fundamental frequency tracking method is applied to the 6 detected cries, then, for each cry, the minimum, maximum, mean, median, and standard deviation values are computed. Next, for each period statistical values are combined through mean and median values. Finally, all further studies are based on these extracted values considering either periods or infants, in that case, period values are averaged for each infant. This process is illustrated in

Reproduction of existing studies

The proposed work is the first one, to our knowledge, to propose a fully automatic processing chain for crying analysis in the NICU. Nevertheless, as mentioned in the state of the art, very few studies have focused on the crying analysis in preterm infants. In this section, we would like to reproduce some studies from the literature in order to validate the proposed cry analysis processing chain from recordings performed in a noisy context of a routine hospital care environment. To replicate this study, we extracted our data into two subsets that matched at best the conditions of the Manfredi et al. dataset. Considering the gestational age, we selected all the periods corresponding to preterm infants while for birth weight, we selected periods corresponding to newborns for whom weight information was available. In both cases, periods were selected with recordings performed during the first month of life (i.e., PNA less than days), the databases used are described in It is worthwhile to mention that the results proposed in this section are important and introduce a range of possible values for some babies presenting a normal evolution. Further studies would be to assess the fundamental frequency of abnormal babies. The analysis performed on manually extracted cries recorded in 6 babies with a PMA greater than weeks and lower than weeks, showed that shorter gestational age was significantly associated with higher F . These results are illustrated in Figure 6. a, where the very preterm newborns (white circles) have higher fundamental frequency values than the full-term infants (black circles).

To reproduce this study, we selected all the periods recorded at the same equivalent age (i.e., PMA between and weeks). In addition, since Shinya et al. used a frequency range between and Hz, we decided to remove all cries with a fundamental frequency maximum above Hz. The database used is divided according to the GA into three groups (i.e., two preterm groups and one full-term) and is described in Table 6. . When comparing the results, the trends of the three parameters are the same for both cases (mean and median). However, we qualitatively observe that the median case offers results closer to those proposed by Shinya et al. (Figure 6. a). This interpretation is, of course, subjective since it is inconsistent to compare mean and median values.

One can also observe a very similar decay in the regression line for the median of the minimum F but with a significant shift in high frequencies for our method. Indeed, based on the numerical results presented in Table 6. most of our parameters have higher values than those of the original method, especially for the minimum and the mean. This might be due to the frequency range where the fundamental frequency analysis is performed which is lower for Shinya et al.

( -Hz) than us ( -Hz).

While Shinya et al. showed that shorter gestational age was significantly associated with higher F we were unable to retrieve such a conclusion. Nevertheless, despite a total automatic analysis performed in the routine care environment, we were able to reproduce the same evolution.

This opens the door to future comparison to better understand why preterm birth is associated with an increase in the fundamental frequency of spontaneous cries at term-equivalent age. From a clinical perspective, Shinya et al. suggested several explanations. First, it might be due to a longer postnatal period. Second, it might reflect the reduced vagal activity in preterm infants.

They reported that the vagal input has an inhibitory effect on laryngeal contraction and results in vocal fold tightening. Thus, the decreased vagal activity is assumed to cause increased vocal fold tension and higher F .

The Digi-NewB database, composed of ECG, respiratory signals, and cries simultaneously recorded, opens the door to exciting studies that could be performed with our proposed method to assess the influence of vagal activity.

Chapter 6 • Automatic processing for cry analysis: deployment

New cry characterization insights

In this section, we propose new visualizations of the fundamental frequency and duration evolution of the spontaneous cries from our database. First, in their study, Shinya et al. suggested that the increased F of spontaneous cries is not related to the body size, but rather might be owed to their different intrauterine and extrauterine experiences [ ]. Thus, we propose to investigate the difference between F values of preterm newborns recorded at birth and after some time living an extrauterine to a group of infants newly born at term. Then, we investigate the general evolution of cry duration and fundamental frequency with increasing post-menstrual for all populations and with increasing postnatal age for the extreme preterm group. Finally, we propose to observe a longitudinal evolution of the fundamental frequency with increasing post-menstrual age, a representation that has never been done before to our knowledge.

. Fundamental frequency comparison at two postnatal ages

The objective of this study is to compare the fundamental frequency of preterm newborns at birth and with a certain experience of extra-uterine life to infants newly born at term. To perform this study, we separate the database into three subsets corresponding to:

• preterm newborns recorded at birth with a PNA less than days;

• preterm newborns recorded with a PNA greater than days;

• full-term newborns recorded at birth with a PNA less than days.

The data used are described in detail in A violin plot is an attractive way to represent the data distribution since it draws a combination of a boxplot and a kernel density estimate. In the left pair, preterm newborns registered at birth are compared to full-term newborns while, in the right pair, preterm newborns with a postnatal age greater than days are compared to full-term newborns. In both pairs, we use the same full-term infant distributions (i.e., PNA less than days), which are duplicated for better visualization. the results show that the fundamental frequency is quite similar when comparing preterm to full-term infants newly born. However F is higher in preterm infants who have already experienced an extra-uterine life.

. New cry characterization insights

For their part, Orlandi et al. showed higher parameter values for preterm than those of full-term infants even when the preterm reaches a post-menstrual age similar or equal to that of the term infant (between weeks and weeks) [ ]. They suggested that prematurity might create a delay in the neuromotor control development in the preterm infant, who would therefore need more than the expected birth age to fully recover.

. Crying evolution with age

This study aims to observe the fundamental frequency and duration of spontaneous crying evolution for all infants, preterm and full-term newborns, and for all recordings included in the maturation database. Data used in this study are described in Table 6. . Chapter 6 • Automatic processing for cry analysis: deployment In future works, this information could also be coupled with melody analyses that also allow a good detection of these different states.

. Fundamental frequency longitudinal evolution

The comparison with studies from the literature as well as the previous sections clearly showed evolution with GA and PMA. Therefore, in this section, we wonder if a longitudinal visualization of the fundamental frequency with increasing post-menstrual age could be interesting. As the normalized weight and height curves are used to monitor the growth of children, we propose to observe the general evolution of the fundamental frequency when combining all the available data. Therefore, the database used is based on the one described in Section 6. including babies, periods, and 6 cries.

To carry out this study, additional statistical values are calculated for each cry, namely, the four quartiles Q , Q , Q , and Q . Based on the same process as before, the cry parameters are combined by period through the median values and all the fundamental frequency statistical values are averaged over the PMA through a two-day rolling average. The resulting process presenting the pseudo-normal evolution of the fundamental frequency observed in infants with no complication during their hospitalization in NICU is depicted in Figure 6. a.

Once these evolutionary trends are defined, we propose to superimpose the minimum, maximum, and median parameters for an extreme preterm (Figure 6. b) and a very preterm newborn (Figure 6. c) from the cohort.

These results are the first to observe the longitudinal evolution of spontaneous crying in preterm infants. Although we have no conclusion to draw on the proposed examples, we think that, with such visualization, it would be interesting to follow infants and check whether abnormal courses have any impact on cry production.

6. . New cry characterization insights

Conclusion

Thanks to the deployment of the complete automatic processing chain, we reached in this chapter some relevant conclusions. First, we showed a comparison with the literature through two interesting and well-designed studies. While the first one compares preterm newborns for different post-menstrual ages and weights at birth, the second one compares preterm and fullterm infants at term equivalent age. In both cases, our results are consistent with the literature which seems to demonstrate that the proposed signal processing chain is robust even in a noisy environment. Considering this powerfulness, we can encourage further clinical applications as well as the exploration of new issues.

Then, through new visualization of the duration and fundamental frequency evolution, we showed that the cry duration is increasing with increasing PNA and PMA while the fundamental frequency tends to decrease with PMA. Last but not least, we proposed to assess the pseudo-normal evolution of the fundamental frequency observed in infants without complications during their hospitalization in the NICU. This work has never been done before and gives new issues for the evaluation of sepsis and pathology during monitoring in the NICU.

It is worthwhile to remind that this is the first automatic processing chain created and deployed on such a large scale. In fact, while previous studies were based on the analysis of a few hundred cries or a few thousand (see Table . ), we presented results obtained on more than cries.

Conclusion

Conclusions & perspectives

In this manuscript, we focused on the presentation of a complete processing chain for the automatic characterization of cries in preterm newborns. This objective is in line with those of the European project Digi-NewB which aimed at combining clinical signs, physiological signals,

and video and sound recordings in a decision support system for the monitoring of newborns.

As a reminder, its two clinical targets were the early diagnosis of late sepsis and the objective assessment of maturation in premature babies cared for in neonatal intensive care units. If physiological signals (cardiac, respiration, . . . ) have already been widely studied to evaluate the risks and evolutions, the analysis of audio and video signals is more recent and tries to bring new clinical indicators. Thus, the work done during this thesis is the first one, to our knowledge, on the analysis of vocalizations in premature infants in a routine care setting in the NICU. This explains why it focused on the development of relevant methods for the automation of audio processing of the data collected during the Digi-NewB project.

The analysis of the literature showed that many studies underlined the interest of analyzing crying in infants to evaluate their neurobehavioral development and thus analyze their maturation stage.

Although teams have already been interested in the spontaneous crying of premature infants, this is the first time that such an audio recording device has been set up in a NICU and that such a database has been created. The acquisition of data in a routine hospital care environment led us to the design of a new automatic processing chain composed of three steps. This chain gathers i) an audio segmentation step using video signal processing to extract only the sounds occurring within the infant's movement; ii) a classification step using a deep-learning approach for the detection of crying among the extracted sounds (adult voices, alarms, etc. and iii) a fundamental frequency characterization step using a contour detection in the spectrogram to track the cry F .

The segmentation method developed was inspired by the one proposed by Orlandi et al. which is based on a calculation of the short-term energy followed by the Otsu method thresholding [ ]. After removing the -minute audio files that do not contain sound, two steps are added to the method to improve it. The first step is a double frequency filter and the second step is a re-segmentation.

The evaluation of the segmentation method in comparison with manual annotations, performed on three -minute files, gave good results. Indeed, we showed that it allows a reliable extraction of events containing cries while reducing the number of extracted audio segments. To go further, we also proposed to use the newborns' motion information computed by another team of our laboratory during the project [ , ]. By focusing only on the sounds appearing in the periods detected as motion, we showed that it is possible to reduce considerably the amount of data to be processed while keeping the vocalizations. Babies also produce a lot of sounds in the presence of adults. However, we have chosen to ignore these periods because of the data quantity and complexity (superimposition of voices and cries, lots of care-related noise, . . . ). The evaluation of this strategy on hours of audio recordings performed on newborns showed that they are very little in motion ( % of the time). We showed that collecting the sounds within these motion periods helped to remove up to 8 % of the segments initially extracted.

Then, the classification method, after the segmentation step, is necessary to identify the cries among the extracted sound segments. We chose to use a time-frequency representation of the cries (spectrograms) as input to a Resnet convolutional neural network algorithm. The classification is thus performed in four steps: ) calculation of the spectrogram by Fast Fourier Transform (FFT) using successive Hamming windows of . ms and an overlap of %, ) slicing the spectrograms into images of the same duration with an overlap of %, ) using the convolutional neural network for the prediction of cries in the images, and ) reconstruction of the sound predictions by retaining the majority prediction on the whole set of images. Thanks to transfer learning, the initial weights of the ResNet model were pre-trained with ImageNet and then optimized to our task (i.e., the crying vs. non-crying classification) by performing new learning. To adapt the model to our data, the parameters of input image duration, neural network complexity, and learning rate were optimized. In a two-step strategy, we first set the learning rate, then the evaluation of several combinations using cross-validation allowed us to identify the model with the best precision. This model corresponds to input images of . s duration, a ResNet architecture, and an initial learning rate of -4 . After being trained again on babies ( sounds), the classification performance obtained on three new babies ( 6 sounds) showed that 8 . % of the initially annotated cries were detected (sensitivity) and that .6% of the sounds classified as cries were indeed cries (precision).

The particularly noisy hospital sound environment (beep, machine, voice, etc.) complicates the task of automating crying detection. Even if by proposing a two-step strategy through the sound segmentation and classification for cry detection, we succeeded in extracting segments containing crying, there are some interesting issues to explore.

First, it is worth remembering that infants can be recorded in an open bed or in an incubator. In the latter case, very fragile premature newborns may have respiratory difficulties and will not be able to produce the same cries as older infants. It could therefore be possible to train a deep-learning model for both bed configurations. It would also be necessary to optimize more parameters of the neural network such as the optimization algorithm, the cost function, or the regularization by degradation of the weights which, in our case, have been fixed a priori.

Secondly, we are interested in sounds that contain several superimposed sound sources. Indeed, in an environment as noisy as the neonatal intensive care unit, it is normal that several sounds are mixed. In this thesis, we focused only on the periods when the baby was moving to limit this type Detection and characterization of vocalizations in preterm newborns of data to be processed. It is surely appropriate to consider methods of source separation that will allow broadening the periods studied and to study the vocalizations of babies in the presence of adults. Once the different sources are separated, it may be easier to detect and characterize the crying segments.

For the estimation of the fundamental frequency characterization, we proposed a new method for tracking the infant cry F in the context of real-time monitoring in the NICU. While methods in the literature typically set the frequency band in which to perform the tracking F Orlandi , Manfredi , Orlandi a, orlandi testing, we proposed an initial step to automatically identify this band. Once computed, the fundamental frequency tracking is performed using contour detection in the spectrogram.

To validate the method, we compared our estimation results to those computed by the software BioVoice which we identified as the reference program for the analysis of newborn cries. In fact, the method developed by Manfredi et al. obtained good performances on synthetic melodic forms of newborn cries [ , ]. A qualitative comparison of the fundamental frequency tracks performed on 8 6 cries showed correct estimations in 8 % of cases with BioVoice and % of cases with our method.

Although the proposed method offers good results in a large part of the cases, it could nevertheless be improved. The relevance of this method lies mainly in the automatic detection of the frequency band. This step is very little implemented in the literature and yet allows to improve considerably the estimation performances. Indeed, by concentrating the analysis band on a reduced frequency range, it avoids jumping to high energy frequency components. The currently proposed method is based on empirical parameters optimized experimentally which should be refined in order to allow the estimation of hyperphonations whose F can be well beyond Hz.

In addition, many studies sought to identify crying either to know the cause [6-] or to identify pathologies [ -]. If these analyses necessarily involve characterization of the fundamental frequency, it is especially based on the characterization of the cry melodies produced by the infants [ , -]. Melody detection and evaluation could not be studied in this thesis but is a necessary step for anyone wishing to continue on the subject.

Finally, the automatic processing chain was deployed on a database of babies born prematurely and at term for a total of days of recording. Thanks to the successive treatments by the three proposed methods, we were able to automatically detect and characterize cries. In a comparison with the literature, we showed that our results are consistent with two studies that inspect the fundamental frequency of crying in ) preterm infants according to their gestational frequency for a population of preterm infants of normal evolution has been described and traced.

These results are a major advance for the evaluation of the maturation of preterm infants during their hospitalization.

In conclusion, if this thesis brings the tools for the evaluation of the maturation and the tendencies of the cry parameters evolution according to the age in neonatal intensive care units, there is still much to be done. Some of these technical improvements have already been listed in this conclusion, but more clinical perspectives are also to be drawn. They are naturally part of the dynamics already studied and will aim to process as much data as possible in order to confirm and reinforce the trends observed and to cover the widest possible period of hospitalization with the aim of assessing possible deviations linked to infections or pathologies. This work will then be the basis for future developments in order to develop a fully automatic solution for a new generation of non-invasive monitoring systems for premature newborns through audio analysis.

Detection and characterization of vocalizations in preterm newborns

Figure :

 : Figure : Illustration d'équipements médicaux nécessaires à la survie d'un nourrisson en USIN.

Figure :

 : Figure : Sources sonores susceptibles de produire du son dans les chambres des nouveau-nés. Pour répondre à cette problématique, nous avons choisi de développer une chaîne de traitement composée de trois étapes. D'abord à l'aide d'une segmentation des enregistrements, on extrait les portions d'audio qui contiennent les évènements sonores, puis grâce à un modèle d'apprentissage profond, on détecte parmi les segments isolés ceux qui contiennent des pleurs. Enfin, on estime la fréquence fondamentale (F ) de ces derniers à l'aide d'une méthode de détection de contours dans le spectrogramme. Ce processus est illustré sur la Figure .

Figure :

 : Figure : Chaîne de traitement automatique proposée pour l'analyse automatique des pleurs.

  bébés nés prématurément et à terme et correspondant à jours d'enregistrement. Grâce aux traitements successifs des trois méthodes proposées, nous avons été capables de détecter et de caractériser automatiquement pleurs. Lors d'une comparaison avec la littérature, nous avons montré que nos résultats sont cohérents avec deux études qui observent la fréquence fondamentale ) des prématurés en fonction de leur âge gestationnel ou de leur poids de naissance [ ] et ) des nourrissons prématurés et des nourrissons nés terme à un même âge post-menstruel [ ]. Ensuite, grâce aux enregistrements longitudinaux réalisés auprès des bébés tout au long de leur hospitalisation, nous avons présenté les évolutions de la durée et de la fréquence fondamentale des pleurs en fonctions des âges post-menstruels et postnatals. Enfin, pour la première fois l'évolution de la fréquence fondamentale pour une population de nourrissons prématurés d'évolution normale est décrite et tracée. Ces résultats sont une avancée majeure pour l'évaluation de la maturation des nouveau-nés prématurés pendant leur hospitalisation.

  Je remercie également chaleureusement le Laboratoire de Traitement du Signal et de l'Image et particulièrement son directeur, Lotfi SENHADJI pour l'accueil qui m'a été réservé depuis mon arrivée. Merci à Soizic, Patricia et Murielle pour tout ce qui concerne le côté administratif de la thèse et aussi pour toutes les conversations pleines d'énergie positive et de sourires qu'on a pu partager. Merci également à Huazhong SHU, directeur du Laboratoire Image, Sciences et Technologies de l'université du Sud-Est de Nankin (Chine). En effet, grâce à la collaboration de ces deux laboratoires, j'ai eu l'incroyable opportunité de réaliser une partie de mon stage de Master à Nanjing où j'ai eu la chance de découvrir une population généreuse, une cuisine délicieuse et une culture merveilleuse ! Je remercie également les étudiants chinois qui m'ont accompagné sur place et notamment ma tendre amie Pan Tan ainsi que mes partenaires de discussion Neil et Sylvia. Je remercie affectueusement Murun de m'avoir acceptée dans ta chambre à la cité internationale de Nanjing et aussi pour m'avoir donné la chance de découvrir ton pays natal : la Mongolie. Je n'oublierai jamais ces voyages alors un grand merci à tous ceux que j'ai croisé. Je remercie également l'équipe pédagogique de l'IUT GEII Rennes qui m'a également réservé un accueil doux et chaleureux et qui m'a fait découvrir le monde de l'IUT. Merci particulièrement à mes collègues Emmanuel et Mickaël pour les grandes discussions sur l'électronique et la programmation ainsi que pour les récits d'aventures époustouflants. J'ai eu la chance de pouvoir voyager à travers des souvenirs précieux alors merci pour le partage ! Un grand merci aussi à Jean-Claude pour toutes les conversations courtes ou longues qu'on a pu échanger au détour d'un bureau, d'un couloir ou d'un café. Notre rencontre à Nanjing a été formidable et je suis ravie d'avoir pu continuer de te croiser et d'échanger avec toi ses dernières années. Ton sourire est lumineux et tes moqueries m'ont permis de lâcher prise dans des moments laborieux. Merci pour ta bonne humeur et ton humour authentique ! Je te souhaite beaucoup de bonheur et peut-être un peu plus de voyages ou d'aventures ? On a qu'une vie alors profitons en ;)

  l'énergie et dō : la voie. À travers vos enseignements, j'ai découvert un art qui m'a transporté à la fois physiquement, par la technique, et spirituellement. Merci Hélène pour les riches explications et nombreux questionnements qui ont profondément attisé ma curiosité. Tu m'as enseigné l'art du mouvement, que rien n'est jamais fixe et qu'il ne tient qu'à nous d'en prendre conscience. Grâce à toi, j'ai ouvert mon esprit et j'ai découvert mon corps. Merci à vous trois pour vos enseignements et vos pratiques et surtout merci pour le cadre convivial et toutes les rencontres que vous m'avez offert. Je pense notamment à toute la fine équipe d'aikidoka du Budo Raji, merci pour les séances à vos côtés et pour les pizzas aussi ! Merci aussi à François pour ta générosité, ta douceur et ta grande sensibilité. Tu fais partie de ceux qui offrent des compliments dans l'instant présent et je tenais à te remercier pour tes mots.

6

  which proposed a new approach of monitoring based on the acquisition of three sources (electrophysiological, clinical and audio-video data) to help clinicians in their diagnosis. During four years, seven teams worked on a decision support system proposed to gather composite indices collected from clinical data and multi-signal analysis, including heart rate, respiration rate, video, and sound signals. The two main aspects of neonatal health targeted were sepsis and neuro-behavioral maturation.

Figure . :

 . Figure . : Age terminology during the perinatal period according to the American Academy of Pediatrics definitions [ ].

Figure . :

 . Figure . : Infant crying mechanisms. Body and brain anatomy and nervous system parts responsible for crying.

Figure . :

 . Figure . : Open and close vocal chords positions.

Figure .

 . Figure . and described below.

  Some of the mentioned prosodic features are depicted in Figure . along a basic cry unit. The signal is represented in time, frequency and time-frequency domains. The usual basic cry formant frequencies are:

Figure . :

 . Figure . : Crying features along a cry unit.

  Crying analysis involves three steps which are data acquisition, signal processing, and feature extraction (Figure .6). In this section, we review the literature on these steps to give the methodological background and thus propose our strategy.

Figure . 6 :

 6 Figure .6: Framework of the acoustical processing chain used in cry analysis.

  Audio signals are usually represented in time and frequency domains, and, recently several studies investigated the time-varying frequency features along with a cry unit. The characteristics derived . . State of the art

Figure . :

 . Figure . : Scheme of the two strategies used to automatically extract cries from noisy recordings.

Figure

  fundamental units. Nowadays, six basic melodic forms of crying are retained and are illustrated schematically in

Figure . 8 :

 8 Figure .8: Basic and schematic melody shapes of infant cries in the time-frequency plane.

Figure . :

 . Figure . : Workflow of the proposed cry analysis automated processing chain.

Formerly

  , mothers used to give birth and care for their infants at home without any medical assistance. Physicians started to take an interest in providing care to reduce mortality due to prematurity during the 88 s, especially with the creation of the first incubator by Dr. Stephane Tarnier at Paris's Maternité Hospital (illustration in Figure . ). This new technology aimed to prevent many premature newborns from succumbing to hypothermia (low body temperature). However, at this point caring for premature babies was expensive and, many thought, pointless.

Figure . :

 . Figure . : Tarnier's incubators in the Maternité Hospital, Paris, 88 . Source: Illustrated London News, 8 March 88 , p. 8.

  Figure . -). In addition to avoiding hypothermia and dehydration, it minimizes exposure to germs and external noise. The use and parameters of an incubator depend on each infant's specific needs. Thermal regulation is based either on the temperature measured on the baby's skin (large variations) or a configured targeted ambient temperature. Then, according to the thermal regulation capacity of the baby, he or she can be transferred to different environments, from a radiant warmer (see Figure . -) to a cradle (i.e., without thermal regulation, see Figure . -).

Figure . :

 . Figure . : Three examples of beds in NICU.

Figure . :

 . Figure . : Non-invasive respiratory assistance: air (enriched or not with oxygen) is delivered to the infant's airway through a nasal mask. The mask is attached to the nose/face with headgear.

Figure . :

 . Figure . : The naso-gastric tube is connecting the nose to the stomach. Milk is delivered by automatic syringes.

Figure . :

 . Figure . :The journey of an extremely premature baby in the NICU. TOP -In an incubator with feeding assistance, i.e., a naso-gastric tube (coming out of the nose) and a venous catheter (with cable surrounding the head) connected to an automatic syringe. The cardiac, respiratory, and pulse oximeter sensors are invisible but present and the respiratory assistance was removed at the time of the picture. The arterial blood pressure armband is visible in the bottom right corner. BOTTOM -In a radiant warmer with noninvasive ventilation and naso-gastric tube. The pulse oximeter is visible with the cardiac and respiratory sensor wires.

Figure . 6 :-

 6 Figure .6: Intensive care medical equipment at the Rennes CHU for a radiant warmer from two different points of view. LEFT -Bedroom global equipment overview. RIGHT -Zoom on the crib. CENTER -Physiological signal representations on a scope.

  Digi-NewB project aimed to improve neonatal care thanks to the development of a new generation of non-invasive monitoring systems in neonatology. Particularly, this setup was intended to assist clinicians in the early sepsis diagnosis and the analysis of newborns' cardiorespiratory and neuro-behavioral maturation. On a larger scale, the main purpose was to decrease the mortality, and morbidity rate, reduce the risk of neurodevelopmental disorders as well as diminish the hospitalization health cost and duration. To fulfill these objectives, the French clinical network Hôpitaux Universitaires du Grand Ouest (HUGO), and our laboratory (LTSI-INSERM) collaborated to recruit five partners from four European countries composed of two companies and four university groups with multidisciplinary expertise. The Figure . shows the key figures of the Digi-NewB project, which received funding from the European Union's Horizon research and innovation program under grant agreement No. 68 6 and was carried out between March 6 and May . The aim was to propose a decision support system gathering composite indices collected from clinical data and multi-signal analysis, including heart rate, respiration rate, video, and sound signals. Audio and video were chosen, on the one hand, because they have proven their relevance in the evaluation of the problems addressed by the European project (see [ ] for a 6 Chapter • Work context

Figure . :-

 . Figure . : Overview of the Digi-NewB project with keys, tasks, and locations.

Figure . 8 :

 8 Figure .8: Digi-NewB acquisition device components.

Figure . :

 . Figure . : Digi-NewB data acquisition system in real-life settings in NICU. Digi-NewB main recording device, including microphone and video cameras, auxiliary microphone and camera, scope, monitoring the physiological signals (heart rate, respiration and oxygen saturation), Digi-NewB computer, handling acquisition and monitoring systems.

Figure . :

 . Figure . : Digi-NewB maturation recording protocol.

  are overlapping in time, which makes the automatic processing of recordings in the NICU very challenging. In this context, Raboshchuk et al. presented the acoustic environment of a preterm infant in NICU (see Figure . ), and addressed extensively the problem of acoustic alarm detection [ -6].

Figure . :

 . Figure . : A general sound taxonomy of a typical NICU. Taken from Raboshchuk et al. [ ].

Figure . :

 . Figure . : The noise sources occurring in the NICU and heard in the Digi-NewB recordings.

Figure . :

 . Figure . : A typical NICU bedroom layout at the Rennes CHU.

Figure . :

 . Figure . : Left: Percentage of minutes considered as "noisy" by voices or alarms within the -hour recording. Middle: Distribution of the noise categories polluting the sound environment. Right: Alarms distribution after discrimination.

  time and are mixed we annotate each of them individually. The boundaries were set at the points where the sound could no longer be heard. A total of sound events have been labeled through the three files. An example of the annotations performed in Audacity software is given in Figure . 6 and the resulting distribution of segment duration by labels for the three files is illustrated in Figure . .

Figure . 6 :

 6 Figure . 6: Examples of annotations performed in Audacity.

Figure . :

 . Figure . : Duration of sound events by the label category for all three recordings combined ( sounds). The four annotated sounds, longer than four seconds, are not displayed.

  based on STE thresholding were investigated in[ , -8].Additionally, Orlandi et al. used two thresholds calculated through the Otsu's method to perform the segmentation [ ]. Since this segmentation inspired our own, an overview of this method is proposed in this chapter (seeSection . . ).Cry segmentation was also performed in combining the two short-time methods, STE and ZCR, [ ] and applying a threshold to extract CU. In the continuity of this study, a third step was added to distinguish harmonic and non-harmonic audio segments [ ]. Some authors also investigated Simple Inverse Filter Tracking (SIFT) [ ] or word reliability [ ]. . . Methods for cry segmentation in long and noisy recordings Then, with the advancement of technologies, longer recordings were made and required new processing methods. Thus, the recent approaches have considered cry segmentation as a classification problem. In these methods, the whole signal is considered and cut into frames. These frames are then classified into different categories according to the studies. For example, Reggiannini et al. [ ] started with a KNN classifier and proposed the three basic classes: voiced part, unvoiced part, and silence. However, it appeared that in long recordings, besides cries surrounding sounds were also recorded in the signals. Therefore, new sound classes emerged. It the case in [ ], where Abou-Abbas et al. considered six classes dividing infant voiced parts (i.e. cries) into expiratory and inspiratory phases with a Hidden Markov Model (HMM). Later, these results were improved by decreasing the Chapter • Audio-Video segmentation number of classes and gathering the non-cry sounds in a class called "others" [ ] or "residuals" [ 6]. The discrimination of the three classes achieved with the KNN resulted in an Area Under the Curve (AUC) of .88 [ ]. In comparison to HMM, Gaussian Mixture Model (GMM) gave the best results with a classification error rate of 8. % [ ], while Naithani et al. reached a total accuracy of 8 . % with HMM [ 6]. Then, researchers began to perform long recordings in real-life settings which are much less controlled. Thus, once again, new methods emerged to deal with the many other sound events occurring, besides the infant's cries, in the recordings. Most of these methods, based on deep learning approaches, are used to detect cries in domestic environment [ -] or in the NICU [ , ]. All these studies worked with a Convolution Neural Network (CNN) and the input layer is computed from Mel-Frequency Cepstral Coefficients (MFCCs) associated with either the Mel-Filter Bank (MFB) [ , 8, -] or the Linear-Filter Bank (LFB) [ ].

  averaged Area Under Precision-Recall Curve (PR-AUC) of % in [ ]. Moreover, the promising results in [ ] were confirmed with considerably better performance compared to a traditional machine learning classifiers (SVM and logistic regression) in [ ], especially for low false-positive rates. While, in the NICU, an average accuracy of 86. 8% was obtained [ ] and a PR-AUC of . % was reached on real data in [ ].

Figure . :

 . Figure . : Segmentation workflow proposed by Orlandi et al. in [ ]

Figure . :

 . Figure . : STE values (blue) computed on a -second signal (grey) containing two cry units.

.

  Next, the upper threshold (T U ) detects the sound event apparition. It is computed on the whole STE value distribution and is illustrated with the corresponding histogram in Figure . a. The segments of value lower than T U are considered as silences and those higher than T U as sound segments. The application of the threshold on the signal containing the two cries is presented in Figure . b. We can see that by considering only the upper threshold, the second cry is mis-segmented and divided into two segments. . . Audio segmentation method(a) T U and STE histogram (for all values).(b) UP -short time energy vector with upper threshold, BOTTOM -segmented audio signal through upper threshold only.

Figure . :

 . Figure . : Upper threshold (T U ) computation.

( a )

 a Segmentation technique illustration where boundary points are detected for all STE intervals that surpass T U . In the picture, the initial point is determined as the first point under T L located on the left side with respect to the sound onset. The final point corresponds to the first point occurring below T L after the sound offset. In this example, the resulting segmentation returns two segments. While segment matches the first cry, segment gathers (thanks to the double threshold method) two detected sounds corresponding to the second cry unit that has lower energy in the middle.(b)

Figure . :

 . Figure . : Lower threshold (T L ) computation.

  relation to the physiological infant voice properties as four times a second is how fast the vocal cords can change and is what is needed to obtain a complete acoustic profile of the newborn [ ]. . . Audio segmentation method Thresholds computation on Digi-NewB data As described in Chapter , in the framework of the Digi-NewB project the audio signals recorded are performed in a routine hospital care environment, and data are stored in -minute WAV files (i.e.

Figure . :

 . Figure . : Sound segments resulting from the segmentation process for a -second noisy audio signal represented on its original time axis.

  alarms, voices, doors, ...). Therefore, it is normal that in such a noisy environment several sounds mix. Thus, the segments resulting from the segmentation might gathers several overlapping sounds since once again the STE values cannot decrease. An example is given in Figure.6b where the resulting audio segment is composed of alarm, voice and cry signals.

( a )

 a Sound segment containing several cry units. (b) Sound segment containing several types of sounds.

Figure . 6 :

 6 Figure .6: Examples of extracted sounds poorly segmented and lasting more than seconds.

Figures . .

 . Figures . . On the left image, we can see the histogram of energy values that are concentrated around a small amplitude. If the two thresholds calculated with this distribution are too low to separate sound from silence, they are sufficient to segment silence (i.e., noise) such as illustrated in the right picture.

( a )

 a STE histogram with low energy values. (b) STE with thresholds.(c) Segmentation vector with many silence/noise segments are detected.

Figures . :

 : Figures . : Segmentation example with an extract from a quiet recording. Since few sound events occur, STE values are mostly located around low amplitude (a) thus, thresholds cannot help to detect sound events (b) leading instead to the detection of many silences/noises during the segmentation step (c).

Figures . 8 :

 8 Figures .8: Segmentation example with an extract from a recording containing numerous sounds. In this case, STE values are spread over the whole range (a) thus, thresholds help to detect sound events (b) that are correctly extracted from the background silence during the segmentation step (c).

  Figure .6a), overlapping sounds (see Figure .6b), or noisy periods. In order to process these segments efficiently, we propose a re-segmentation step within the final duration filtering step (see Figure .

Figure . :

 . Figure . : Flowchart of the duration filtering step with the re-segmentation procedure for each segment with duration greater than five seconds.

Figure . :

 . Figure . : Workflow of the updated segmentation method.

  requires a deep respiratory activity, helps the newborn to communicate discomfort. Thus, both of these aspects can lead to movement. Studies investigating the correlation between crying and movement in preterm infants are mostly related to the assessment of the behavioral sleep stages [ ]. The three usual categories are: active sleep, quiet sleep, and wake. According to a recent study, while the sleep states include reflexive body movements with sobs, sights, and distressing noises, the wake behavioral state includes high body activity level and crying [ ].

  the classification step such as illustrated in Figure . . To our knowledge, it is the first time that video signals are used for crying segmentation. In practice, it means to collect sound segments occurring in specific motion segmentation intervals. Nevertheless, in order to evaluate the validity of this approach, we propose a preliminary study to investigate i) the amount of motion in the recordings, ii) the sound distribution and iii) the cries distribution.

Figure . :

 . Figure . : Audio segmentation strategy by extracting sounds occurring within infant's motion intervals.

Figure . :

 . Figure . : Example of a motion signal with two sample frames acquired when the infant is still (left) and in movement (right).

( a )

 a baby is still. (b) baby is moving. (c) adult is present. (d) baby is absent.

Figure . :

 . Figure . : Illustration of the possible configurations of images in the recordings. The intervals corresponding to baby presence such as images (a) and (b) are processed, while the intervals with adult presence (c) and/or baby absence (d) are excluded.

Figure . :

 . Figure . : Illustration of the motion and non-motion interval segmentation steps (from top to bottom): the raw motion signal, the clean motion signal, the synthetic motion signal resulting from the segmentation, the final segmented motion signal.

  Figure . a and shows that newborns do not move much (i.e. % of time on average) and are mostly immobile (i.e. % of time on average). The remaining time corresponds to intervals where an adult is present or when the baby is absent from the image. While the quantity of non-movement and NaN intervals vary greatly between the 6 recordings (see Figure . b), the movement intervals are steadier and remain lower than %.

  Overall distribution for the 6 recordings.

Figure . :

 . Figure . : Distribution of motion segmentation (including babies, 6 recordings, sound segments).

( a )

 a Sound segment quantity in recordings. (b) All sound segment durations accumulated.

Figures . 6 :

 6 Figures . 6: Sound segments distribution in motion segmentation (including babies, 6 recordings, sound segments).

( a )

 a Cries quantity in recordings. (b) All cry durations accumulated.

Figures

  Figures . : Cries distribution in motion segmentation (including babies, 6 recordings, cries).

Figure . :

 . Figure . : Illustration of the segments comparison parameters.

  threshold T computed over up to a two-hour window. This threshold is illustrated with the black line in Figure . b and we can note that it varies less abruptly than the T U and T L thresholds.Then, for each audio file, the number of intervals whose STE values exceed the corresponding threshold level T is estimated (such as performed to detect sound event intervals with STE values over T U . Files with less than intervals are represented by dots while the others are represented by squares. In this -hour recording example, seven WAV files are detected as files with minimal audio content (such as) and are further processed through the segmentation (such as and ). The three annotated WAV files used in this study are highlighted in blue.Therefore, when considering the whole recording including WAV files (i.e., 66 sounds extracted with NFB+RS), the Long-term threshold step helps to reduce the file quantity to be processed to (i.e., sounds extracted with NFB+RS+LTT).

( a )

 a Upper (grey) and lower (red) thresholds. (b) Illustration of the sliding threshold T computed up to two-hours window depicted by the black line. Files containing less than energy intervals above T are discarded (black squares), while the others are processed (black dots).

Figure . :

 . Figure . : Thresholds computed for every -minute files over a -hours recording . The three annotated WAV files used in this study are highlighted in blue.

Figure . :

 . Figure . : Otsu's method application.

Figure . :

 . Figure . : Optimal threshold computation through Otsu's method.

  Mel-filterbank energy features.

Figure . :

 . Figure . : Illustration of the usual audio features used in sound classification methods. In both examples, four cry harmonic sequences are separated by unvoiced breath of the baby which produces noisy-like sounds in the lower frequency.

  non-cry False Positive True NegativeTable.: Confusion matrix for a binary classifier.(S e ) (R) describes how well cry sounds were classified as cry. It answers the question: "How much of the actual cries were correctly classified ?": ) describes how well actual non-cry sounds were correctly classified as non-cry. It answers the question: "How much non-cry sounds were correctly classified ?": describes how good the classifier is in predicting if a sound belongs to the cry or non-cry class. It is especially useful when the classes are unbalanced and it is computed as the arithmetic mean of the two previous metrics: BAcc = S e + S p ( . )(P) describes how well sounds detected as cry were actual cries. It answers the question: "How mush of non-cry is classified as cry?describes also how good the classifier is in predicting if a sound belongs to the cry or non-cry class. It is computed as the harmonic mean of the model's precision and recall. F = × P × R P + R ( . ). . State of the art .

Figure . :

 . Figure . : Binary cry classification framework based on CNN using spectrograms.

Figure . :

 . Figure . : Resulting spectrogram with temporal and frequency resolutions for cry unit sampled at kHz.

( a )

 a Resulting frames. (b) Resulting frames.

Figure . :

 . Figure . : Illustration of the spectrogram division process with . s duration frames.

  Hence, the ResNet weights were pre-trained with ImageNet to initialize the classification model [ ]. Then weights are optimized to our task (i.e., the crying vs. non-crying classification), by performing a new training through the last fully-connected layer. In our case, this step aims to minimize the cross-entropy loss associated with a class-weighting. In addition, we decided to explore two network depths using the Resnet 8 and Resnet architectures illustrated in Figure . . This approach may appear unusual since it uses natural images to classify sound spectrogram.

Figure . :

 . Figure . : Residual Network or ResNet architectures used in this work. All layers are described with the convolution kernel, the number of output channels and the stride value except the last one which is a fully connected layer with one predicted class in output: cry or non-cry.

  Moreover, since the inputs of the CNN are spectrogram frames, the final sound prediction (P sound ) is computed based on the distribution of the frame predictions (Figure .6) such as: P sound = i.e., cry if the frame majority decision is or balanced, i.e., non-cry if the frame majority decision is . ( .6)To limit the number of calculations, we carried out a two-step parameter optimization strategy with defined combinations of parameters which are explained in the next section.

Figure . 6 :

 6 Figure .6: Sound prediction reconstruction using majority voting.

Finally

  , the cross-validation is performed for the four combinations, and the one resulting with the least variation in performance and with the highest averaged precision is considered the best and final model.

Figure . :

 . Figure . : Illustration of the -folds cross-validation strategy.

Figure . 8 :

 8 Figure .8: SoundAnnoT user-friendly interface.

Figure . :

 . Figure . : Functions of the audio player buttons.

Figure . :

 . Figure . : SoundAnnoT user procedure diagram.

Figure . . 6

 6 Figure ..

Figure . :

 . Figure . : Annotation training score page.

  babies were recorded up to four different dates at least 8 hours apart. The data distribution for all babies is depicted in Figure . with the dots representing the dates of the recordings used. In the top part of the figure, all recordings are merged in terms of GA where one can see the lack of recordings of babies born between and GA, and in terms of PMA where the distribution is quite homogeneous.

Figure . :

 . Figure . : Data distribution used for sound annotation.

Figure . :

 . Figure . : Performance of the four best candidate combinations based on sound predictions obtained on the validation set.

Figure

  Figure . b.

( a )

 a Results in numerical values. (b) Results in barplots.

Figure . :

 . Figure . : Performance of the -folds cross-validation for the four best candidate combinations based on sound predictions.

  time domain when compared to a delayed version of itself. Designed to determine the periodicity of the signal, it was firstly used in [ ] and was then implemented in in the famous PRAAT software [ ] widely adopted nowadays for the F characterization of crying [ , -8]. However, described as hard to manipulate [ ], it can require manual correction of the F estimation tracking errors [6]. For their part, Naithani et al. proposed to use the YIN algorithm which is known to be more robust on quasi periodic signals [ ].

( a )

 a Expected estimation. (b) Resulting estimation.

Figure . :

 . Figure . : Sample -phonation cry with high energy harmonic components.

Figure . :

 . Figure . : Sample -hyperphonation cry with F outside the analysed band.

Figure . :

 . Figure . : Fundamental frequency tracking flowchart.

Figure . :

 . Figure . : Spectrogram computation with frame resolution for signal sample at kHz.

Figure . :

 . Figure . : Automatic frequency band selection framework.

Figure . 6a .

 6a Figure .6a. Main spectral components contours are obtained by selecting the low-height isolines. Moreover, as the melody tends to be continuous over the cry unit width, contours of duration less than . s are disregarded. At last, contours included inside another contour are neglected and when two or more contours are temporally overlapping, only the wider one is kept Figure .6b.

Figure . 6 :

 6 Figure .6:Illustration of the contour detection step with the detected contours (left) and the selected ones (middle). The small right contour is discarded since it is shorter than . s otherwise only the wider contours are kept (i.e., those including other contours). Then, fundamental frequency tracking (right) is computed within the selected contours. Thus, in this example, the resulting vector v is filled with two averaged contours.

. .

 . Evaluation strategy to be relevant for infant cries analysis since their performances were validated on synthetic basic melodic shapes of the newborn cry [ ].First we present the BioVoice software tool for acoustic analysis. Then, we propose to perform a qualitative comparison based on visual annotations as well as a statistical comparison of the usual parameters evaluated in such a study, i.e., min, max, mean and median values of the F estimation.

Figure . :

 . Figure . : BioVoice setting interface -Newborn cry analysis selection.

. .

 . Evaluation strategy To remain as objective as possible, the signals were anonymized in the interface. During the annotation step the signals were displayed randomly in blue or black without any distinction criteria.The annotation procedure is illustrated in Figure.8 with both method estimations anonymized and superimposed a spectrogram. In this example we considered that both estimations correctly tracked the fundamental frequency and that they are equivalent, i.e., there is not one better than the other.

Figure . 8 :

 8 Figure .8: Python interface designed for the annotation of anonymized signals (randomly affected to blue or black color). In this example, both methods achieved similar good F tracking.

  . .Qualitative comparison with BioVoiceThe qualitative comparison of the 8 6 cries, based on the visual annotation of the anonymized results, shows that both fundamental frequency tracking methods have good accuracy with (8 %) cries correctly estimated by BioVoice and ( %) by the proposed method (Figure . a). When comparing the two methods with each other (Figure . b), 6 cry estimations were considered equivalent ( .

( a )

 a Accuracy of both methods. (b) Comparison of the two methods.

Figures

  Figures . : QualitativeF methods annotation results.

  Figures . : QualitativeF methods annotation results.

Figure . :

 . Figure . : Correct estimation examples for both F tracking methods.

Figures . :

 : Figures . : Partial estimation with the proposed method.

Figures . :

 : Figures . : Small frequency shift with the proposed method.

Figures . :

 : Figures . : Bad frequency band selections in the first step of the proposed method.

Figures . :

 : Figures . : Cries with high energy harmonics.

Figures . :

 : Figures . : Cries with subharmonics. -It is common for babies to produce vibrations when crying. Three examples are given in Figures . 6 with vibrations occurring at the start (left), middle (center), or end (right)of the cries. In such cases, the BioVoice method tends to track something within the vibration region while the proposed method correctly estimates the phonation parts only.

Figures . 6 :

 6 Figures . 6: Cries with breaks are estimated in the middle by BioVoice.

Figures . : Figures . 8 :

 :8 Figures . : Segments with multiple cry units estimated as a single one by BioVoice.

Figures . :

 : Figures . : Cries with jumps in the BioVoice frequency estimations due to low-frequency noise.

Figure . :

 . Figure . : Fundamental frequency tracking parameters comparison between BioVoice (left) and the proposed method (right) with F parameter values computed for each cry for the 8 6 cries.

Figure 6 .:

 6 Figure 6. : Illustration of the maturation database detailed for each of the newborns. Babies with stars are recorded in shared-bedroom or co-bedding settings.

Figure 6 .

 6 Figure 6. .

Figure 6 .:

 6 Figure 6. : Fundamental frequency statistical parameters computation workflow.

Figure 6 .

 6 Figure 6. : From Manfredi et al. [ ], boxplots comparing the fundamental frequency for the newborn cry data, divided according to gestational age (g.a.) and weight at birth (w.a.b.). Results consistently show a decrease in frequency with increasing age or weight.

(

  a) F comparison considering GA. (b) F comparison considering w.a.b.

Figures 6 .:

 6 Figures 6. : Boxplots comparing the fundamental frequency for the newborn cry data, divided according to gestational age and weight at birth.

  Results of the fundamental frequency are presented in terms of minimum, mean, and maximum for Shinya et al. in Figure 6. a, and with statistical parameters averaged for each infant with cries values combined per period for our database through the mean in Figure 6. b and the median values in Figure 6. c.

( a )

 a Averaged for each infant of the mean values (n=6 ), from Shinya et al. [ ]. (b) Averaged for each infant of the mean values (n= 6), Digi-NewB database. (c) Averaged for each infant of the median values (n= 6), Digi-NewB database.

Figures 6 .:

 6 Figures 6. : Scatter plots showing the relationships between gestational age and minimum (left), mean (center) and maximum (right) fundamental frequency F of spontaneous cries at term-equivalent age. The groups of infants were VP (white circles), MLP (grey circles) and FT (black circles).

Figures 6 . 6 :

 66 Figures 6.6: Violinplots comparing the fundamental frequency for the newborn cry data, divided according to post-natal age and prematurity status. For each pair, left-hand side distributions (light) represent preterm newborns while right-hand side distributions (dark) represent infants newly born at term. In the left pair, preterm newborns are observed at birth and in the right pair preterm newborns are observed with a postnatal age greater than days.

  Database used to assess crying evolution.Fundamental frequency evolution with PMAHere we present values of the fundamental frequency in preterm and full-term newborns with increasing post-menstrual age. Results of the F are computed with statistical parameters (i.e., minimum, maximum, mean, and median) combined per period through median values. The results are presented in Figures 6. .

  We can observe that mean F tends to decrease with increasing PMA and that the values are more dispersed for low PMA. This is in agreement with the literature. Actually, Tenold et al.suggested that the differences observed in spectral variability between cries of premature and full-term infants probably reflect neurophysiological maturity [ ]. In addition, in their study, Thoden et al. showed, in the case of pain cries, that the more premature a newborn is, the higher the fundamental frequency and that it decreases with increasing post-menstrual age [ ]. A statement in accordance with the previous work proposed byMichelsson et al. who also mentioned that even if the cries of the smallest premature babies were generally high-pitched, they were also sometimes lower-pitched and thus look like the cries of infants born at term [ ].

  Mean fundamental frequency. (d) Median fundamental frequency.

Figures 6 .

 6 Figures 6. : Cry F evolution considering all newborns according to the PMA.

Figures 6 . 8 : 8 .

 688 Figures 6.8: Cry duration evolution considering all newborns according to the PMA.

Figures 6 .: 6 Chapter 6 •From Figure 6 .

 6666 Figures 6. : Cry duration evolution focusing on the EP newborns according to the PNA.

Figures 6 .:

 6 Figures 6. : Averaged fundamental frequency evolutionary trends of the healthy babies without reported complication during their hospitalization in NICU (a). Additional minimum, median and maximum values are superimposed for two babies: (b) an extreme preterm and (c) a very preterm.

  age and birth weight [ ] and ) preterm and term infants at an equivalent post-menstrual age [ ]. Through the analysis of longitudinal recordings from infants during their hospitalization, we presented changes in the duration and fundamental frequency of crying as a function of post-menstrual and postnatal ages. Finally, for the first time, the evolution of the fundamental Conclusions & perspectives

  Je voudrais ensuite remercier chaudement tous les fourbes escapadés d'être mes amis de longue date maintenant. Entre la Turballe, Bordeaux, Montaimont, Paris, Le Mans, Agen, Lyon, Rodez, Montpellier, Oléron on en a vécu des aventures . . . Ces escapades m'ont fait le plus grand bien et ont été une source d'énergie à l'état pur. Merci d'être là les copains et merci pour tous les encouragements et le soutien que vous m'avez donné ! Merci surtout à Ninouche qui a fait le déplacement le jour J et dont la présence est d'un immense soutien. Babylone ma soeur, je n'oublierai jamais notre périple à vélo et tous nos débats existentiels sur le monde actuel ! Bien moi et merci de m'avoir malmené sur les dead-lines, rien de tout cela n'aurait pu être possible sans toi ! Sans oublier les nombreux week-ends et excursions surtout en bord de mer qui ont fait, à chaque fois, souffler un vent nouveau sur mes sentiments embrumés. Tu fais partie de celles qui m'offrent un amour et un réconfort sans faille et tu m'impressionnes par l'énergie que tu déploies dans chacun de tes projets. Pour la source d'inspiration que tu es, je te souhaite de ne jamais te tarir et de tracer ta route. Quelle qu'elle soit, je suis convaincue qu'elle sera belle et qu'elle est vouée à croiser sans cesse la mienne. Merci aussi à tous mes amis cévenols avec qui j'ai partagé ces dernières années des morceaux d'étés et des fins d'années. Merci à Ulysse et Aude d'être ceux qu'ils sont, vous m'avez donné votre coeur et je suis sincèrement heureuse de vous connaître. Merci à Rose et Vincent pour tous les moments de complicité aux repas de famille et ailleurs, avec vous, je rigole librement et je me Mon Cher Ghassan, je suis si heureuse de t'avoir rencontré et d'avoir vécu tous ces moments forts en ta présence. Si ton histoire est dure, tu m'impressionnes par ton énergie et les ondes si positives que tu dégages. Tu es celui qui m'a chanté l'encouragement pendant ces longs mois de rédaction et ta voix résonne encore dans ma tête : "Bertille, Bertille, Bertille". Merci pour ton soutien, merci pour tes mots, merci de m'avoir rassuré encore et encore. Je te souhaite beaucoup de bonheur et d'épanouissement, tu le mérites sincèrement. Justin, ces mots sont les plus durs à écrire, car je n'arrive pas à trouver assez de recul pour mesurer l'envergure de ce que tu représentes pour moi. Je te remercie infiniment de toujours m'avoir soutenue pour tous les projets que j'ai entrepris. Avec tes encouragements, j'ai non seulement réussi à finir cette thèse, mais je suis aussi partie en Erasmus à Prague, en stage en Chine puis en voyage en Corée, au Japon, en Mongolie et en Russie. Merci de m'offrir la liberté d'être moi-même et de m'accepter entièrement. Merci d'écouter attentivement mes peurs, mes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prematurity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	Deux heures durant lesquelles Ellana batailla ferme pour avancer. Batailla contre la montagne et contre ses chaînes. Deux heures de combat épuisant où elle prit des risques incroyables. Deux heures passées sans échanger le moindre mot avec Jilano. Preterm births are defined by the World Health Organization as babies born alive before Introduction About infant crying . weeks of pregnancy are completed [ ]. Each year in the world, approximately million babies are born prematurely, that is to say more than one baby out of ten. In France, it is 6 births per day with a total of 6 , births which represent 8% of the births each year. Moreover, due to the increase in the average age of pregnant women, the evolution of lifestyles or the use of medically assisted Introduction . . Introduction reproduction, this number is rising [ ].
	Deux heures de bonheur.
	La referma.

Merci à toi Arthur, d'être venu et d'avoir parcouru toute cette distance, littéralement, pour être revenu jusqu'à moi. Le chemin est encore long, mais tous ensemble rien ne pourra nous arrêter sois en sûr. sûr, je tiens à remercier plus que quiconque my Jane pour tous les messages d'encouragements quotidiens qu'il m'a fallu pour amorcer cette fichue rédaction de manuscrit... Merci d'avoir été XII Détection et caractérisation des vocalisations chez des nouveau-nés prématurés là pour sens vivre. Bien que la distance nous sépare, je savoure chaque instant en votre compagnie ! Merci aussi à vous, Sylvie et Pascal de m'avoir acceptée telle que je suis. Votre maison est chaleureuse et je m'y sens bien, merci de m'offrir un environnement aussi convivial et généreux fort de discussions et de gourmandises ! Merci aussi à mes deux meilleures amies qui sont physiquement les plus loin, mais qui m'ont guidée jusqu'à la personne que je suis aujourd'hui. Merci Philippine pour tout ce qu'on a partagé depuis notre tendre adolescence, si nos chemins s'écartent parfois, je sais qu'ils finissent toujours par se recroiser. Je te souhaite encore beaucoup d'aventures et de rencontres que nous pourrons ensuite partager toute une nuit sous quelques étoiles. Merci aussi à Myway d'être ma soeur de vie depuis dix ans déjà, si notre rencontre était loin d'être gagnée, je sais qu'elle est le fruit du destin et qu'avec toi aussi nos chemins sont loin de s'écarter. Après tout, nos routes se sont déjà croisées à Angers, Rendsburg, Aigle, Paris, Prague, Malmö, Hamburg, Berlin, Toulouse, et si on ne s'est pas croisé à Shanghai, c'est que la suite nous réserve encore bien des surprises ! Malgré la distance, avec toi je partage tout, mes doutes, mes erreurs, mes envies, mes rires, . . . Ensemble rien ne nous arrête, ensemble on apprend à s'envoler ! Merci d'être dans ma vie et merci pour tous les encouragements que tu m'as offerts. Nos discussions m'ont été très précieuses. Je te souhaite d'être libre et heureuse ma très chère amie. Enfin, je remercie tous les colocs avec qui j'ai partagé mon quotidien à Saint-Hélier à savoir, dans l'ordre, Mathilde, Jordan, Léa, Rafael, Justin et Lucille. Les Français Libres c'est un lieu où il fait bon vivre et où je me sens vraiment à la maison. Un grand merci à ma famille rennaise ! Merci Lucille pour la douceur et la sérénité que tu apportes à la maison depuis ton arrivée. On ne s'est pas rencontré au meilleur moment de ma vie, mais sache que ta présence m'a fait beaucoup de bien dans ces moments de tensions intenses. Je te souhaite beaucoup de joie et d'expéditions pour la suite. Où que ce soit, je suis sûre que tu trouveras ton moment pour te lancer dans des aventures solaires et merveilleuses. envies, mes pulsions et de m'accompagner dans mes réflexions et dans mes projets. Ton calme légendaire attendrit mes tempêtes intérieures et faisant état du chemin parcouru, je me dis qu'on a encore beaucoup de choses à s'apprendre et d'excursions à mener. Jusqu'ici, j'admire nos progrès, chacun avec nos forces et nos faiblesses, et je suis fière et heureuse de ce que nous avons construit. Si la distance a fait partie intégrante de notre vie ces dernières années, je suis comblée de te savoir venir "éprouver ton corps et ta tête dans les champs de bananes d'Océanie où l'on se lève à l'aube". J'ai hâte de savourer chaque instant de vie et d'aventure en ta présence.

Merci de tout le soutien que tu m'apportes et de m'offrir à la place de beaucoup de mots, un regard qui en dit long, lorsque tu me souries.

Léa, les mots sont durs à trouver tant il y a de sujets à aborder. Merci pour tout. Merci d'avoir été la plus incroyable colocataire dont je puisse rêver un jour. Tu es le soleil de ma vie dans cette région si réputée pour sa météo. Merci pour tout ce qu'on a partagé. Je repense aux concerts, aux cinés, aux restos, aux promenades, aux repas, aux activités manuelles, aux rangements, aux questions existentielles, aux doutes, aux réconforts, aux larmes, aux joies, aux rires. Tu as partagé mon quotidien depuis trois années maintenant et tu es mon âme, tu es mon coeur et je t'ai dans la peau comme personne. Merci d'être toi, merci pour tout ce que tu m'as appris pour toutes les remises en question et tous les mots rassurants que tu m'as donné. Je me sens comblée de cette rencontre par tout ce que tu m'as apporté et que je n'aurai jamais imaginé. . . Alors c'est ça ce qu'ils appellent grandir ? Merci d'avoir un si grand coeur et une telle sensibilité. Comme tu l'as dit, mon départ de la coloc n'est pas une fin, mais le début d'une autre histoire. J'ai hâte de te retrouver pour nos futures soirées pyjamas où l'on n'aura pas fini de refaire le monde ! Merci encore à tous ceux que j'ai croisés au cours de ma vie et qui m'ont permis d'atteindre mes objectifs. Aujourd'hui, j'ai reçu le titre de docteur et si l'obtention du diplôme marque la fin d'une époque, il marque surtout pour moi le début d'un nouveau voyage au cours duquel mes années d'études me permettront de chercher celle que j'ai envie d'être.

XIV

Détection et caractérisation des vocalisations chez des nouveau-nés prématurés Atteindre le sommet leur prit encore deux heures. Après une ultime traction, elle se retrouva à plat ventre dans la neige. Il n'y avait plus rien au-dessus d'elle que l'infini du ciel. Elle se leva lentement. Le pic qu'ils venaient de gravir se dressait isolé, comme unique prétendant à l'absolu et, debout à son sommet, Ellana eut soudain l'impression qu'elle pouvait tutoyer le soleil. Elle ouvrit la bouche pour une exclamation ravie... . About crying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prematurity is the leading cause of newborn death worldwide and the second leading cause of child death after pneumonia. Most premature infants who survive face a lifetime of disability [ ].

. . About crying . State of the art

	. .	Clinical investigations of crying
	Research on infant crying started with auditory analysis in the 6 s thanks to the Finnish
	Wasz-Hockert research group when it was shown, by spectrographic analysis, that four distinct
	types of cries could be distinguished as birth, pain, hunger, and, pleasure [ ]. Then, crying
	analysis was studied in newborns and small infants with good or poor health conditions, but
	also in premature newborns (see [ ] for a historical review). From there, two other research
	groups largely contributed to this topic: Lester's team in Providence (USA) and Manfredi's team
	in Firenze (Italy). Led by these three groups, several studies have later shown that cry signals
	hold valuable information in the infant health status evaluation according to the clinical context,

which are gathered in Table

  date, the most commonly used database in cry analysis is the Baby Chillanto database with

	data collected by the National Institute of Astrophysics and Optical Electronics, CONACYT Mexico
	[ ]. Initially developed by Reyes-Garcia et al., it divides the cries according to their cause and
	proposes five types, including pain, hunger, normal, asphyxia, and deafness. Cries are equally
	segmented into -s duration with a total number of 68 cries recorded from infants ranging from
	newborns to nine months of age.
	Otherwise, cohort size varied from few infants crying to	cries in [ ] (collected in hospital
	from	babies) or 6 cries in [ ] (recorded by parents using a smartphone). However, very
	few crying databases have been recorded in the NICU [ -8], and there is little information
	about the recordings and the subjects processed,

Table . :

 . Databases recorded in NICU.

  Automatic newborn cry analysis: A non-invasive tool to help autism early diagnosis. In Engineering in Medicine and Biology Society (EMBC), Infant cry language analysis and recognition: an experimental approach. IEEE/CAA Journal of Automatica Sinica, vol. 6, 8-88 ( ). Infant cry detection in adverse acoustic environments by using deep neural networks. In 6th European Signal Processing Conference, EUSIPCO 8. European Signal Processing Conference, EUSIPCO (

	[ 8] O M. Annual International Conference of the IEEE, S., M C., B L., S -[ 6] F D., S M., P E., C A.,	6. IEEE ( S	S.	).
	[ ] G		R.V., J			C.C.,	C	K.D. Neonatal facial and cry responses to invasive and non-
	invasive procedures. Pain, vol. ,	-	(	).
	[ ] [ ] R Analysis with sound spectrogram. Acta Paediatrica, vol. 8 , 68-( P., A E., E G., M K. Newborn infants' cry after heel-prick: . Who: Recommended definitions, terminology and format for statistical tables related to the perinatal period and use of a new certificate for cause of perinatal deaths. Acta ).
	[ ] R	Obstetricia et Gynecologica Scandinavica, vol. 6, -G O.F. R -G C.A. Infant cry classification to identify hypo acoustics and -( ).
	[ ] asphyxia comparing an evolutionary-neural system with a neural network system. In Mexican Inter-. Age terminology during the perinatal period. Pediatrics, vol. 6 -6 ( ). national Conference on Artificial Intelligence, -8. Springer ( ).	,
	[ ] [ ] R pathologies in recently born babies with anfis. In International Conference on Computers for Handi-. Prématurité, ces bébés qui arrivent trop tôt ( -G O.F., T E.A., R -G C.A. Classification of infant crying to identify ). Accessed on / / from: https://www.inserm.fr/dossier/prematurite/ capped Persons, 8-. Springer ( ).
	[ ] W [ ] W	H N., S	P.,	O	H	. Born too soon: the global action report on preterm birth ( M. Automatic infant cry pattern classification for a multiclass ).
	[ ] T problem. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), vol. 8, -J.E., P N.A., L J., G C., H R.D. Intensive care for extreme prematurity -moving beyond gestational age. New England Journal of Medicine, vol. 8, 6 -68 ( 8). ( 6).
	[6] C [ ] O paediatrics: A review. Physiological Measurement, vol. S., P F., S A., R O., P P., S., B L., D G., M C. Central blood oxygen saturation vs crying in C G. Video and audio processing in ( ). preterm newborns. Biomedical Signal Processing and Control, vol. , 88-( ).
	[ ] A H., L [ ] O S., G developmental care for the very low-birth-weight preterm infant. medical and neurofunctional effects. G., D F., M G., G -G R., B A., B A., B V., B S.D., T G., M S., S J. Individualized M.L.,
		JAMA : the journal of the American Medical Association, vol.	, 8 -8 (	).
	[8] L 8 -. Elsevier ( B. L G Developmental Psychobiology: The Journal of the International Society for Developmental Psychobi-L. Crying. In Social and Emotional Development in Infancy and Early Childhood, ). ology, vol. 6, -( ).
	[ ] R language. Early Human Development, vol. , -6 ( H. Analysis of the sounds of the child in the first year of age and a comparison to the [ ] M K., J A., R A. Sound spectrographic analysis of pain cry in preterm ). infants. Early Human Development, vol. 8, -( 8 ).
	[ ] G [ 8] J Zachariah Boukydis, editors, Infant Crying: Theoretical and Research Perspectives, -8 . Springer H.L. C M.J. A Physioacoustic Model of the Infant Cry, In B.M. Lester and C.F. C.C., S B., C K.D., G R.V. Developmental changes in pain expression US, Boston, MA ( 8 ). in premature, full-term, two-and four-month-old infants. Pain, vol. , -8 ( ).
	[ ] L G [ ] W parental perception. Mental Retardation and Developmental Disabilities Research Reviews, vol. , L.L., N A.R., L K., M W., M C., B P. Developmental aspects of infant's cry B.M. Assessment of infant cry: Acoustic cry analysis and melody and formants. Medical Engineering & Physics, vol. , -( ).
	[ ] S	8 -( Y., K	). M., N	F.,	M	-Y	M. Preterm birth is associated with an in-
	[ ] W creased fundamental frequency of spontaneous crying in human infants at term-equivalent age. Bi--H O., P T., V V., M K., V E. The identification of some specific meanings in infant vocalization. Experientia, vol. , -( 6 ). ology Letters, vol. ( ).
	[ ] W [ ] J C., M Infant Crying, 8 --H O., M T.B., G . Springer ( 8 ). K., Y., P EURASIP Journal on Audio, Speech, and Music Processing, art. 8, 68 -L Y. A review of infant cry analysis and classification. J. Twenty-five years of Scandinavian cry research. In ( ).
	[ ] B		I.A., C	H., B	A., B	D.,	B	C. Baby cry recognition in real-world
	[ ] L conditions. In B.M. Spectrum analysis of the cry sounds of well-nourished and malnourished infants. Child 6 th International Conference on Telecommunications and Signal Processing
	Development, (TSP), -8 (	-6).	(	6).
	[ ] L [ ] C	C.Y.	B.M. T	D L.Y. A CNN-based method for infant cry detection and recognition. In L. Barolli, M. Effects of marijuana use during pregnancy on newborn cry. Child Devel-
	opment, 6 -M. Takizawa, F. Xhafa, and T. Enokido, editors, Web, Artificial Intelligence and Network Applications, ( 8 ).
	86-	. Springer International Publishing (	).
	[ 6] Z		P.S.	L	B.M. Acoustic features and auditory perceptions of the cries of newborns
	[ ] S	with prenatal and perinatal complications. Child Development, 8 -8 ( M., F D., P E., S S. Automatic detection of cry sounds in neona-8).
	tal intensive care units by using deep learning and acoustic scene simulation. IEEE Access, vol. ,
	[ ] G 8 -	H.L. (		).
	6 Chapter • About infant crying	Bibliography

C M.J. Infant cry: A clue to diagnosis.

Pediatrics, vol. 6 ,

. M C. AVIM -A contactless system for infant data acquisition and analysis: Software architecture and first results. Biomedical Signal Processing and Control, vol. , 8 -(

).

[ 6] Z P.S., P -P S., B R.G. Rhythmic organization of the sound of infant crying.

8 Chapter • About infant crying [ ] L L., L W., W X., Z B.X.

  ). International Journal of Scientific & Engineering Research, vol. , Baby cry sound detection: A comparison of hand crafted features and deep learning approach. In G. Boracchi, L. Iliadis, C. Jayne, and A. Likas, editors, Engineering Applications of Neural Networks, 68-.

										-8
	(	).
	[ ] O		S., D		P.H., S		J., L	J., R	N.,	M	C. Effective pre-
	processing of long term noisy audio recordings: An aid to clinical monitoring. Biomedical Signal
	Processing and Control, vol. 8,	-8 (	).
	[ ] O		S., B		A., F	F.,	M	C. Testing software tools for newborn cry analysis
	using synthetic signals. Biomedical Signal Processing and Control, vol. , 6-(	).
	[ ] M		C., B	A., M	D., V		R., K	M.,	O	S. Automated detection
	and classification of basic shapes of newborn cry melody. Biomedical Signal Processing and Control,
	vol. ,	-8 (	8).
	[ ] Z	A., M		W., K	L.Y., S	R.,	R	F. Mel-frequency cepstrum coefficient
	analysis of infant cry with hypothyroidism. In		th International Colloquium on Signal Processing
	& Its Applications,	-8. IEEE (	).
	[ ] C	R.	L	Y. Infant cry analysis and detection. In	IEEE th Convention of Electrical
	and Electronics Engineers in Israel, -. IEEE (	).
	[ ] V		J G. Z.B.	I	. A. Automatic infant cry detection. page -(	).
	[ ] A	-A	L., T	C.,	F		H.A. A fully automated approach for baby cry signal segmen-
	tation and boundary detection of expiratory and inspiratory episodes. The Journal of the Acoustical
	Society of America, vol.	,	8-	(	).
	[ 6] X J., L	X., O	R.,	S	C. Convolutional neural networks for audio-based continuous
	infant cry monitoring at home. IEEE Sensors Journal, vol. ,	-	(	).
	[ ] L	Y., C	R., R	D.,	IJ	H. Baby cry detection in domestic environment using
	deep learning. In	6 ICSEE International Conference on the Science of Electrical Engineering, -.
	IEEE (	6).
	[ 8] T	R., B		D.,	L		L. Springer International Publishing (	).
	[ ] C	R., R		D., Z		J., IJ	H., L	Y	" .W.,	C	S.M. Baby Cry
	Detection: Deep Learning and Classical Approaches, In Development and Analysis of Deep Learning
	Architectures,	-6. Springer International Publishing, Cham (	).
	[ ] S		M., P	E., C	S., G	L.,	S	S. Who cried when: Infant cry di-
	arization with dilated fully-convolutional neural networks. In	International Joint Conference on
	Neural Networks (IJCNN), -8 (	).

Table . .

 .

		qty dur. (s)	qty dur. (s)	qty dur. (s)
	Cry	6.	6	.	-	-
	Baby Other	6 .		. 8	6	.8
	Alarm	.		. 8		. 6
	Voices	.		.	-	-
	Other	.		. 6		. 6
	T	.8	6	.		. 6

  based approach to automatic detection of equipment alarm sounds in a neonatal intensive care unit environment.IEEE journal of Translational Engineering in Health and Medicine, vol. 6,

	[ 6] R			G., N	C., J		P., L	A.P., K	M., M	B.M.,	D V	A.R.
	A knowledge--( 8).
	[ ] R			G. Automatic analysis of the acoustic environment of a preterm infant in a neonatal
	intensive care unit. Ph.D. thesis, UPC, Departament de Teoria del Senyal i Comunicacions (	6).
	[ ] C M [ 8] C care units. Frontiers in Public Health, vol. , page S., C R., P M., L M., P M., M M., C E., P Z., D A., P T., D I., A J. A selected review of the mortality rates of neonatal intensive J., S M R.N., L H., D., K P. Observational study found that ( even small variations in light can wake up very preterm infants in a neonatal intensive care unit. Acta ). Paediatrica, vol. ( 8).
	[ ] H [ ] F of physiological signals in the neonatal intensive care unit. Physiological Measurement, vol. 8, page J., T C., T M., H M., N C., M E A. Advanced analyses J., B M., L H., K P. Cortical responses to alien odors in newborns: R ( ). An fnirs study. Cerebral cortex (New York, N.Y. : ), vol. 8, -( ).
	[ ] A H., L [ ] K born very preterm react to variations of the acoustic environment in their incubator from a minimum G., D F., M G., G -G R., B P., C Z., P T., H A., L C., E B., A D., D A. Infants J. Individualized developmental care for the very low-birth-weight preterm infant. medical and neurofunctional effects. JAMA : the journal of the American Medical Association, vol. , 8 -8 ( ). signal-to-noise ratio threshold of to dba. Pediatric research, vol. , 86-( ).
	[ ] B [ ] H Press ( Review, vol. 6, page 6 -T.B. N J.K. Neonatal behavioral assessment scale. P.G. S B. The development of fetal hearing. Fetal and Maternal Medicine . Cambridge University ). ( ).
	[ ] P [ ] G ( ). as a function of age. Psychonomic Science, vol. , 6 -6 ( F., B K., B W., J J., H H., K H. The behavioural states of the newborn infant (a review). Brain Research, vol. 6, 8 -S. Cardiac orienting response ).
	[6] P	H.F. Qualitative changes of spontaneous movements in fetus and preterm infant are a
	marker of neurological dysfunction. Early Human Development, vol. ,	-8 (	).
	[ ] P	H.F., E		C., C	G., B A.F., F		F.,	S	D. An early marker for
	neurological deficits after perinatal brain lesions. Lancet, vol.	, 6 -(	).
	[8] B A.F., M	A.,	A		R.M., H	-A	M., O	A.,	P	H.F. Qualitative
	assessment of general movements in high-risk preterm infants with chronic lung disease requiring
	dexamethasone therapy. The Journal of Pediatrics, vol.		,	-6 (	8).
	[ ] C	S., P	F., S	A., R	O., P	P.,	C	G. Video and audio processing in
	paediatrics: A review. Physiological Measurement, vol.	(	).
	[ ] C	S., P	F., C	G., R	O., G	F., P		P., S	A.,	C	G. Voxyvi:
	A system for long-term audio and video acquisitions in neonatal intensive care units. Early Human
	Development, vol.	, page		(	).
	[ ] L	M., P	B., R	A., R S., S	V., N	M., R	A.,
	D	M. Spectral analysis of noise in the neonatal intensive care unit. Indian journal of pediatrics,
	vol. ,	-(		8).
	[ ] R		G., N	C., G	O., S	S., M		B.M.,	V	A.R.,	H	S.N.
	On the acoustic environment of a neonatal intensive care unit: initial description, and detection of
	equipment alarms. In Proc. Interspeech	,	-	(	).
	[ ] S	A.	M	H. Sound environments surrounding preterm infants within an occupied
	closed incubator. Journal of Pediatric Nursing, vol. , e	-e	(	6).
	[ ] R		G., J	P., N	C., L	A.P., K		M., M	B.M.,	V	A.R.
	Automatic detection of equipment alarms in a neonatal intensive care unit environment: A knowledge-
	based approach. In INTERSPEECH	: 6th Annual Conference of the International Speech Commu-
	nication Association,	-	6 (	).
	[ ] P	L	. A., R			. G.,	N	. C. A neural network approach for automatic detection of
	acoustic alarms. In Proceedings of the th International Joint Conference on Biomedical Engineering
	Systems and Technologies -BIOSIGNALS, (BIOSTEC	), 8 -. INSTICC, SciTePress (	).
	8 Chapter • Work context						. . Conclusion

[ ] G S. Sound and the developing infant in the nicu: Conclusions and recommendations for care. Journal of perinatology : official journal of the California Perinatal Association, vol. , S88-(

).

[ ] H S.M.A., R N.M.E., S A.A. Neonatal nursery noise: practice-based learning and improvement. The Journal of

 vol. 6,

Table . :

 . Fixed hyperparameters used to train the last layer of the CNN.

	Cost function	cross entropy
	Optimization algorithm	stochastic gradient descent
	Learning rate scheduler	standard decay
	Momentum of the optimizer	.
	Regularization by weight decay	. -
	Regularization by batch learning	6
	Number of learning iterations (epoch)
	Class imbalance management (class weighting according to data distribution, see Section . . )	non-cry .66, cry .

Table . :

 . Details of predefined labels in SoundAnnoT software.

Table . .

 .

					.		.
				.		.		.
	cry	cry cry+	6	.66% . 8%	8 8 6 8	. % . 8%		8. % . %
		baby others		.6 %		. %	8	. %
	non-cry	voices alarms others	6	. 6% . 8% 8. 6%	6 8	.6 % .88% . 8%	6	.6 % .86% . %
		mixtures		. %	88	.86%	8	8. 8%
				. %		. %	8	. %

Table . .

 .

							.		.
				.			.		.
							( = )
	TRAIN	cry non-cry	8	8. 6% . % . %	6	8	.68% 6 . % . %	8 6 6	.6 % 6 . % . %
							( = )
		cry	6	. 8%		6	. %	8 8	. 8%
	TEST	cry+ non-cry		.8 % 6 . 8%		8	. % . 8%		.6 . %
			6	. %	8		. %		. %
				. %	8 6 8	. %		. %
									. . Results

Table . 6

 . : Subsets of the data used in this study.

							-
	. s	ResNet 8	-----	.8 6 .868 . . .8	.8 .8 .8 6 .8 . 6	. . . .86 6 .68 8	. 6 .8 .86 .8 .
		ResNet	--	.8 6 . .8 8	.8 .8 .	. .886 .86 6	. 6 .8 8 .8
	. s	ResNet 8	-----	.8 .8 . . .86	.8 .8 .8 8 .8 .8	. .8 .86 6 . .	. 6 .8 .8 .8 .8
		ResNet	---	. .8 8 .	.8 .8 . 88	. .8 8 . 8	.86 6 .8 .8

-

Table . :

 . Performance of the 6 candidate combinations based on sound predictions obtained on the validation set.One can see that the four best candidate combinations with the associated learning rates giving the highest precision score have also high recall values ranging from % up to %. The best combinations retained for the following training step, compared in Figure., are:

Chapter • Classification for cry detection

Table . 8

 . : Performance of the final model on the test set including three babies.

		cry	non-cry
		cry		6	6
		cry+	8
	baby others	8
		voices		6	6
		alarms			8
		others		8	8 6
		mixtures		6
	(a) Detailed confusion matrix with the seven labels.
		cry		non-cry
	.	cry		6	6
		non-cry
					6
		(b) Confusion matrix without cry+.
		cry		non-cry
	.	cry
		non-cry
		8			6
		(c) Confusion matrix with cry+.

Table . :

 . Confusion matrix results for the test-set.

. . Results

Table . :

 . Classification for cry detection goal to design a robust classifier for deployment in clinics with a good precision score. Indeed, by ensuring that the sounds predicted as cries are actual cries, we ensure the reliability of the further cry analyses. Detailed database used during the best candidate combinations selection using a simple-validation with a train set including babies and a validation set of baby.

				-
							.	.
					.			.	.
								( = )
			cry	8	8. 6%	68	. %	8	.6 %
			non-cry		. %	8	6 . %	8	6 . %
				6 8	. %	6	6	. %	. %
								( = )
			cry		. %	6		6.6 %	6.6 %
			non-cry	6	8.8 %		8	. 6%	. %
				8	. %	6		. %	. %
					. %	6		. %	6	. %
						fi		1 -
	. s	Resnet 8	-----	.8 .8 .86 .8 .8		.8 8 .8 8 .8 .8 6 .8		.6 . . 8 . .6	. . .8 8 6 . .
		Resnet	---	.8 .86 .8 6		.8 .86 .8 6		.66 . 6 .	8	. .8 6 . 8
	. s	Resnet 8	-----	.8 .8 8 .8 .8 8 .8 6		.8 .8 6 .86 .8 .8		.6 8 . 6 . . 6 .	. 6 .8 6 .8 . .
		Resnet	---	.88 .86 8 .86 8		.8 .8 .8 6		.8 . 66 . 8	.8 8 .8 .8
								.6. Conclusion

Chapter •

Table . :

 . Performance of the classifier for the frame database during the best candidate combinations selection using a simple-validation on one baby.

				-fi
								-
	mean	+6		mean	6+6	sounds	88
	s.d	+		s.d	+	frames ( . s)		8
	range	+ -+	range	+6-+	frames ( . s)	8	8	6
	mean	+		mean	+6	sounds	8	8	8 8
	s.d	+		s.d	+	frames ( . s)	68	8 8	6
	range	+ -+	range	8+ -+	frames ( . s)		6	8 6
	mean	+		mean	+	sounds
	s.d	+		s.d	+	frames ( . s)		8
	range	+ -+	range	8+ -+	frames ( . s)	6	8 6	6
	mean	+		mean	+	sounds			6
	s.d	+		s.d	+	frames ( . s)	8
	range	+6-	+	range	+ -+6	frames ( . s)	6		6
	mean	+		mean	+	sounds
	s.d	+		s.d	+	frames ( . s)		6	8
	range	+ -+	range	+ -+	frames ( . s)	6	8	86

Table . :

 . Annotated cries database for the fundamental frequency estimation evaluation.

	. .	Annotated database
	The SoundAnnoT software (see Section . ) was used to identified cries to evaluate the funda-
	mental frequency tracking method. A total of 8 6 cries recorded in a preterm female infant (GA:
	weeks + 6 days) were collected from three different recordings in open and closed bed with
	characteristics described in Table . .
		(w+d)
		+	closed	8
		+	closed
		8+	open	6
				8 6
	Chapter • Fundamental frequency characterization

Table .

 . ) which did not reveal a linear cor-

	relation between the different parameters of the two methods. Therefore we investigated the
	same comparison by taking into account only the cries that were correctly estimated by both
	methods (	). The new results computed on 6 cries are illustrated in Figure . b and the
	new Pearson's coefficients are also given in Table . .
		8 6	.	. 6	.	.
		6	.	.8	. 8	.8

Table . :

 . Person's coefficient for both cases.

  or because preterm infants display a more stressful response to pain stimuli. In addition, through the evaluation of pain from facial expressions and crying performed in premature infants, but also in full-term and -and -month-old infants, Johnston et al. showed that: i) premature infants were different from older infants, ii) full-term newborns were different from others, but iii)

	-and -month-old were similar [ ].
	6. .	Spontaneous cries
	The analysis of spontaneous crying is much more recent and only a few studies have investigated
	the subject [8-]. In	, Wermke et al. compared the spontaneous crying of six preterm
	infants (three pairs of twins) recorded at different PMA (8-weeks, -weeks, and -
	weeks) [8]. Essential changes in the cries were observed from the 8th-th week of life up to
	the rd-th week of life, where they showed that the melody increased in complexity, from
	simple rising-falling patterns to composed patterns. This development was interpreted as an
	intentional articulatory activity related to neurophysiological maturation. In	, Orlandi et al.
	Regarding the comparison between preterm and full-term infant crying, Tenold et al. could not
	show significant differences in fundamental frequency between the two groups. Yet, they did
	show greater variability in the preterm infant cry spectra which was interpreted as likely reflecting Finally, in , André et al. proposed the first vocal repertoire of preterm newborns in non-painful differences in neurophysiological maturity [ ]. Later, Michelsson et al. showed that the cries of the resting contexts [ ]. They observed a broad range of vocalizations that they separated into nine smallest premature newborns were shorter, with higher fundamental frequency, and included bi-phonation and glide more often compared to control newborns [ ]. Such results were also reported vocal types distinguishable acoustically and non-randomly associated with behaviors.
	in [ ], which shows that pain cries in preterm infants, observed between and	weeks PMA,
	are higher-pitched than those of full-term infants. In addition, with increasing post-menstrual age
	comes an increase in the pain-cry duration and a decrease in the pain-cry fundamental frequency,
	which can represent a maturation of the central nervous system. Both last studies indicated
	that the cry characteristics changed with increasing post-menstrual age and the older the infant
	the more the crying pattern resembled that of the full-term [ , ]. For their part, Goberman et al.
	identified clear differences in first spectral peak, mean spectral energy, and spectral tilt between

or dedicated methods

[ -]

. The pain cries were induced by a pinch at the infant arm [ ], rubber band hit [ ], pinch in the infant ear

[ , ]

,

or during health check-ups such as heel-stick procedure

[ , 6]

and auditory brainstem response hearing screening test [ ]. When comparing sick and healthy infants, Michelsson et al. showed in that the sick infant's cries were higher-pitched than those of symptomless premature babies, which were themselves higher pitched than those of healthy full-term newborns [ ]. Later, Stevens et al. included two variables, severity of illness and behavioral state (sleep or awake) in the analysis [6]. The behavioral state was found to influence the facial action variables, and the severity of illness modified the acoustic cry variables. full-term and preterm infants [ ]. According to them, the observation of higher F in the preterm infant's cries may either be related to smaller vocal folds, resulting from physical size differences Chapter 6 • Automatic processing for cry analysis: deployment at birth investigated if the distress occurring during crying in preterm newborns was related to central blood oxygenation. The results indicate that a similar decrease in oxygenation level occurs in both groups of patients, but that the recovery time after the crying episode is more stable and rapid in term infants than in preterm newborns. For their part, Shinya et al. inspected the effects of gestational age, body size at recording, and intrauterine growth retardation [ ]. The acoustic analysis of spontaneous cries before feeding in both healthy preterm infants at term-equivalent ages and full-term newborns showed that shorter gestational age was significantly associated with a higher fundamental frequency, although no relation was found with smaller body size at recording or IUGR. Regarding the fundamental frequency and formants of preterm newborns, Manfredi et al. showed a decrease in frequency with increasing age that can be explained by increasing length and structural changes of both vocal folds and vocal tract [ ] and Orlandi et al. showed that preterm newborn cries were generally higher than those of full-term infants [ ].

Table 6 .

 6 .

	≤	w.	< GA <	w.	≤ . kg > . kg
						6
				6		6
				6	6	6 6	6
	(a) Database considering gestational age.	(b) Database considering weight at birth.

Table 6 .

 6 : Databases for the comparison of fundamental frequencies with gestational age and weight at birth cut-offs.Fundamental frequency results of our database are presented in Figure6. a considering the gestational age and in Figure6. b considering the weight at birth. We decided to show all the statistical parameters since we don't know which one is presented in Figure6. . In our case,

	statistical parameters are averaged for each infant, and cry values are combined per period
	through median values. These results show a decrease in frequency with increasing age or weight
	for all statistical parameters, which is consistent with what Manfredi et al. reported in their study
	[ ]. However, the results when comparing infants with a birth weight of less or greater than	g
	are less significant.

Moreover, it has to be mentioned that the F value distribution is located in much lower fundamental frequency values in our case, which is surprising since our frequency range of analysis is 8 Chapter 6 • Automatic processing for cry analysis: deployment

Table 6 .

 6 : Crying duration evolution database according to GA divided in two preterm and one full-term infant groups.Unfortunately, by removing the babies in double rooms, we have reduced the number of moderateto-late preterm infants to be analyzed. In addition, in the chosen baby subset, no preterm infant was born before weeks. Therefore, the data distribution is quite sparse and does not cover the gestational age range with the same efficiency as in the original study [ ]. However, where Shinya

	< w.	w. ≤	<	w.	≥ w.
						6
	6	6			8 8	6

6. . Reproduction of existing studies

Table 6 .

 6 : Comparison of the fundamental frequency of spontaneous crying according to GA divided into two preterm and one full-term infant groups for both methods.

Table 6 .

 6 .

	-

Chapter 6 • Automatic processing for cry analysis: deployment

Table 6 .

 6 : Database used to assess F at two postnatal ages. Results of the fundamental frequency are computed with statistical parameters (i.e., minimum, maximum, mean, and median) averaged for each infant and cries values combined per period through median values. The results are presented in Figures 6.6 as two pairs of violin plots, with preterm newborns represented by the left-hand sides and full-term infants by the right-hand sides.

-We deploy the complete processing chain combining the three methods described in Chapter , and and we present the clinical results computed for newborns. First, we

Chapter • Audio-Video segmentation

Chapter • Audio-Video segmentation

Chapter • Classification for cry detection

Chapter • Classification for cry detection

Chapter 6 • Automatic processing for cry analysis: deployment

Chapter 6 • Automatic processing for cry analysis: deployment

Chapter 6 • Automatic processing for cry analysis: deployment

Remerciements

Remerciements

XI

Remerciements XIII

Fundamental frequency characterization

Work context

Chapter .

• True Positive Annotations (TP M ): the number of annotated segments overlapping one or more segment detected by the segmentation,

• False Negative (FN): the number of segments annotated but not detected.

The same process is symmetrically repeated for the segments resulting from the automatic segmentation. In that case, a segment is considered detected when at least % of the samples within the annotated detection signal are equal to , otherwise not.

• True Positive Segmentation (TP A ): the number of detected segments overlapping one or more annotated segment,

• False Positive (FP): the number of segments detected but not annotated.

Based on the previously defined segment parameters, we can describe the segmentation method performance through the sensibility and precision, defined as:

• Sensibility (S): percentage of annotated segments that have been detected through segmentation. It answers the question: "How much of the annotated segments were detected?":

• Precision (P): percentage of segmented sounds that are actual annotated cries. It answers the question: "How much of the extracted segments are really cries?":

. .

Duration comparison

Manual annotation and automatic segmentation are compared in terms of duration through three parameters, which are illustrated in Figure . and defined as:

• ∆ M -the manual annotation total time;

• ∆ A -the total time of segments resulting from the automatic segmentation.

• ∆ M∩A -the total manual and automatic segmentations overlapping time. .

. Evaluation strategy

The total annotation or segmentation time is obtained by counting the number of samples equal to in each detection signal while the overlapping time is computed by counting the number of samples equal to in the signal resulting from the two detection signals summation. Then the length of the detected samples is converted to duration using the sampling frequency.

Once again, the segmentation method performance are assess in terms of sensibility and precision, defined for durations as:

• Sensibility (∆ S ): percentage of cries detected through the automatic segmentation:

• Precision (∆ P ): percentage of segmented sounds that are actual annotated cries:

.

Results

In this section, we deploy our segmentation method on three -minute audio files. After a description of the database, we evaluate the various improvements brought to the method initially proposed by Orlandi et al. [ ] thanks to the parameters defined previously.

.

Database

To observe and evaluate the performance of this segmentation method, three -minute files were manually annotated using the Audacity software (see Section . . for exhaustive acoustic environment annotations). The start-and end-points of each audible cry event in the soundtrack were identified and the annotation boundaries were set at the point where the cries could no longer be heard. It is worthwhile to remind that only cry annotations whose duration is between .

and seconds are considered to be consistent with the segments derived from the automatic methods.

The three -minute sound files were selected from a -hour recording made for one baby .

These files were selected for their sound event variety and show once again the acoustic environment variability in a single recording:

• WAV -h -sounds ( cries and non-cries);

• WAV -h -sounds ( cries and non-cries);

• WAV -h -6 sounds ( 6 cries and 8 non-cries).

. i.e., baby recorded during night time the 8--.

Chapter • Audio-Video segmentation

In the following, we propose to review the results derived from the different improvements provided to the reproduction of the method proposed by Orlandi et al. as well as the results derived from the proposed audio-video segmentation.

The results of the three annotated -minute WAV files processed by all steps and methods are presented in the Table . in terms of segment and the Table . in terms of duration.

The results are presented according to the methods used, whose acronyms are given below:

• REP: reproduction of the method proposed by Orlandi et al.;

• NFB: results obtained after applying the Narrowing Frequency Band solution;

• NFB+RS: results obtained after applying both, the Narrowing Frequency Band and Re-Segmentation solutions.

• LTT: results obtained after identifying audio files with enough sound content. True if more than sounds are detected above the threshold T calculated over two-hour sliding windows, false otherwise.

• AV: results obtained with the proposed enhancements (i.e., NFB+RS+LTT) and after collecting the sounds included in infants' motion intervals only.

.6.

Audio segmentation improvements evaluation

Reproduction of Orlandi's method

With these results, we can justify the remarks made earlier in Section . . when discussing the issues encountered when applying the reproduction of Orlandi et al. method on our data.

First, we saw that Otsu's method does not work on recordings containing few sound events and no cry. This issue is illustrated by processing the file whose audio content is poor and where no less than segments are extracted.

Then, what can also be observed is the difference between the quantity of manually annotated cries and the number of segments resulting from the automatic segmentation. In the case of the file we can notice that cries are automatically detected (TP M ) by the segmentation (out of the 6 that were manually annotated), however only segments are extracted (TP A ). This means that the method gathers sounds in the resulting segments (i.e., automatically extracted segment = n annotated segments).

. De nombreuses études ont montré que l'analyse des pleurs de nourrissons permettait d'obtenir des informations sur leur état de santé et dans le cas des prématurés sur leur maturation. Si les premiers travaux se basaient sur une segmentation manuelle de pleurs souvent induits (généralement par la douleur), les travaux actuels s'intéressent aux pleurs spontanés, ce qui nécessite le développement de méthodes d'extraction automatiques. Cette approche non-invasive de monitoring apparaît comme extrêmement pertinente au vu de la fragilité des sujets étudiés. Cependant, l'environnement hospitalier particulièrement bruité où se déroulent les enregistrements complexifie grandement l'automatisation des méthodes.

Dans ce contexte, et dans le cadre du projet européen Digi-NewB, l'objectif de ces travaux est de présenter une chaîne complète de traitements automatiques pour l'analyse des pleurs des prématurés enregistrés en USIN. Cette chaîne regroupe : i) une nouvelle approche de détection des pleurs composée d'une segmentation, réalisée à partir de la fusion de vidéos et de bandes son ; ii) une classification par deep-learning pour l'identification des pleurs parmi tous les sons segmentés (voix d'adultes, alarmes. . . ) ; iii) l'estimation de la fréquence fondamentale des pleurs détectés par une nouvelle approche basée sur la détection de contours dans le spectrogramme. Le déploiement de la chaîne de traitements sur une base de données de pleurs enregistrés en USIN montre des résultats en accord avec ceux publiés dans la littérature. Cette validation est encourageante et annonce la possibilité d'observer automatiquement sur des grandes cohortes l'évolution des pleurs des prématurés, notamment en vue de caractériser leur développement.

Title: Detection and characterization of vocalizations in preterm newborns.

Keywords: preterm newborn, neuro-behavioral development, monitoring, neonatal intensive care units, audio analysis, spontaneous cries Abstract: The number of premature births is estimated at million per year worldwide and represents % of births in France. These babies are cared for in Neonatal Intensive Care Units (NICU) and are subject to special surveillance because of the immaturity of their organs and the complications that may arise. Numerous studies have shown that the analysis of infant crying provides information on their health status and, in the case of premature infants, on their maturation. While early work was based on manual segmentation of often induced crying (usually by pain), current work focuses on spontaneous crying, which requires the development of automatic extraction methods. This non-invasive monitoring approach appears to be extremely relevant given the fragility of the subjects studied. However, the particularly noisy hospital environment where the recordings are made makes the automation of the methods very complex. In this context, and within the framework of the European project Digi-NewB, the objective of this work is to present a complete chain of automatic treatments for the analysis of the cries of premature babies recorded in NICU. This chain gathers: i) a new approach of crying detection composed of a segmentation, realized from the fusion of videos and soundtracks; ii) a classification by deep-learning for the identification of crying among all the segmented sounds (adult voices, alarms...); iii) the estimation of the fundamental frequency of the detected crying by a new approach based on the detection of contours in the spectrogram. The deployment of the processing chain on a database of cries recorded in NICU shows results in agreement with those published in the literature. This validation is encouraging and announces the possibility of automatically observing the evolution of crying in premature babies on large cohorts, in particular in order to characterize their development.