Introduction 5

IFI ertiultory speeh synthesis F F F F F F F F F F F F F F F F F F F F F F F F F F F F F S IFP wotivtion F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F S IFQ rolem sttement F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F U IFR gontriution F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F U Chapter 2
Background and context 9

PFI peeh prodution F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F W PFIFI eessing lnguge resoures F F F F F F F F F F F F F F F F F F F F F F F F F IH PFIFP wovement plnning nd exeution F F F F F F F F F F F F F F F F F F F F F F II PFIFQ edjusting the plnX selfEorretion F F F F F F F F F F F F F F F F F F F F F F PW PFP peeh modeling nd synthesis F F F F F F F F F F F F F F F F F F F F F F F F F F F F QH PFPFI peeh prodution models F F F F F F F F F F F F F F F F F F F F F F F F F F F QH PFPFP peeh synthesis F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F QH PFPFQ wultimodl speeh synthesis F F F F F F F F F F F F F F F F F F F F F F F F F QQ Chapter 3
Articulatory speech synthesis from static MRI data 39

QFI sntrodution F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F QW QFP yjetives F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RH QFQ fuilding n rtiultory speeh synthesis system F F F F F F F F F F F F F F F F F F RH QFQFI htset F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RH
QFQFP trtegies for trnsitioning etween the rtiultory trgets

F F F F F F F F RS i QFQFQ ytining the sound F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RT QFR ivlution F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RU QFRFI he rtiultory model nd the trjetories F F F F F F F F F F F F F F F F F RU QFRFP qlottl opening ontrol F F F F F F F F F F F F F F F F F F F F F F F F F F F F RU QFRFQ he synthesized sound F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RU QFS gonlusion F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RV QFSFI yverview of results F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RV QFSFP puture work F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RW Chapter 4
Articulatory speech synthesis from real-time MRI data 51

RFI yjetives F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F SI RFP wethods F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F SI RFPFI ht preprtion F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F SP RFPFP smplementtion F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IHI RFQ ivlution F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IHR RFQFI ivlution omponents nd riteri F F F F F F F F F F F F F F F F F F F F F IHR RFQFP ivlution dt nd methods F F F F F F F F F F F F F F F F F F F F F F F F F IHR RFQFQ ivlution results F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IHS RFR gonlusion F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IQS RFRFI yverview of the results F F F F F F F F F F F F F F F F F F F F F F F F F F F F IQS RFRFP puture work F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IQT Chapter 5
Static targets versus running speech for articulatory speech synthesis 139

SFI yjetives F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IRH SFP ht nd methods F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IRP SFPFI reting ws nd Ews ptures F F F F F F F F F F F F F F F F F F F F F IRP SFPFP smge omprison mesures F F F F F F F F F F F F F F F F F F F F F F F F F F IRQ SFQ ixperiments F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IRS SFQFI emporl ehvior F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IRS SFQFP histriutions nd orreltions F F F F F F F F F F F F F F F F F F F F F F F F F ISI SFQFQ enlyzing the spekers F F F F F F F F F F F F F F F F F F F F F F F F F F F F F ISP SFQFR honeme omprisons F F F F F F F F F F F F F F F F F F F F F F F F F F F F F ISV SFR ivlution F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F ITR ii SFRFI ertiultory similrity mesure F F F F F F F F F F F F F F F F F F F F F F F F ITR SFRFP ertiultory error F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F ITW SFS owrds ringing the two diretions together F F F F F F F F F F F F F F F F F F F F IUP SFT gonlusion F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IUP SFTFI yverview of results F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IUP SFTFP puture work F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IUR Chapter 6
Conclusions 175

TFI qlol overview F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IUS TFP puture work F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IUT

Appendixs

Appendix A Prompts for spontaneous speech in the RT-MRI corpus Appendix B Detailed summary in French fFI ynthèse rtiultoire de l prole à prtir des données sw sttiques F F F F F F IVI fFP ynthèse rtiultoire de l prole à prtir des données sw en temps réel F F F F IVQ fFQ giles sttiques et l prole en temps réel pour l synthèse de l prole rtiultoireIVR

fFR gonlusion glole F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IVR Bibliography iii List of Figures PFI rumn peeh wehnism F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IP PFP honeme oundries F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IQ PFQ he rinipl wusles of espirtion F F F F F F F F F F F F F F F F F F F F F F F F F IR PFR rumn ol yrgns F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IU PFS pper eirwy F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IW PFT hivision of the hryngel gvity F F F F F F F F F F F F F F F F F F F F F F F F F F PH PFU les of ertiultion F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F PI PFV ghrt of owels y their petures F F F F F F F F F F F F F F F F F F F F F F F F F F F PP
PFW he even rmeters of wed9s ertiultory wodel F F F F F F F F F F F F F F F QU QFI widEsgittl slie of the stti Qh imges of sujet S A for severl of the prenh vowelsF F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RI QFP widEsgittl slie of the stti Qh imges of sujet S A for some of the prenh onsonntsF gonsonnt ws pronouned in ontext of the following vowelF F F F F RI QFQ en exmple of dtset imge nnottion @GaGAF F F F F F F F F F F F F F F F F F F F RI QFR he geEsed rtiultory modelX urve hnge diretions enoded in the (rst three ftors of eh rtiultor @the jwD the tongueD the lipsD the epiglottisD the lrynxAF F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F RP QFS ipiglottis nd velum enterlines reonstruted y the model F F F F F F F F F F F F RR QFT e humn9s utterne of GaSaG nd its synthesis long with the glottl losure ontrol s opied from the iqq dt

F F F F F F F F F F F F F F F F F F F F F F F F RV RFI EwsX efore proessing F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F SW RFI EwsX ilterl (lter F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F SW RFI EwsX dptive threshold F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F SW
RFP ol trtD velum nd lips windows for the Ews frmes F F F F F F F F F F F F TI RFQ flurred tongue shpes in Ews F

F F F F F F F F F F F F F F F F F F F F F F F F F TP RFR flurred velum shpes in Ews F F F F F F F F F F F F F F F F F F F F F F F F F F F TP RFS flurred epiglottis shpes in Ews F F F F F F F F F F F F F F F F F F F F F F F F F TQ RFT flurred lrynx shpes in Ews F F F F F F F F F F F F F F F F F F F F F F F F F F TQ RFU egmenttion templte for x + IR F F F F F F F F F F F F F F F F F F F F F F F F F TR
RFV e proessed window for the lips nd their seed points F F F F F F F F F F F F F F F TT RFW hi'erent ses for proessing the lips F F F F F F F F F F F F F F F F F F F F F F F F F TT RFIH gomputing the prmeters of lips

F F F F F F F F F F F F F F F F F F F F F F F F F F F TV RFII roessed velum window F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F TW v
RFIP wovement in the window of the velum F F F F F F F F F F F F F F F F F F F F F F F F TW RFIQ gontour reognition in the window of the velum F F F F F F F F F F F F F F F F F F UH RFIR hryngel wll seed prolem F F F F F F F F F F F F F F F F F F F F F F F F F F F F F UP RFIS ell rtiultors in ontt with eh other"no need to lulte the distnesF flue mrks the ontour tht ws leled s the velumY purple the ontour tht ws leled s the velum nd the phryngel wllY ottle greenD the tongue nd the phryngel wllF F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F UQ RFIT essigning ontours in the speil se of two rtiultory pirs in ontt F F F F UR RFIU ertiultory prmeters extrted from the window of the velum F F F F F F F F F UT RFIV ertiultory prmeters extrted from the window of the velumD the se of ontt F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F UU RFIW he tongue nd the velum reognized s in ontt F F F F F F F F F F F F F F F F F UV RFPH xo di'erentiting etween )eeting nd (rm ontt in the window of the velum UW RFPI ertiultory prmeter onsisteny in the originl orpusD overll F F F F F F F F F VQ RFPP ertiultory prmeter onsisteny in the originl orpusD roken down y spekers VU RFPQ ertiultory prmeter onsisteny in the originl orpusD roken down y sponE tneity F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F WI RFPR ertiultory prmeter onsisteny in the originl orpusD roken down y spekE ers nd spontneity

F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F WS RFPS vip ontour utomti nnottion F F F F F F F F F F F F F F F F F F F F F F F F F F IHH
RFPT plsely reognized spe etween the lips F F F F F F F F F F F F F F F F F F F F F F IHI RFPU he full art synthesis of onjour4 GbÕZuKG with the voie of S A F he trnsition of formnts orresponds to the hnge of phonemes in produtionF F F F F F F F F IHT RFPV he synthesized sequenes of lsdist nd lscont for onjour4 GbÕZuKG with the voie of S A F he lip losure @in redA is onsistent with the prodution of the lil stop GbG nd the nrrowed lil opening for GuG nd with the sene of lil ontt throughout the rest of the utterneF F F F F F F F F F F F F F F F F F IHU RFPW uplprotr nd lwlprotr sequenes for GbÕZuKG F F F F F F F F F F F F F F F F IHV RFQH tvdist nd tvcont sequenes for GbÕZuKG F F F F F F F F F F F F F F F F F F IHW RFQI he distne etween the tongue nd the velum when produing GKG F F F F F F F IIH RFQP vwdist nd vwcont sequenes for GbÕZuKG F F F F F F F F F F F F F F F F F III RFQQ twdist nd twcont sequenes for GbÕZuKG F F F F F F F F F F F F F F F F F F IIP RFQR ertiultory prmeter onsisteny in the synthesized 261 sentenes F F F F F F F IIQ RFQS ertiultory prmeter onsisteny in the synthesized 261 sentenesD roken down y spekers F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IIW RFQT h lignment etween the losest mthing originl nd generted rtiultory prmeter sequenes F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F IPS RFQU ertiultory prmeter onsisteny ompred on the originl sentenes tht were tken out nd the generted onesD roken down y spekers F F F F F F F F F F F F IQI SFI sdentil phonemes in the ws nd Ews dtsets F F F F F F F F F F F F F F F IRH SFP ertiultory omprison riteri F F F F F F F F F F F F F F F F F F F F F F F F F F F IRU SFQ petrogrm for the similrity mesures9 temporl ehvior exmple F F F F F F F IRV SFR he temporl ehvior of

EMD F F F F F F F F F F F F F F F F F F F F F F F F F F F F IRW SFS he temporl ehvior of SSIM vlues F F F F F F F F F F F F F F F F F F F F F F F F IRW vi
SFT he temporl ehvior of SIFT nd SIFT l vlues F F F F F F F F F F F F F F F F F F ISQ SFU wthing the stti smples of GfG ginst vrious dynmi ontexts of GfG F F F F ISR SFV yverll mesure distriution F F F F F F F F F F F F F F F F F F F F F F F F F F F F F ISS SFW wesure distriution for prtiulr phonemes F F F F F F F F F F F F F F F F F F F F IST SFIH gorret mthes identi(ed y SIFT F F F F F F F F F F F F F F F F F F F F F F F F F F ITT SFII snorret mthes identi(ed y SIFT F F F F F F F F F F F F F F F F F F F F F F F F F ITU vii List of Figures viii e hh thesis is journey tht you tke on your own ut see through to the end only thnks to olletive e'ort of those round youF ell prts of tht huge ontriution re importntF ometimes it is someone who gives dvie on how to uild your model or looks t your results nd sys if they gree with your onlusionF ometimes it is someone who listens to the ounts of n elusive ug nd suggests n ide or works their mgi on n dministrtive issue you re fing on top of everything elseF st my lso e someone who does not give up on you until they onvine you tht you n relly do itF yr someone who stys up until very lte just to tlk to youF yr someone who drgs you wy from work for o'ee rek nd mkes you smileF yr someone who stops y nd does the dishesF s hve never herd of nyone who would mnge to pull o' hh without the dishesF oD my thnks goes to my supervisor ves vprieD who took me through very ig hunk of my reer nd personl trnsformtion ndD despite ll hurdlesD mde sure s did not hve d timeF re is n dmirle exmple of someone who hs unique expertise in the (eldD loves wht they do nd delivers tht to the world with sense of justieD kindness nd humorF hnk youF s would lso like to thnk sonnis hourosD without whom this thesis would hve proly ended up eing defended round PHQI nd looking very di'erentF e ig prt of wht mde this work possile is owed to himF hnks to fenjmin ilieD whom s worked with on ousti simultionsD to esterios outios for helping out with their rtiultory ontour trking toolD s well s to other ollortors of the ertpeeh projetD espeilly ierreEendré uissozD uryn ssiev nd hinji wedF relpful dvie lso me from the disussions with eter firkholzD idwin ws nd sn rowrdF s m grteful for the very well developed suggestions for improvement mde y the reviewers of my thesisD sl errier nd hmien voliveD nd for the omments y wrtine eddEheker nd enne foyerF edditionllyD s relly ppreited the possiility to tlk out ll things tht hve to do with rtiultion with other memers of wultispeeh who work on itX héo fisuttoE vervtD r hhmniD eghils iniD wnfred stätterD lim yuniF pinllyD my grtitude nd respet goes to ll our interns nd volunteersD who showed lot of endurne through mnul nnottionD urgent oding nd the hrdships of dt quisitionF st is inevitle tht sometimes one gets stuk with n issue in softwreD hrdwre orD if you re relly out of lukD oth @repetedlyAF his is where my ig thnks goes to ejinky uulkrni nd eml rouidhek for their help with werlinD r hhmni for evite rD xiols urpult nd wnuel riente for qridSHHHD henis touvet for his generl guidneD nd proly mny othersD inluding the people from tehnil ssistne servies t voriD who t some point hd to visit my omputer on n lmost dily sisF sn generlD s need to sy tht vori nd in prtiulr our wultispeeh tem re gret ples to work tD full of people who mke for I mzing o0e mtes @speil shoutouts to smrn heikh for eing this weloming t my (rst o0eD edity erie xugrh for his ility to shre good lughD hyn is for trnsforming every ple she ppered tD emélie qreiner for our o'ee reksD quillume grjl for his fmous line nd vou vee for our shred selfEindulgent rntsA nd l mtes nd ollegues @espeilly wthieu ruD unit ivsnkrnD xiols purnonD wuriio wihel ylverD Élodie quthierD phël huroselleD hiego hi grloD fdr edullhD uhet qoshD enn urvhenkoD emeur houiD wohmed emine wenerD qeorge urit nd ermelle frunAF elsoD hertfelt thnks to sselleD rek nd groline from the nteen nd eroussik from the reeption desk t vori " they relly mde my dys righterF hen it is the turn of my personl fortressF e lot of wrm support me from uentin frnt @who would e there nd help me out with everythingD from prenh to ureury to deling with my tehing to ssemling furnitureAD enstsii hemikhovskD gristin érez ivsD elon wlukhinD eitor igurtzegiD pelii yerrzherD ni wohmmedD iolet gervntesD renndii veienkoD sekterin flginD evhen vosD éronique rukD srene toukouD uristin himnskD geline winierEfidrtD engel worenoD gitlin wqrth @whom s espeilly thnk for ll the lstEminute proofreding nd editing she didD nd y lst minute s do men last minute AD xikoli iuzinD enn sevlevD enn werkulov nd qilyn wndzhievF elsoD useni imshkovD oth on professionl nd personl levelF s feel tht eh of you rought in something ruilD something no one else would hve een le toD something tht is integrl to me nowF hnk youD you re the estF s hope tht one dy s will e le to py you kF wy hh rought me two inredile peopleX sordn sordnov nd tin wkhlovF foth re people of immense kindness nd ultimte inner integrityF s wouldn9t hve known wht s would hve missedD hd s not joined this progrmF sordnD thnk you for putting yourself out there nd relly getting to know those round youF s think s hve improved in tht deprtment over the ourse of the studiesD nd my role model ws you @lsoD you re the hero of ghpter Q thnks to tht emotionl te rek t our o0eAF niD there re so mny things tht we understnd in eh other thtD s feelD no one else n extly relte toF he feeling s ws not lone ws ruil to e le to (nishF foth hd their word of thnks in the demi prtX sonnis houros nd eml rouidhekF ou elong here tooF hnk youF wy tronus goes to rrun iljk for eing on the sme wvelengthD wve mplitudeD wve skewness nd wve everything s s mD for eing there for meD for mking me elieve ginD nd for showing me tht life does not end with hhF e er hug is given to er(m hherinD whoD ross ll kilometersD yers nd irumE stnesD ontinues to e the most loyl nd loving friend everF hen s get in whirlpoolD you help me rememer whih wy is upF XyX s would like to dedite speil pirozhenk to my fvorite pitnits prtner enstsi himorinD who is superhero in disguiseX she is the friend who will hek your linguisti resoningD write you odeD trnslte things etween ll lnguges imginleD orgnize you ny event you n think ofD tret you to her deliious ookingD welome you into her pleD mind nd hert nd then tell you she did not do nything worth of mentioningF xstiD s know s m lowing your overD ut it needs to e doneF ou re so genuine nd so uniqueF hnk you for everythingF eronik folshkovF ou re the sprkD you re the spieD you re the dneF ithout youD P everything is so lnd nd inertF ou re the only person s know who n look in the drkD fully tke in the monsters who live there nd then undo them with srsti remrkF st is thnks to none other thn you s know tht one should not wit for life to ommodte to eome the person they wnt to eY we re wht we re nowD nd wht mtters now is oth ig nd omplited things suh s shred serets nd profound dvieD nd simple nd seemingly smll onesD suh s who omes over for teF ell my te is yoursF pinllyD lots of love to my fmilyD who hve supported me in ll wys they ouldD followed my journey through ll the ups nd downs nd who only wished me the estF s m fortunte to know tht there is ple in this world where someone is witing for meD lwysF ht ple is homeF

Q 1 Introduction 1.1
Articulatory speech synthesis es tehnology gets more nd more inorported in our lives nd the mount of exeedingly heterogeneousD multimodl dt irulting round eh of us inresesD it eomes essentil to lern to swith etween dt modlities nd onvert one into notherF yne of suh types of onversion is tht of written text into the sound of speehX textEtoEspeeh synthesisD or F sn the modern soietyD this rti(il prodution of humn speehD long with onstelltion of other speeh tehnology tsks suh s speeh reognition nd understnding nd speker reognition nd veri(tion nd othersD is very well dopted nd deeply rooted in everydy useX we her syntheti speeh in our ell phones nd other gdgetsD ompny hotlines hevily rely on itD puli nnounements re synthesizedF st my pper tht the presently ttined nturlness nd intelligiility of speeh synthesis does not leve muh room for sienti( pursuitF roweverD there still re quite few hllengesD mong whih one n identify lowEresoure ndGor smllEfootprint speeh synthesisD speker or domin dpttionD expressivenessD treting the multimodl spets of speeh within the sope of spei( pplition or for prtiulr gins nd emring speeh synthesis reserh results within the umrell domin of speeh studiesF he lst two points re where we n situte the work on rtiultory speeh synthesisX the tsk to synthesize not only the speeh wve ut lso the movement of the rtiultors tht ould use itF 1.2 Motivation es pointed out oveD t present is rightfully uiquitousD its pplitions rnging from the most mundne suh s reding out loud your text messge while you re usy driving to the more osure suh s simulting Ötzi9s voie"the voie of mn who lived etween QRHH nd QIHH fg nd ws found on the order etween eustri nd stly in IWWI egpIUF roweverD there still re domins tht yet re to fully explore its ene(tsF yne suh domin is lnguge lerningF es is ommonly elievedD round hlf of the world9s popultion speks more thn one lnguge iurIUD yIQF sn the se of inglish loneD there re QUWFH million ntive spekers nd USQFQ million lerners ieIWF hese sttistis show dire need to stremline lnguge lerning proess so tht humn tehersD ntive nd nonE ntive spekers likeD n provide more imptful help to their studentsF here re spets to S their work tht nnot"t lest t present"e done y mhineX uilding rpport etween the teher nd the studentD getting retive with tehing methodsD providing originlD nonE mehni feedk to students9 workF st is ruil tht tehers hve enough time to dedite their e'ort thereF nfortuntelyD they often (nd themselves tsked to prepre study mterils nd exerisesD while tehnology should tully e lredy le to tke over t lest prt of this dutyF yne of the prolems to solve in this diretion isD indeedD the onversion of text into speeh nd kD s suitle for the (eldF xonethelessD the ontext of lnguge lerning dittes the need to tke into ount deeper understnding of how humn speeh is produedX for exmpleD when evluting lerner9s spekingD not ll pronunition mistkes should e treted like sine some ttempts re phonetilly loser thn others even when oustilly they my e quite di'erentY esidesD when orreting those pronunition mistkesD it my e neessry not only to give orretly performed soundD ut lso to guide the student through the rtiultion of the sound they re lerningF nfortuntelyD the two min rnhes of pprohes of "prmetri nd ontentive speeh synthesis"oth re quite tehnil solutions withD s of nowD no esy wy to introdue ny informtion on the wy the speeh orgns ould moveF enother domin with window of opportunity is mediineF hen investigting speking prolem tht is not esy to see or ptureD there is glring sene of tools for speeh therpist to simulte the suspeted dysfuntion in the vol trt nd see whether it would led to the phenomen they tully oserve in the ptientF elsoD when plnning surgery tht ould 'et the ptient9s speehD t present there is no ompletely relile wy for dotors to mke prognosis on its eventul qulityD while nturlly this question is ig onern for the wellEeing of the ptientF pinllyD it hs een shown tht for mny ptientsD the tretment of suh speeh prolems s misrtiultion frIRD wg + IRD vwIUD prxi wup + IHD uwqIHD fvIQ or fro9s phsi ufgWW my e filitted with iofeedkD generlistion nd mintenne eing espeilly promising in the se of hildrenF uh tretment requires regulr sessions on siteD whih mkes intensive trining hllengingF ith oustiEtoErtiultory inversion nd rtiultory speeh synthesisD howeverD the sessions would not require nything ut n esily portle piee of softwre or even online ess to loud omputerD thus giving this method n unprlleled oostF yn top of suh prtil pplitionsD there lso re deep reserh questions in speeh studies tht hve remined open for dedes t lestF por instneD wht re rtiultion units nd how do we proess them mentllyc ho we hve ny onept of rtiultory trgets nd wht hppens to them in )uent speehD when neighoring sounds in)uene eh other nd there is not enough time to omplete speeh movements s rtiultely nd preisely s we ould do for stndlone soundc ht re the onsiously ontrolled ftors in rtiultionD nd wht is just the e'et of the lws of physis in plyc sf we hve model oneptulized with prtiulr outlook on these questions nd it produes su0iently nturl nd intelligile resultsD rtiultion n serve s ridge etween the iomehnis of speeh s mehnil proess nd its linguisti spets nd give us lueY nd vie versD if model is uilt fully omptily with how speeh sientists understnd speeh prodution nd it gives unexpeted resultsD it is very resonle to revisit the knowledge nd look for lterntive theoriesF he prolem isD stteEofEtheErt systems re uilt with no reltion to ny speeh prodution theories nd nnot help give stisftory nswer to these questionsF ell the points ove is wht mkes me rgue tht even while the qulity ttined y presentE T 1.3. Problem statement dy systems is unpreedented nd speeh synthesis n sometimes e on pr with rel spekerD we need to proeed further nd ugment the sound of speeh with other modlitiesD spei(lly rtiultionF st must e pointed outD howeverD tht rtiultion mostly hppens hidden from the viewD whih mkes dt olletion di0ultF ome methods re dngerousD some re too invsiveY some re notD ut mke it too unomfortle for the sujet speker to spekY some re not fst enoughD nd some mnge to keep up with the speker9s movementsD ut only in some spei( pointsF henever method is e0ientD it seems to lso e expensive nd ome in with herulen tsk of dt nnottionF yn top of thtD treting rtiultion within most speeh prodution theories indues systems with highly onvoluted pipeline tht is hrd to tuneF ropgting errors n esily umulte to the point where the syntheti speeh is of underwhelming qulityF he result is tht despite the need for dopting rtiultory spets in speeh synthesisD the topi hs een out of the spotlight in speeh reserh ommunityF portuntelyD with the dvent of onsiderly more informtiveD detiledD verstile nd voluminous rtiultory dt suh s relEtime mgneti resonne imging @EwsAD the (eld is in its prime to tkle these issuesF 1.3 Problem statement he present disserttion ims to work towrds fullE)edged rtiultory speeh synthesizerD ple of treting vriety of speeh phenomen nd fithful to the e'ets of in)uene of nery sounds on eh otherD ortiultionF en input of suh system is textF he generted output is speeh nd the synhronous rtiultory informtion relted to itF he ojetive is to explore two kinds of pprohesX one drwing on the ide of frozen rtiultory trgetsD sed on rih system of rules governing trnsitionsD nd nother on treting rtiultory prmeters sttistillyD with no expliit knowledge on their interprettion put inside the systemY thenD to mke link etween the twoF hese gols onstitute multiEstep piee of workD whih indues numer of suEojetivesX sdentify n pproprite type of dt for eh of the ojetives nd prepre the dt E ordinglyF fuild the textEtoErtiultory speeh system in ordne with the priniples of speeh prodution nd the pprohes ommonly undertken in speeh synthesis ommunityF ivlute the results visullyD pereptullyD oustilly nd numerilly " not only sepE rtely for the two pprohesD ut lso mening to ompre themF hrw onlusions on the reltion etween the two pprohes nd ompre their e0ienyF

1.4 Contribution his disserttion strts with the work on rtiultory speeh synthesis from stti mgneti resonne imging @wsA dt pturing ontextEwre rtiultory trgetsD presented in ghpE U ter QF he presented method is primrily ruleEsedD essentilly solving the prolem of how to trnsition etween ptured stti ontextEinformed vol trt on(gurtionsF hen ghpter R follows with deep neurl networks @hxxAEsed prmetri rtiultory speeh synthesis sed on relEtime ws @EwsA dtF rereD no onept of rtiultory trgets existsD nd rtiultion is synthesized the sme wy s speeh in prmetri speeh synthesisF ghpter S uilds ridge etween the two pprohes to investigte the reltion etween the stti nd dynmi dt used in the work of the previous two hptersF pinllyD ghpter T gives (nl overview of the results of this work nd the lessons to e drwn from itD s well s its potentil extensionsF he pulitions mde over the ourse of the hh studiesX peeh prodution n e seen s mnifesttion of the ompromise etween the omplexity of wht weD s humn eingsD need to e le to expressD nd the limited pities tht re t our disposlF he proesses involved in speeh prodution re intrite just enough for the simpli(tions nd shortuts not to hinder hieving the purposes of ommunitionF he two ontrditory foresD one to simplify the underlying proesses of speeh nd the other to hve enough sutlety to deliver messges s nuned s they were intendedD re wht drives lnguge hnge t ll of its levelsF his dittes wht we expet to lern in speeh produtionX we shll (nd evidene of ertin vrition nd )uidity tht men to redue the strin of the proess of speehD nd tht vrition will onstntly test the physilD iologil nd ognitive onstrints fed oth y the speker nd y their listenerF qenerlly spekingD one n identify severl suproesses in produing speehX

Conceptual thinking X the thought proess tht predtes speeh nd genertes the speker9s intentY

Lexical and grammatical selection X (ltering the speker9s intent through the ville mens of the lngugeY Movement planning and execution X prepring nd exeuting oordinted ommnds for speeh orgnsX to ontrol the rethD operte the vol folds nd put the rtiultory orgns where they need to e to produe the desired outputY Movement correction X proessing mismthes etween the intendedD possily updtedD outome of ny of the lyers oveD nd the speeh tht is produedY orreting themF sn running speehD ll these suproesses our t overlpping timesD nd it is through reE fully ontrolled experiments tht we n disssoite themF uh experiments n e ognitiveD neurosienti(D rtiultory nd ousti in ntureD nd they del with suh prmeters s feE ture timing @eFgF voie onset time"yAD rtiultor positioningD phoneti feture presene or seneY the evidene n ome in the form of timing di'erenes nd hnges in the ple nd mnner of rtiultionF yne prtiulrly illustrtive se of hnges is speeh errorsF W he ontriutions of the present thesis re situted t the third levelD movement plnning nd exeutionF roweverD other elements lso need to e exmined nd tken into ountF he levels of oneptul thinking nd of lexil nd grmmtil seletion re disussed in ghpter PFIFIY the level of movement plnning nd exeution in ghpter PFIFPY (nllyD movement orretion in ghpter PFIFQF 2.1.1

Accessing language resources peeh on its own does not signify nythingF st is olletion of soundsF ht mkes it meningful is the experiene of the environmentD its reltion to the speker9s inner worldX wht it mkes them think nd feelD nd how wht they sy n in)uene nd hnge it xeIQF row do we trnsform the sense of self nd the desire to intert with the world in ertin wy into the t of spekingc st ll strts with the rinF he rin is n orgn in the hed tht serves s the enter of the nervous systemF st is divided into vertil hlves lled hemispheres nd omes out of the rin stemD the pssge to the spinl ordF he mtter overing hemispheres is lled the ortexD nd it is wht provides for higher ognitive funtionsF he evolution of the rin preferred the ortex to hve lrger surfe re while oupying less speD whih is why this tissue is lid in folds nd hs grooves nd wrinklesF e n identify some prtitioning in the rinX (rst into loes @frontlD temporlD prietl nd oipitlA sed on physil feturesD nd then we n single out spei( res tht serve n identi(le funtionF qenerlly spekingD in the mjority of people the dominnt hemisphere is the leftD nd it dels with lngugeD logil nd nlytil opertions nd mthemtisF wenwhileD the right hemisphere opertes emotionsD reognizing fesD pereiving strutures glolly @without detiled nlysisAD musi nd nonElinguisti soundsF roweverD when we tke more detiled lookD we (nd tht this division is not s lerEutD nd oth hemispheres perform some funtions ssoited with their ounterprt xeIQF yne importnt region for speeh prodution is situted in the lower k prt of frontl loe of the dominnt hemisphere @typillyD the leftAF st is lled fro9s reD nmed fter prenh pthologist nd neurosurgeon ierre ul fro @IVPR!IVVHA who studied itF yne the stimulus for reting n utterne is formed"lredy with or yet without the syntxD depending on the omplexity of the sentene uHPD"this is where speeh is formulted xeIQF sf fro9s re is dmgedD it uses type of nonE)uent phsi tht is lled fro9s phsiF sn most ses speeh eomes telegrphi nd dis)uentD mde up of ontent voulry words stked together insted of eing properly joined into synttilly orret sentenes @for exmpleD Carry, couch, Sophie insted of Sophie and I had to carry the couch.A he ptients feel they know wht they wnt to syD ut nnot get it outD neither orlly nor in writingF st my even hppenD suh s in the histori se of veorgneD tht whenever the ptient tries to spekD only limited omintion of sounds will ome out @suh s the word tnAF peeh omprehension is prtilly impeded s wellD despite the ft tht this is not the re primrily responsile for itF hsgHUD xeIQ sndeedD region tht proved to ply mjor role in understnding of speeh is ernike9s reD nmed fter qermn neurologist grl ernike @IVRV!IWHSAF tust s fro9s re is right next to the motor ortexD ernike9s reD eing situted in the upper k prt of the temporl loe nd extending upwrds into the prietl loeD is losely onneted to the uditory re in IH the temporl loe through (ers of the rute fsiulusF he sound of the word is trnsmitted from the er to the uditory reD nd then to ernike9s re for proessing xeIQF end k in reverse for speeh produtionD it is thought tht this is in ernike9s re where the si struture of the utterne is onstruted efore eing pssed to fro9s re disussed ove xeIQF hmge to ernike9s reD espeilly in the left hemisphereD uses ernike9s phsiD whih is )uentF tients tend to spek in drgged out sentenes tht hve no meningD nd sometimes they lso use mde up or irrelevnt wordsF heir utternes mke sense to themD nd they do not see their speeh errorsF he two )ows of informtion"for prodution nd for pereption"form two stremsX the ventrl strem @speeh omprehensionA nd the dorsl one @speeh produtionA rHHD rHRD rHUF hey re symmetriX speeh omprehension essentilly depends only on the uditory system rIWD while speeh prodution lso relies on somtosensoryEmotor systemsD using mpping from ousti speeh signls to prietl nd frontl loe rtiultory networksF he funtion of the left nd right hemispheres of these two )ows di'erX in speeh omprehensionD despite plying di'erent roles @understnding the mening nd treting the intontionD respeE tivelyA peHRD igIWD oth hemispheres re involvedD nd in speeh produtionD the left hemisphere domintes hevilyF i'etivelyD fter the lifelong trining side y side nd ssisting eh other ll long the wyD the deep levels of memoryD speeh prodution nd speeh omprehension proesses get tightly intertwinedF 2.1.2

Movement planning and execution yne formed in fro9s reD the tivtion is trnsferred through the nerve (ers of the rE ute fsiulus to perform phonologil enoding in left frontl ortil regionsD inluding the operulumD insulD lterl preEmotor ortex nd nterior supplementry motor re fqrHIF he motor ortex ontrols the musles orresponding to the vol trtX the tongueD lipsD jwD soft plteD vol ords nd others xeIQF rere the mehnis of speeh is ontrolledX it is produed through mnging ll speeh orgns together with ping the rethF st is remrkle tht this system originlly developed for rething nd eting rther thn for onsiously mkE ing ny kind of meningful signlsD whih rings n dditionl ontriution to the omplexity of the speeh prodution orgniztion in the rinF pigF PFI shemtilly illustrtes midEsgittl setion of n dult9s vol trtD with the orgns involved in speeh produtionF es sid eforeD these orgns move ording to the ontrol from the rinD ut how is this ontrol orgnizedc yne of the entrl issues in speeh prodution is the notion of n elementry speeh moveE mentF prom the uditory perspetiveD the elementry unit is phonemeX the smllest unit of sound ple of hnging the mening ut not ering ny mening of its ownF ith ertin greement rteD oundries etween phones n e piked through the temporl nlE ysis of the spetrum frequenies quite ertinlyF he time spns of trnsition from one phone to nother tht re truly unstle nd impossile to identify one wy or the other generlly re very short @more on the physil point of view on speeh nd how it is rried out is to follow in setion PFIFPAF his is not t ll the se for movementsF vet us tke sequene GfaG s n exmple @pigF PFPAF sn the spetrogrmD the presene of fritive noise GfG is very distint from the regulr frequenies of the vowel GaGY this is due to the threshold e'et triggering the II pigure PFIX rumn speeh mehnism emIH turulene in the vol trtF roweverD within the tul vol trtD the movement is smoothF st ould e nlyzed in two prinipl wysX his syllle n e seen s stored s two trget on(gurtions of the vol trtX for GfG nd for GaGF hen the trnsition is just grdul trnsformtion of the (rst shpe into the seondF ht is stored is the wy we rry out this trnsitionD the gesture for it on the level of individul rtiultors tht reD howeverD neessrily oordintedF his gesture is exeuted just enough to produe the desired sensorimotor outomeF foth views hve given rise to vriety of theoretil nd pplied models disussed in ghpE ter PFPF xo mtter how the speeh orgns re opertedD the (nl result is tht t every single moment of speeh or preprtion for it the speker ttins their prtiulr on(gurtionF he following susetionD ghpter PFIFPD disusses wht hppens thenF IP pigure PFPX en exmple of spetrogrmX GfaGF he oundry etween GfG nd GaG n e seen t round 1170 msX the fritive noise stopsD formnt nds pperD nd the wveform exhiits the onset of voieF Emitting speech sounds he ruil elements for produing sound of ny kind re soure of n ousti wveD propgtion mediumD nd the presene of this medium9s oundryF por humnsD this is their respirtory system @the soure of pressureAD the irD nd the vol folds @ virting elementA long with the vol trt tht is le to produe onstritionsF sn terms of ousti nd eletril engineeringD it mens tht we my desrie the speeh wve in terms of the soure nd (lter hrteristisX the humn vol trt is soundEemitting (lter system tht responds to one or more sound souresD whih n e written s the following equtionX

| P(f) |=| U(f) | • | H(f) | • | R(f) |, @PFIA
where f is the frequenyD | • | is the module funtionD | P(f) | is the sound pressure spetrum t distne from the mouth openingD | U(f) | is n mplitude versus soure frequeny hrE teristisD nmely volume veloity spetrumD | H(f) | is the frequenyEseletive gin funtion of vol trnsmissionD nd | R(f) | is the rdition hrteristis t the lips onverting volume veloity lulted on the mouth opening into sound pressureF his vol trt interprettion is the foundtion of the source-lter theory of voie prodution pnUIF Respiration he respirtion system of humn involves orgns suh s the trheD ri geD thorxD domenD diphrgmD nd lungsF he mehnis of rething is lrgely explined y Boyle's lawD whih sttes tht if gs is kept t onstnt tempertureD pressure nd volume re inversely proportionl to one nother nd hve onstnt produtY it mens tht we n regulte the pressure of the ir in the lungs y expnding nd reduing their volumeD sending the ir into nd out of the lungs emIHF et restD the pressure within the lungs @lveolr pressureA mounts to the tmospheri oneD nd the diphrgmD whih is the prinipl musle of inhltion nd n ntomil divider IQ pigure PFQX he prinipl musles of respirtion emIH etween thori nd n dominl vityD is not tenseD pertining the form of n inverted owlF henD the exterior ir is drwn into the orgns of respirtionD iFeF the lungsD y ontrtion of the posterior nd nterior musle (ers tht drws the entrl tendon downwrd nd shifts it forwrdD thus expnding the hest vityD lowering the diphrgmD nd elevting the pressure in the dominl vity @see pigF PFQAF ith n inrese of thorx volumeD s the lungs press upon the wlls of the thorx thnks to sutmospheri pleurl )uid pressureD the pressure in the lungs eomes negtive with respet to the tmosphere emIHF he ir goes down the respirtory trtD freely nd diretly pssing the orl nd nsl vitiesD phrynxD lrynxD trheD nd ronhiY the inhltion musles grdully relxD tivting the pssive fores of exhltionF yne the outside nd inside pressures re in lne due to the nturl physil limittions of the trhe nd ronhil treeD reltively high pressure in the dominl vity tends to restore the relxed shpe of the diphrgm s well s the ris nd soft tissuesF ith this ir inhledD the resoure for produing speeh is villeY it is used either for silent exhltion or for speehD nd the )ow volume will e proportionl to the di'erene etween tmospheri pressure nd the pressure within the lungsF o rethe out or tully spekD humns expel the drwn ir y ontrting the ri ge whih dereses the volume of the thorx nd onseutively inreses the pressure in lungs @the greter the lung pressureD the louder nd more highEpithed sounds omeA nd pushes the ir outD up through the trheD into the phrynxD throt vity plIQF he lveolr pressure flls from round RH m r 2 y when exhling pssively or s high s PHH m r 2 y when dding musulr e'ort to negtive vlues t low lung volumesF es for speeh produtionD it requires n IR ir)ow rought upon y n lveolr or suglottl pressure in the rnge of S to PH m r 2 yF peeh n our t highly vrile lung volumes @we my even spek 4out of our reth4D for exmpleD when sentene ends up eing longer thn intended (rstA emIHF wost of the utternes re pronouned on expirtionY ingressive soundsD for whih the irstrem )ows inwrd through the mouth or noseD re rre vwWVF o in order to void exhusting the drwn ir mid n utterneD humns hve to regulte their lveolr nd suglottl pressure nd hek themselvesD ontrting the inspirtory musulture to mke the expirtion slowerF st is this system of rtionl usge of inspirtory nd expirtory musles wht llows us to spekD ltering the vitl proess of nturl rethingF Phonation hen leving the trheD the ir psses the lrynxF sts rtilges hold two folds of ligment nd musulr tissue whih re lled vol foldsFhe opening etween them is the glottis nd serves s gte for the ir)owF hue to their moilityD the vol folds re soure of highly vrile resistne for the ir )ow whih instigtes the speeh soundsF hen the ori(e etween the vol folds is losedD it mens tht there is soure of greter resistne on the wy of the ir )owD nd t lest some of the ir will ome kD rising the lveolr nd suglottl pressures even higherF gonsequentlyD the ir )ow will get only hevierD until it (nlly fores the vol folds prtD letting the ir)ow pss throughF henD ording to the fernoulli9s lwD the lol pressure fllsD urging the ords lose up ginF ith the )ow reduedD the lol nd suglottl pressures mount to eh other s in the eginning of this yleF oD the vol folds open nd lose rpidly on loop for speker to produe voied soundsD iFeF to phonteF his de(nes the period of the osilltion fored onto the ordsF sn ontrstD to produe n unvoied soundD the vol folds neither lose together nor virte"they sty open instedF he rte of vol fold virtion is desried s voie musil toneD or pith @pereptullyAD or s fundmentl frequeny mesured in rz"yles per seond @physillyAF he sso voie orresponds to TH rz or lower or f 1 in the musil sleD nd y rising the voie up to the soprno register one will reh the frequeny of over ISTV rzD or q 6 emIHF por every individul spekerD the qulity of the voie rnges with vol fold virtion freE queniesF here is omfortle middle or modal pitch rangeF et its upper limits the qulity of the voie suddenly hnges into the falsetto registerD lso lled loft registerD orD possily for femle soprno singersD laryngeal whistleF sn flsettoD the ontt re in the vol folds is muh smllerD nd the glottis turns into tense nd nrrow slit tht virtes only t the edgesF he mehnism of lryngel whistle is the sme s of flsettoD ut with higher tensionD pressure nd resulting frequenyF es for the lower limits of the modl pith rngeD the voie hnges there into glottal fry or pulse registerD whih gives the e'et of reky voieF o produe itD the vol folds re drwn together tightlyD ut let suglottl ir ule up etween them in disrete ursts in synopted rhythm wvSVF wthemtil models of the lrynx for speeh simultion inlude singleEdegreeEofEfreedom model y pvTV where the vol folds must move s single mss towrd nd wy from the midline @with one degree of freedomD hene the nmeA whih n e simple solution ut does not desrie ehvior of the rel lrynxY twoEdegreeEofEfreedom models suh s y spUP where the vol folds re two msses insted of single oneD ple of n independent horizontl motion whih is etter ut not devoid of rtifts nd unrelisti onsequenes for the prmetersY nd the sixteenEmss model y itUQ tht ws to tke into ount the muos in the virting lrynx nd llow more degrees of freedom for the vol foldsF IS he importnt prmeters of voie prodution re s follows emIHX IF wximum pith rngeX how )exile is the voie pithc PF wen rte of vol fold virtionX wht is the most omfortleD hitul pith for the spekerD in reltion to the pith rngec QF eir ostX how long n the speker phonte omfortly without running out of irc RF winimumEmximum intensity t vrious pithesX long the frequeny rngeD how do the sound pressure level mesurements hngec SF eriodiity of vol fold virtionX wht is the nturl period of vol fold virtion when other prmeters re onstntc TF xoiseX re there noisy res in the sound spetrumc row re they relted to intentionl voie qulities suh s horsenessD rethinessc UF pinllyD resonneX how does the vol trt resonte when the ir is propgted from the lrynx to the mouth openingc he (nl point onerns the positioning of the further vol trt rther thn tht of the lrynxD whih rings us to the next setionF Articulation oD phontion involved virtions of the vol folds tht n essentilly e summrized in prmeters of frequenyD intensityD nd durtionF o otin the speeh soundD there hs to e resontor to reeive the pu's of ir from the lrynxF hile the fundmentl frequeny is de(ned y the rte t whih the ir olumn is driven into osilltionsD it is the form nd dimensions of the ousti ojet wht estlish the resonting frequenies nd in suh wy determine the qulity of the toneF esontion is wht the vol trt serves for in sound emissionF he vol trt strts just ove the glottis ndD from n ousti point of viewD is n ousti tueD round IU m long for n dult mleD with vrying rossEsetionl reF he vol trt onsists of the orl trt nd the nsl trt nd ends with lips nd nostrils respetivelyD from where the sound is propgted in the tmosphereF hen emitting voied soundD the vol trt reeives qusiEperiodi pulses of irF ine the glottl ori(e is reltively smllD its ousti impedne is domintingD nd unless there is pronouned onstrition in the further vol trtD the glottis is the min soure of turuleneF ytherwise ostles on the wy of the ir)owD whih re mde y positioning the rtiultors tht ompose soure of widely rnging resistne to the ir )ow @from minimlD suh s for open vowelsD to neutrlD suh s for uttering sound like 4uh4D nd soluteD suh s the moment of onstrition t the lips to produe bAD ring out vorties"their experimentl evidene in the speeh ir)ow ws provided y hoVTF hen these (ndings were theoretilly supported y WH nd wqVVF orties n ourD for instneD due to hnges in the veloity of the )ow t the oundriesD )ow disruption suh s y dverse pressure from vityD )ow seprtionD or pperne of rottionl motion euse of moving through the urved form of the vol trtF hen formedD vortex n twistD streth or spred further downstrem wrWRF IT sntensifyingD these e'ets will eventully led to turulent )owF he hrteristi tht helps predit whether the )ow stys lminr or eomes turulent is the eynolds numerD noted s ReX Re = inertial forces viscous forces = ρ UL µ , @PFPA where ρ stnds for the ir densityY U is veloity sleY L is liner sle suh s the dimeter of the vol trtY nd µ is the ir ulk visosityF vow vlues of the eynolds numer re ssoited with lminr )ows"the )ows where the visous fores dominteF e high vlue of the eynolds numer will indite turulent )ow with hoti instilitiesF hese e'ets re neessry to produe fritionD spirtion nd whisper or ontriute to the e'et of rethy or reky voieF o operte the ir)ow nd di'erentite the resulting sounds of speehD humns position their articulators so s to mke onstrition of the )ow t prtiulr ple nd in prtiulr wyF his delierte sound formtion is lled articulation Y every phoneme hs ple nd mnner of rtiultion ssoited with itD though some degree of freedom is llowedF he phoneti detils nd how they ply out in the lnguge will e overed elow vhIPF pigure PFRX rumn vol orgns vtIR pigF PFR shows n outline of the vol trtF he vol trt omprises (ve 1 resE onting vitiesX the ulD orlD phryngelD nd two pired nsl onesF ine they re inE teronnetedD the division is mde from the ntomil perspetiveF ertiultion n e formulted in terms of operting the vitiesX s the speker rtiultes to produe speehD the vities n grow or diminish in volume up to omplete lokgeD nd this hnges the ousti properties of the vol trtD nmely the resonnt hrteristisF he result is sound s intended y the spekerD with orE ret energy distriution @how muh energy onentrtes t whih frequenies"oth sE pets re pereptully importntAF he buccal cavity is the spe extending from lips nd heeks to the teeth nd gumsF st is onneted to the orl vity through the spe etween the teeth nd ehind the lst molrsF sts volume vries etween sujetsD ut is smllF he oral cavity is ounded y the roof of the mouthD the teethD the glossopltine rhD nd the musulr )oor whih is mostly the tongueF hue to the tongue9s )exiility nd high moility of the lipsD during speeh the volume of this vity vries gretlyF his vity ommunites with the phryngel nd nsl vities through port lled phryngel isthmusD ounded y the nterior fuil pillrsD soft plteD nd the dorsum of the tongueF he pharyngeal cavity proper extends over the phrynx @see pigF PFSAD whih is vertilly ligned musulomemrnous tueD ovl in trnsverse setion @wider in the frontl plne nd more nrrow in the sgittl oneA nd rehing from the level of the sixth ervil verter @the posterior positionA nd the rioid rtilge @the nterior oneA to the se of the skullF he muous memrne of the tue ontinues into the one of the nsl vityF e de(ne three mjor regions in the vity of the phrynxX the nasopharynxD the oropharynxD nd the laryngopharynx @see pigF PFTAF he nasal cavities re two pproximtely symmetril hmers with the nsl septum etween themF enterior nres re nostrilsD the wy from the nsl vities to the exteriorF osterior nres re honeD the wy to the nsophrynxF he superiorD middleD nd inferior nsl onheD rrnged in lyrinthElike wyD long with their nsl pssges omprise lterl wlls of the vitiesF here re passive articulators tht nnot hnge their position"the upper jwD the hrd plteD the teeth"nd active articulators tht re free to moveX the lower jwD the lipsD the tongueD nd the velumF he mandible is very importnt rtiultor"the only truly movle one in the feF xot only does it di'erentite vowels y the degree of opennessD it lso helps enunite sounds etterF he jw minly rises nd fllsD though it n lso e protruded nd retrtedD or mke grinding motionF he jw is set on the temporomndiulr nd ginglymorthrodil jointsF he vol trt ends with n ori(e formed y the lipsF hey onsist of musulr nd grndulr tissues nd some ftF yf the twoD the lower lip is fster nd more moileF o prtiipte in speeh produtionD the lips n lose nd open nd protrude nd retrtF he tongue is the most )exile vol orgn with gret degree of freedomD whih plys suh n importnt role in speeh rtiultion tht in mny lnguges the word 4tongue4 hs eome synonymous to either 4lnguge4 or 4speeh4F sn the tongueD we disern the tipD ldeD nd odyD nd the tongue ody is divided into the frontD enterD nd k further thenF he tongue moves thnks to n intrite nd omplex system of extrinsi nd intrinsi muslesD usully one or two musles dominting in prtiulr motionD nd other musles grdully tking hrge when it is their orderF rrUT sees most tongue movements s omposed of just seven omponentsX horizontl nd vertil forwrdEkwrd nd upwrdEdownwrd movements of the tongue tip nd the tongue ldeD two prmeters for the trnsverse rossEsetionl on(gurtionD nd the form of the tongue dorsum plne"spred or tperedF sn the roof of the mouthD we single out smll ridge ehind the upper front teeth lled the alveolar ridgeD hard palateD nd soft palate @lso lled the velum AF he velum n lok the ir)ow from pssing into the nsl vityD whih is neessry to produe n orl sound @then the velum goes upA or let the ir through @then the velum goes downD nd the sound is nslAF e )eshy extension t the k of the soft plte whih hngs ove the throt is lled the uvulF fehind nd elow the k of the tongue omes the phrynxD whih n lso e n rtiultor or e ignored y the lngugeX for instneD prenhD on the mterils of whih the present disserttion is uiltD does not feture ny phryngel sounds tht would e signi(nt on the IV pigure PFSX pper irwy ypeIQ nomintive levelF hen there is )p of rtilge ehind the root of the tongueD whih is depressed during swllowing to over the opening of the windpipe"this is the epiglottisF Places and manners of articulation he two sudisiplines of linguistis whih study sounds of humn lnguges re lled phonetics nd phonologyF hile phonetis dels with the physil nd physiologil spets of soundsD phonology trets sounds s prts of prtiulr lngugeD disregrding the informtion tht is linguistilly irrelevntF vet us disuss the overll piture (rstD nd then go further into the spei(s of the phonetis of the lnguge treted in this disserttionD prenhF he smllest distintive unit of speeh is lled phonemeF st does not rry mening of its ownD ut n distinguish t lest one word from nother in the prtiulr lnguge where it elongs toF he mnifold of ll ousti vritions of phoneme omprises its allophonesF honemes of nturl lnguge re usully not in oneEtoEone reltion with the written system of this lngugeD nd thus phonetiins"sntWW"hve introdued speil lphet to yn the other hndD consonants re pronouned y mens of reting n ostle on the wy of the ir stremF o get pst the ostleD the ir strem hs to e hevyF ynly those rtiultors tht re responsile for the ple of onstrition re tenseD nd the others re lxF gonsonnt prodution depends on fst nd impertively preise motions of rtiultorsF sully the trnsition from voied onsonnts to vowels in terms of the degree of onstrition is grdulX the lnguge9s phonemi ontinuity mkes sure tht there re no rupt jumps in the openness G losure rnge vtIRF owels n e lssi(ed sntWWX by height X the lowest resonne of the voie"the (rst formntD ssoited with the vertil position of the tongue with respet to the roof of the mouthD orD lterntivelyD the degree how open the jw isF he more open vowel isD the higher is the frequeny of pIF owels n rnge from close ones"when the tongue is lose to the roof of the mouth"to open ones"when the jw is lowD openF here re seven degrees of vowel heightF by backness X de(ned y the seond formnt of the voieD ssoited with the position of the tongue reltive to the k of the mouthF he more front vowel isD the higher is the PI frequeny of pPF owels n rnge from front ones"when the tongue is forwrd in the mouth"to back ones"vie versF here re from (ve to seven degrees of vowel knessF by roundedness X de(ned y the third formnt of the voie nd indiretly ssoited with the rounded or unrounded lipsF by nasality X vowels n e oral or nasalD depending on whether the velum is rised or lowered nd whether the nsl trt is prtiipting in the vowel produtionF by movement of the tongueD by voicingD by secondary constrictionsD by tenseness F F F pigure PFVX he vowels9 hrt sed on their min fetures sntWWF he vowels9 hrt is given in pigF PFVF xturllyD the ft tht vowels n e rrnged oth in the rtiultory nd the ousti spes hs led to reognition of cardinal vowels s the extreme ones tht ll others n e ompred toF tonST suggested set of eight vowelsD nd grTV updted this notionF hree vowels in this setD GAGD GiGD nd GuGD hve rtiultory de(nitions outside of the sope of ny prtiulr lnguge on irth nd my e lled corner vowelsD nd the others re rrnged etween them so tht they divide the ousti spe into evenEsized resF hen vowE els of ll lnguges n e set in this vowel speD expressed through the cardinal onesF

gonsonnts n e lssi(ed vtIRD vvWRD sntWWX by voice into voiced nd voiceless X the ones during whih the vol folds virte nd the ones during whih they do notY by place @see pigF PFUAX t eh rtiultor onstritionD speeh sound n e formedD whih my or my not elong to the lnguge phoneme inventoryF gonsonnts rtiulted y the lips re labial with further distriution into bilabialD labiodentalD dentolabial depending on how the sound is produed @with oth lipsD with the low lip ginst the upper teethD or vie versAD nd othersF gonsonnts rtiulted y the tongue reD in se of the rised tongue pex nd ldeD lled coronalD nd in se of the tongue dorsumD dorsalF hepending on where the tongue tip must e putD mong othersD we pply the orresponding termsX linguolabial @the tongue tip in ontt with the upper lipAD interdental @the tongue omes etween the teethAD dental @the tip of the tongue or the tongue lde omes in ontt with the k surfe or ottom of the top teethAD denti-alveolar @the tongue touhes the upper prt of the k surfe of the top teethAD alveolar @the tongue is in ontt with the lveolr ridgeAD postalveolarD retroexD palato-alveolar @the tip or the lde of the tongue omes in ontt with the k re of the lveolr ridgeAF por the tongue dorsumD the pplile terms re palatal @when the front of the tongue rtiultes with the domed prt of the PP hrd plteAD velar @with the soft plteAD uvular @with the very k of the soft plte nd uvulAF hen the sound is produed with the tongueD there is further dihotomy depending on the tongue shpeF sf the ir )ows ross the entre of the mouth over the tongueD the onsonnt is lled centralF sf there is onstrition t the enter of the tongue nd the ir prts to )ow long the sides of the tongueD suh onsonnt is lateralF gonsonnts pronouned t the phrynx re pharyngealD produed y the ful pillrs moving together or rising the lrynxF Glottal rtiultion ours diretly on the vol foldsF by manner vtIRD vvWRD sntWWX Fricative onsonntsX produed y exittion of noise y mens of nrrowing the vol trt t the point of rtiultion without omplete ostrution of the irwyF his ostrution genertes turulent ir )owD whih is pereived s slightly hissing noiseF gommon wys to mke fritive re to mke the tongue pproh the teeth or the lveolr ridge or mke the lower lip pproh the upper teeth"tullyD in generlD ny other two rtiultors tht n ome lose enough to eh otherF Stop onsonntsX produed in three phsesX £ Catch X rtiultors ome into the onttD mking omplete losure in the vol trtF his losure n e lilD lveolrD pltlD velrD nd glottlF £ Hold X even if the point of ontt is not immoileD the rtiultors sty tightly loked nd do not lek the irF he rtiultors re tenseF he ir is eing umultedF he pressure risesY £ Burst X the speker lets the pressure fore the rtiultors prtD nd ll the drwn ir gets momentrily relesedF his explosion is the most helpful perepE tul ue to identify stop onsonntF tops n e either oral "then their ehvior is s desried ove"or nasalF xsl stops re produed with onstrition somewhere in the orl vityD fter the velophE ryngel portF prom the temporl point of viewD they re just like orl stopsF fut sine the velum is opened wideD the ir is not umulted s muhD sine it goes to the nsl trt nd rdites from the nostrilsF Aricate X stop immeditely followed y fritiveF Approximant X one rtiultor is lose to notherD ut the nrrowing is wide enough to void turulent irstremF Glides and semivowels X the glides re dynmi sounds tht re produed on the vowel they preedeD nd semivowels re very muh like vowelsD ut only with greter degree of onstritionF ythersX trills, taps or aps, clicks F F F Phonetics of the French language es we sidD the setion ove desried the generl piture of how the phoneti orgniztion of prtiulr lnguge my look likeF his is importnt to onsider to produe speeh synthesis PQ systems tht re not for oneEse use nd n e generlized for other lngugesF roweverD every lnguge hs its own spei(sY n issue tht my e negligile for generi lnguge in the grnd sheme of thingsD n e ruil in this prtiulr seF his is why it is fundmentl to lso go into detils relevnt for the lnguge in questionD prenhF he phonetis of the prenh lnguge hs the following distinguishing hrteristisX e reltively lrge numer of vowels @(fteenAY pour degrees of vowel opennessX openD openEmidD loseEmidD nd loseY wost vowels re openD nd most vowels re roundedF sn ftD lip protrusion n e exE tremely pronounedD suh s in the se of GyGY here re oth orl nd nsl vowelsY he vowels re rtiulted very lerlyY owels retin their properties fully nd re not reduedD prenh eing syllleEtimed lngugeY fefore puseD onsonnts re pronouned very lerlyD often with tre of G@GF hey re not unvoied like in some other lnguges suh s ussinD qermn or some ents of inglishY essimiltion of vowels y opennessY essimiltion of onsonnts y voiednessY esene of ssimiltion y ple nd mnner of rtiultionY viisons @FrFX liisonA nd linking @FrFX enhînementA"two phenomen whih mke sure prenh speeh remins onneted whenD sed on the lexil hoie loneD it would not e the seY gertin ptterns in entution nd rhythmisF Phoneme inventory of the French language here re (fteen vowels in prenh vonVRD glVWD eleven orl nd four nsl onesX IF /i/X 4qui4D FrF 4who4D GkiG"is phoneme euse it n e ontrstedD for exmpleD with 4que4D FrF 4tht4D GkoeGX lose front unrounded vowelY PF /e/ 2 X 4dé4D FrF 4die4D GdeG"ontrsting with 4dais4D FrF 4nopy4D GdEGX loseEmid front unrounded vowelY QF /E/ 3 X 4fait4D FrF 4done4D GfEG"ontrsting with 4fée4D FrF 4firy4D GfeGX openEmid front unrounded vowelY 2 There is a tendency to neutralize the dierence between /e/ and /E/.

3 /E:/, as in "fête", Fr. "holiday", /fE:t/, is usually replaced by /E/. However, there are rare pairs where some speakers still make a distinction in the vowel length: "mettre", Fr. "put", /mEtK/, vs. "maître", Fr. IPF /Ã/X 4emmener4D FrF 4ring4D GÃm@neG"di'erent from the word with n sent GÃGX 4mener4D FrF 4think4D Gm@neGX nsl open k unrounded vowelY IQF /Õ/X 4mont4D FrF 4mount4D GmÕG"ontrsting with 4mot4D FrF 4word4D GmoGX nsl openEmid k rounded vowelY IRF / oe/ 5 X 4un4D FrF 4MASC 4D G oeG"ontrsting with 4an4D FrF 4yer4D GãGX nsl openEmid front rounded vowelY ISF /Ẽ/X 4fin4D FrF 4end4D GfẼG"ontrsting with 4fine4D FrF 4(ne4D GfinGX nsl openEmid front unrounded vowelF wenwhileD G@GD while eing present in the lngugeD is not phonemeF sf it were oneD dding or dropping it in speeh would rete new wordsD ut it does not ring out semnti hnges 6 X hemin GSmẼ ! S@mẼG @FrF 4wy4AD lentement GlÃtmà ! lÃt@mÃG @FrF 4slowly4AD nd so on nd so forthF por this resonD G@G is stylisti vrint of the phoneme GoeG rther thn phoneme on its own endVPD rVUD nd its omission in words is elisionF o summrize the footnotesD there is @t times regionlA tendeny towrds neutrliztion in pirs GaG vsF GAGD GẼG vsF G oeGD GeG vsF GEGD GoG vsF GOGD nd GøG vsF GoeG to the point when they n e indeterminle putHTF here re twenty onsonnts in the prenh lnguge vonVRD glVWD ll of whih re pulE moni egressive soundsF welve of them re ostruents @six plosivesD six fritivesD ll tegorized into pirs of voied nd n unvoied onstituentAD nd eight of them re sonornts @three nslsD two liquidsD nd three semivowelsAX 4 /A/ is often replaced by /a/, though /A/ is preferred, for example, before /z/ or after /Kw/ [Tra87]. PT IWF /4/X 4huile4D FrF 4oil4D G4ilGX the entrl orl lioEpltl voied pproximnt"semivowel glideD orresponding to the lose vowel GyGY PHF /j/X 4yeux4D FrF 4eyes4D GjøGX the entrl orl pltl voied pproximnt"semivowel glideD orresponding to the lose vowel GiGF ome dilets of prenh lso feture GNGF his sound lso ours in some loned words suh s 4prking4 nd n e repled y GNgG or GñG qqqqIIF he ousti properties of prenh n e found in the work y vonVRF Phoneme duration and timing in the French language he length of speeh sound depends on its neighoring phonemesD on the position within the utterneD nd on the syntti nd semnti struture of the utterne glVWF vong consonants n e used for emphti stress or in gemintionF hen the utterne involves oEourrene of onsonntD the (rst onsonnt instne usully loses its third phse of prodution @the urstAD nd the seond one loses its (rst phse of prodution @the thAF hen there is no puse etween the onsonntsD ut they elong to two sylllesF he onsonnt from the syllle od9s intensity is ttenutingD while on the syllle onset the intensity is risingF sn gemintion of voieless plosivesD GpGD GtGD nd GkGD the hold phse is voielessF he puse efore the urst is longer thn in nonEgeminted plosiveD nd the rtiultors re tightly loked during this puseF o produe geminted voied plosivesD GbGD GdGD nd GgGD the vol folds strt virting erlierX the losure is ompletely voiedF rene the puse efore the urst is not entirely voieless"there is n insigni(nt glideF pritivesD liquidsD nd nsls"GfG nd GvGD GsG nd GzGD GSG nd GZGD GlGD GmGD GnGD nd GKG"re not silent even during the hold phseF sn prenh vowel ontrstsD there is evidene tht durtion is not relile ueD nd though in prodution it is normlly held s per the rules of prenh phonologyD the di'erene etween vowel durtions is not sensed in pereption qfVVF Prosody and Voice Stream Segmentation he miniml rtiultory unit is the syllableF yllles re formed y vowels whih n e ompnied y onsonnts efore them @in the syllable onset A nd fter them @in the coda AF sf syllle ends with vowelD it is open @eFgF 4répéter4D Fr. 4repet4D GKe.pe.teGAY otherwise it is losed @eFgF 4teur4D Fr. 4tor4D Gak.toe:KGAF fy the generl prinipleD open vowels re used in losed sylllesD nd lose vowels re used in open syllles hPID nd more preiselyD syllle segmenttion is done ording to rules tht di'erentite etween the numer of onsonnts to e distriuted etween vowelsD their lssesD nd position within the wordF he units of utterne segmenttion reX Rhythmic groups X groups of words"tullyD given the prenh onnetedness of speehD groups of sylllesD where syllles n overlp word oundries nd the numer of syllE les oinides with the numer of pronouned vowels"hving some sense s whole nd stressed on the lst syllle qrSHF hythmi groups re seprted from eh other y hnges in speeh melodyD rhythmD nd durtion of the stressed vowelY Syntagms X groups of rhythmi groupsD giving wider view on the units expressed vi rhythmi groupsF yntgm oundries re more free to e estlished y the speker PU thn the ones of rhythmi groupsD nd they re mrked y optionl puses nd hnges in speeh rte nd melodyY Breath groups long with intonational units X even lrger units of rhythmi orgniztionD more or less oiniding with sentene oundriesF freth groups re seprted from eh other y puses tht re used for strting new reth yleF es mentioned oveD from the rhythmi point of viewD prenh utternes re very ontinuousD whih is supported y two sndhi phenomenD liaison nd enchaînementD tht our in word sequenes tht re losely linked y sense nd eliminte oundries etween words in fvor of merging them into sequenes of sylllesF inhînement regroups the phonemes in n utterne into syllles in suh wy tht the lst pronouned onsonnt of one word is tthed to the initil vowel of the next wordF his phenomenon does not 'et the qulity of the involved soundsD iFeF normllyD there is no ssimiltionF viison is divided into vocalic nd consonantal liaison @the ltter usully simply ering the nme 4liison4AF oli liison hppens when two similr vowels oinide in the )ow of speeh within syntgmD resulting in one long vowel with minor hnge of tone nd intensityD nd when there re two di'erent vowels ourring one y oneD resulting in very fst trnsition from one to nother with temporl overlp etween the (rst phse of the seond vowel prodution nd the third phse of the (rst oneF he exeption to voli liison my e nsl vowelsF gonsonntl liison ours t word oundries tooF prenh lnguge hs gret disrepny etween its written nd spoken formsX most onsonnt lusters t the end of the word re not pronounedF roweverD liison n preserve the speeh )owD mking the previously mute onsonnt link the words y mens of new syllle mde from the onsonnt nd the initil vowel of the next wordF viison n e oligtoryD optionlD or impossile depending on the ontext nd pronunition styleF es study y phHR showsD produtivity of liison nd enhînement in prenh is reltively the sme nd keeps under the level of T ourrenes per IHH wordsF Prosody is n essentil forml feture of sentene tht llows the listener to single it out from the voie stremD divide it into smller semntiD rhythmiD nd melodi segmentsF rosodi ues orgnize the set of words in sentene into the wholeD mke the syntti reltions etween its prts lerY it is them wht is responsile for the expressiveness nd most delite dismigution in speehF rosodi ues re the sme in most lngugesX stressD speeh melodyD voie pithD pusesD registerD nd speeh rteF Stress is used to orgnize the utterne into segmentle units nd to highlight its logil enterF tressed syllles re distinguished from unstressed ones y the voie intensity @dynamic stressD ontrolled y the tension of rtiultion nd mount of the exhled irAD the pith @tonic accent AD nd durtion of the vowel @quantitative stress AF he toni ent is domintive in prenh long with the quntittive stressY voie intensity does not vry muh from stressed syllles to unstressed onesF hepending on the speeh segment in questionD the notion of stress n pply to wordsD phrsesD syntgmsD nd utternes s wholeF PV Word stress is ound in prenhX the ent lwys flls on the end of the wordF Phrasal stress strips most words in the phrse of their stressD leving rhythmicD or normal stress @Fr. "accent d'intensité"AD nd emphatic stress @Fr. "accent d'insistance"AF xorml stress de(nes the voie intensity pttern for the utterne when it is sid with neutrl emoE tionF st is mrked y pithD intensityD nd vowel durtion nd flls on the lst syllle of eh rhythmi groupD eing ttended y seondry stress tht fllsD grdully fdingD on every other syllle from the end of the rhythmi groupF es for emphti stressD it imprts the emotion ehind the words of the speker @emotive stressD usully on emotionl words suh s Fr. "misérable", "admirable". . . "mking the initil onsonntD the (rst onsonntD or the onsonnt from liison longA or highlights her line of thought suh s in mking de(nitionsD orretionsD @didacticD or intellective stress "mking the initil vowel long nd rtiulted tenserD often preeded y glottl stop GPGD orD in se of the initil onsonntD doing the sme for the (rst vowel nd mking the initil onsonnt e rtiulted tenserAF imphti stress does not hve to e present in the utterne nd nnot reple the regulr stress pouSWD vvUHF henD syntgms lso in)uene phrsl stressX it is the lst rhythmi group tht gets ented mostD nd ll groups efore it re mrked y stress less nd less s we move from the end of the syntgmF he greter the speeh rte nd the longer the syntgmsD the less the rhythmi stress instnes re pronounedF pinllyD there is rnking within the utterneD sed on the logi of orgniztion of synE tgmsX the speker highlights the syntgm tht rries the entrl mening in the utterneF Speech melody is the min feture to estlish to ommunitive type of the utterne" delrtiveD interrogtiveD impertiveD nd exlmtory sentenes nd ertin disourse elements suh s dethed ppositionsD itemizingD mrking n utterne tht ws ut shortD sking for ressurne of the interloutorD etF @whih is espeilly importnt in prenhD sine delrtiveD interrogtiveD nd impertive sentenes n e uilt with extly sme sequenes of wordsY howeverD sine prenh is not tone lngugeD menings do not depend on voie pithA"nd to identify syntgms nd their reltion to eh otherF Voice timbre rings in the emphti informtion on the utterne nd depends on the dditionl tones nd overtones inherent in prtiulr spekerF Timing ontrol ndD in prtiulrD pauses dd to the utterne segmenttion s de(ned y the stress nd melodyF he puse serves s ue on how relted the syntgms re ndD dditionllyD is mens of emphsisF emporl vritions within phonemes nd phrses re relted to speech rate X it n inreseD for exmpleD to let the listener identify suordinte luseD or dereseF tpHH modeled prosodi fetures of the prenh lnguge in utternes of vrious ommuniE tive types sed on four spekersF st is lso rgued y pouHI tht the lose reltion etween the segmentl nd suprsegmentl fetures in speehD rtiultion nd prosodyD highlights the neessity to move forwrds to their joint nlysisF 2.1.3

Adjusting the plan: self-correction o spekD humns rely on the lues from motor tivity of their vol pprtus nd on the gp etween the desired ousti result nd the tul oneF wotor feedk is proessed mostly unonsiouslyX the reeptors of muslesD tendonsD nd muous memrne report on their ondiE tion to the rin nd spinl ordD in)uening new neurl ommnds for the musles of speehD PW urging them for ompenstory movementF gonverselyD the speker is more onsious of uditory feedkD nd it is more di0ult to ompenste for its loss thn for prolem with the motor feedk whih n hppen in se of diseseD disorder or surgeryF snterfering with the ousti signl tht rehes our ers leds to extreme speeh degrdtionX prosody gets )tD nd speeh eomes mistimed nd inrtiulte emIHD whih n e oservedD for exmpleD in hildren who lost their hering t very young geF iven studious triningD suh s the one for simultneous interprettionD does not eliminte these e'ets fullyF vet us see now how the prolem of speeh synthesis is pprohed in the stte of the rt nd wht were the ides tht rought tht outF he omponents of presentEdy textEtoEspeeh synthesis systems n ommonly e divided into two ig prtsF et the front endD the system nlyzes nd proesses the input text @tokenizes itD prses it nd performs its struturl nlysisD dismigutes homogrphsD normlizes it ording to the intended use of the systemA"produing wht we ll linguisti spei(tion"nd onverts it into sequene of phonemes to produeD lso mrking stressF he result of this step is set of linguisti feturesF et the k endD the speeh wveform is genertedF roweverD the presene of two suh lerly identi(le loks is not oligtory @endEtoEend speeh synthesisAF he speeh synthesis reserh hs ome long wy from knowledgeE nd ruleEsed tehE niques tht exploited phonemeEspei( ousti prmeters to two mjor fmilies of pprohesX ontentive speeh synthesis tht is sed on dt smples nd prmetri speeh synthesis tht works only t the level of one or nother kind of prmeter evolutionD progrmmtilly generting the sound of speehF QH Concatenative approaches gontentive speeh synthesis ppered in the IWVHs nd the IWWHsD seeing the synthesized speeh s ontention of proessed ousti units tken from preEpreorded speeh dtseF hese units generlly re diphonesD whih re the signl from the seond hlf of one phone to the (rst hlf of the otherF his wyD overing the trnsition time in single hunk llows for pturing ortiultory phenomenX the e'et of the phone on its neighorF es for the utting pointD the phone enterD from the ousti perspetive it is the most stle region nd therefore is most suitle for ontenting with segment from nother reordingF o over ll phoneme pirsD let lone ll their positions in the rhythmi phrse nd other ftors suh s intontion ptterns nd durtionD quite sustntil dtse is neessryF sf it is not su0iently lrgeD the units n e redued to hlfEphonesF es the dtse growsD the units n e mde lrger nd lrgerX phone lustersD syllles nd entire wordsF hen the system needs to del with nrrow domin with little lexil ndGor syntti vritionD for exmpleD trin sttion nnounementsD the units n grow up to segments nd phrses @limited domin synthesisAF Diphone synthesis yne populr ontentive pproh is diphone synthesisF ithin itD the system dtse ontins single instne of eh prtiulr unit ville for ontention @in some implemenE ttionsD though rrelyD fewD to ount for di'erent pith vlues nd speking rtesAF hen the wveforms of the instnes involved in the phrse to e synthesized re ontented t points where the smple wveform hs the sme vlue to ensure ontinuityF hen the wveform n e smoothedF he size of suitle dtse depends on the lngugeF hile theoretilly one would need the numer of ll uniphones squredD not ll omintions tully ourD thus typilly requiring etween VHH nd PHHH diphones @for exmpleD uifHV found tht stlin used VSI diphones nd inglish IUTQD while the numer of uniphones ws TT nd QWD respetivelyY in olishD there re QU uniphones nd IHHV diphones fIHAF sn omprison to more dvned ontentive tehniquesD suh dtse is of modest sizeF xeverthelessD prepring it n still e hllengingF sn the se of prenhD h + WT indites it took them one month to reord orpus with properly monotonous intontionD segment itD equlize the energy levels t the eginning nd the end of segments nd normlize pithF gontenting nd proessing suh smples does not require ig omputtionl powerD so this method n e the solution for moile deviesF peeh synthesized y this method generlly is highly intelligile ut not very nturl due to its sene of vrition in pith nd speeh rteF o solve itD s mentioned oveD one n reord more smplesD onditioning diphones on di'erent pithes nd speedsY howeverD this inreses the dtseD thus removing the primry dvntge of the low footprint of this methodF enother solution would e to mnipulte the ontented smples with timeEsling nd pith shiftingD whih needs to e done refully for multiple resonsX (rstD sine it is the rtiultors wht humns move fster or slowerD not the vol foldsD the signl needs to e seprted into soure nd resontor frequenies efore ny speed djustmentY seondD voieless onsonnts do not hve muh or ny voie ontriution nd therefore need to e proteted from time nd pith shifting to remin nturlF o mnipulte the seleted smples nd ensure smooth trnsition etween themD one uses speeh signl proessing lgorithmsF here re three lsses of their representtions of speeh QI frHUX timeEdomin lgorithms suh s the pithEsynh overlpEdd method @hEyve wgWHAD liner predition oding @vgA nd frequeny dominEsed lgorithmsF sn hEyveD proly most ommonly implemented method of the threeD (rst we identify pith periods nd mrk some identi(le glottl event in eh period in the wveform @for exmE pleD glottl losure"the point of mximum exittionAF henD during runtimeD eh segment is windowed with rnning window round the pithmrkF hen the extrted windowed signls n e shortened to inrese pith nd pdded with zero mplitudes to lower pithY durtion n e djusted y repeting nd omitting frmesF hen the frmes re simply dded one fter notherF es this method does not tke re to ensure the homogeneity of the spetrl shpes or vluesD this imposes n importnt limittion tht one n perform only smll wveform modi(E tionsD using only wellEdpted for eh other unitsF ytherwise the result will ontin udile glithesF Unit selection he mjor ontentive method is unit seletion rfWTF nlike diphone synthesisD it opertes with lrge dtseD where there re multiple opies of diphones s well s lrger units @diphthongs nd phoneme lusters with omplex trnsitionAD eh with its own intontionD speedD intensityD position in the speeh segment nd the prt of speeh of the word or words it elongs toF hen synthesizing with the units from suh dtseD every diphone is to e piked mong numerous optionsF o mke the hoieD we use trget spei(tionX efore generting the wveform from seletion of units from the dtseD the system predits the required prosody @expliitlyD with modelD or impliitlyD through identifying segments typilly exhiiting signi(nt prosodi e'etsX phrse oundriesD puntutionD etFAD speeh rhythm nd intensity for the input textF purthermoreD fter the input text pssed the front endD we lso hve the phoneti ontext nd its phonologil lssesD prts of speehD phrse nd sentene type nd sizeF his wy one n pik sequene of units mthing those supplementry ftors s losely s possileF st must e notedD howeverD tht the onept of lose mth is not strightforwrdF here re multiple ftors to weigh inX fetween omposing sequene from severl shorter units or longer oneD the longer one will proly end up eing more nturlF uh longer unit n eqully e introdued into the system s sequene of onseutive shorter unitsF purthermoreD the results will e more nturl if one hs sequene of pssly suitle units mounting to the vlue of trget ost funtionD syD XD rther thn sequene where ll units ut one re perfetD nd ll the ost X is onentrted in tht single unitD resulting in notiele distortionF enother point to onsider is tht even if eh of the ndidte units is resonly dpted to the trget spei(tion on its ownD together they will not produe nturlly sounding result unless the neighoring units re su0iently similr in terms of oustisD prosodyD voie soure nd other ttriutesD mening tht they together hve to hve low join ostF his is why the weighting the trget nd join omponents of this deision tree is typilly (neEtuned ording to the nturlness resultsF QP henD to msk the trnsition etween segments nd estlish ommon pithD intensity nd speking rte ptternD speeh proessing lgorithms suh s hEyve re pplied s explined in the prt on diphone synthesis oveF yverllD unit seletion synthesis nlysis proves to e highly nturl thnks to the diret reEuse of reordings of rel spekerF wnipultion with its pith nd speking rteD howeverD do not provide su0ient degree of ontrol over the ontinuity of the signlD thus rendering the syntheti speeh less intelligileF roweverD this nturlnessEintelligiility pyo' is still found prtil in rel pplitionsF enother point to onsider is the size nd the omputtionl lod of the systemX the unit dtse size nd the lgorithms of seletion nd proessingD oth of whih n exeed the ville hrdwre pityF o improve themD one n uild strtegy to prune the dtse or t lest the unit serh spe ndGor reple some omponents of the system @the front endD the model of prosodyD intensity nd durtionD the lgorithm of unit seletionA y sttistil modelsF sn the ltter seD the system is lled hyridF imploying sttistis nd mhine lerning does not hve to e restrited to ertin setions of speeh synthesis systemF sn ftD this is nother highly produtive venue of speeh synthesis reserhD whih rings us to the following setionF Parametric approaches sn prmetri speeh synthesisD ousti prmeters @spetrl envelopeD informtion out the soureD usully the fundmentl frequenyD nd noiseElike omponents for ostruentsA re modE eled s time seriesD stohstilly generted s suh from the linguisti spei(tionD nd then used in speeh prodution model" vooder"to otin wveform of speehF his pproh hs instigted very produtive re in the (eld of speeh synthesisF ntil reentlyD the stte of the rt ws typilly hieved when generting ousti prmeters with hidden wrkov models @rwwsAD trined y fore lignment from n nnotted speeh orpus x + IQY s the (eldD like mny othersD underwent the deep lerning revolutionD neurl speeh synthesis hs rehed the stte of the rtF rww synthesis lone is unle to provide the ontinuity of the )ow of speehX the wrkov ssumptionD inorported in itD rings in tendeny to entrlize speehX the spetrl vetors re ll emitted lose to the vergeF o solve this nd dd the neessry dynmi fetures of speehD the system hs to inorporte not only the stti oe0ientsD ut lso the di'erenes nd seondEorder di'erenes etween themF henD in ontrst to utomti speeh reognition where triphone models seem to provide lredy enough ontextD synthesis requires oth phoneti nd prosodi ftors" longerEterm ontext or ess to the whole phrse or even the senteneF en rising ritil prolem of the dt sprsity is solved y prmeter tying nd onstruting spekerEspei( dpttions for generi speeh synthesis systemsF

2.2.3

Multimodal speech synthesis here hs lso een produtive diretion of reserh iming to dd other informtion to speeh synthesis outputF his n del with how the generted speeh will e deliveredD for exmpleD tlking heds mening to nimte n vtr so tht it produes nturl output"eFgF fvhyY orD side from the udiovisul spetD fousing lso on the expressivity suh s hgqyIWF QQ yrD lterntivelyD it my onern with going deeper into how speeh is produedX how it is premeditted @rinEmhine interfes nd neurl speeh qf + HWD qfIID wgIWA ndGor how exeuted @frq + IT for the link with rtiultionAF sf we onentrte on rtiultionD we enter the domin of rtiultory speeh synthesisX method of synthesizing speeh y mnging the vol trt shpe on the level of the speeh orgnsF he vol trt n e modeled with geometri ÖhmTTD ftuHTD toIQD iomehnil vpIPD erw + IS nd sttistil weWHD rwII modelsF he dvntge of sttistil models is tht they use few prmetersD speeding up the omputtion timeF heir disdvntge is tht they follow the dt priori without ny guidne nd do not hve ess to the knowledge of wht is relisti or physilly possileF feuse of thisD to produe orret on(gurtionsD they need to e (nely tunedF yne ruil notion to synthesizing the movement of the vol trt is the smllest unit or some other elementry omponent of rtiulted speehF prom the uditory perspetiveD it is the phonemeX the smllest semntilly distinguishing unitF wenwhileD from the rtiultory point of viewD eh nd every sound we utter is result of the soureD formed y the vol foldsD nd the (lterD formed y the vol trtF he shpe of the vol trt is ompound e'et of oordinted rtiultory movementsF qiven tht eh rtiultor seemingly follows its own timing nd hs onsiderle degrees of freedom in speD it is esy to see how the question of orgniztion of speeh movements is no trivil tskF e mjor rnh of modeling these motions onsists in deomposing speeh into rtiulE tory gestures @rtiultory phonology fqWPD tsk dynmis qfVVD xwrt + IPD motor primitives wsqqWWD xITAF roweverD when deling with overlpping gesturesD we fe the disdvntge thtD t lest s of nowD the ground truth is not villeY every musle ontrtion nd every end t joint n e integrted into vrious gesture elementsD nd it is up to the model only to sty onsistent out determining where the oundries of those gestures reF prom this stndpointD it is interesting to onsider n lterntive view on wht guides speeh phenomenX trgetsF hey n e found in virtul tsk spe @tsk dynmis uVUD wVWY wfIVAD or they n literlly e spei( positions of the vol trt vxIUF st is this line of thought tht the present work followsD hosen due to the ene(ts of the methodologil lrity nd high ppliilityD eFgF in rtiultory speeh synthesis vinWID + ITD ivIUD whih in turn will e ple of serving s further evidene for the underpinnings of speeh prodution erIUD ingHHD + IVD for exmpleD for purely theoretil questions of timing nd rtiultion ISD IRD wr + IU to more psyholinguistilly oriented studies suh s rfIWF vxIU employed trgets s the vol trt on(gurtions ttined t the middle of the durtion of eh phonemeD whih is when it is most stleF roweverD tht study reognized the need to llow for ontextully modi(ed trgets to pture ortiultionF firIQ worked in tht diretion nd di'erentited trgets ording to their voli ontextD following the previous ides of ÖhmTUF he following susetions re going to e dedited to the spei(s of wht goes into treting rtiultion in speeh synthesisF Articulatory data peeh is suh dynmi proess involving so mny di'erent strutures in the humn ody tht there is gret need of dynmi nd preise dt pturing tehniquesD preferly without QR ny hrm to the sujet nd tmpering with the proess of nturl speeh produtionF Aerodynamic measurements were one of the erliest methodsY their im ws to nlyse the pressure of the ir )ow when the sujet is spekingF Electromyography is musle tivity reording tehniqueF st n ollet responses from rnge of speeh orgns exept for those tht re inessileD usully y mens of hookedEwire or surfe eletrodesF rookedEwire ones re inserted into the ody of the musleD using minor disomfort for the sujet nd possily 'eting the wy they spekF urfe eletrodes re nonEinvsive nd esier to pply rrWWF

Photography n e used to pture rtiultory dt for visile or prtilly visile rtiuE ltorsF Radiography X X-raysD X-ray microbeamD cineradiographyD computed tomography @gAF ell of them n use Ery to pture the on(gurtion of the vol trtF he soft tissues pper greyD nd even if g mnges to pture them more lerlyD the edges of the tongue shpes re not shrp enoughF roweverD the rdition exposure mkes these methods unsfe for the sujet frHUF Magnetic resonance imaging @wsAX uses mgneti (eld nd rdio wves to imge setion of tissueF he threeEdimensionl spe is ompressed into two dimensionsD whih my eome soure of error for smll ojets tht will e treted s if they were in the sme plneF por instneD the epiglottis my e ondensed into one single slieD resulting in lurry edges nd misinterprettion of its size nd shpeF purthermoreD ws is not le to pture solid tissuesD nd suh rtiultors s the teeth re invisileF tust like in omputed tomogrphyD the sujet usully hs to ssume the supine positionD whih 'ets the on(gurtion of the rtiultors nd the dynmis of speeh"though vertil ws mhines re villeD tooF ws n e used to pture dynmi speeh"the usul pproh is to repet the sme utterne over nd over gin nd then join the sttered imges tken t di'erent times into whole utterneD or use pst pin iho for imges of poorer qulityD ut etter imging rte @R!PR ptures per minuteAF sn omprison to gD there is no ionizing rditionD ut nevertheless longEterm iologil nd linil sfety for the sujet9s helth remins to e proven uuwIQF ith the dvne of tehnologyD relEtime ws @EwsA is eing dopted more nd more @rw + HTD ef + HWD + IHD xfq + IID xu + IQD ivyITD xITD + IUD + IVD vv + IWAF yne mjor question rised with the use of ny kind of ws is its segmenttionD iFeF deterE mining the ontours of the rtiultorsF his hs een ddressed with geometrilD sttistil nd deep lerning models @x + IRD xITD + ITD + IUD xIUAF Palatography X n erly tehnique to study tongue plement ross the plte nd teethY it hs developed into electropalatography nd is now le to mke ptures in onverstionl speeh rther thn in short sequenesF st is simple tehniqueX esy to operte nd reltively nonEinvsive qxWWF st ws ontinued to improve over the yers with vrition in mterilsD the numer of sensors nd hips fISD fITF e rnge of point tracking techniques is ville tooF heir dvntge is fst smpling rte nd seletivityX it is possile to trk extly tht point in the tissue tht is of interestF roweverD it is not possile to pply enough trkers to otin the whole piture without hindering the speehD while imging tehniques n provide true multiEdimensionl dtF o summriseD there still is no ompletely sfe nd informtive method for olleting dynmi rtiultory dtF iither the method ptures very prtiulr ehvior of the sujetD or the dt re omprehensive nd of high qulity ut ome only in smll mounts nd t muh lower QS frequenies thn the frequeny of 100 rz tht is deemed to e neessry to pture speeh phenomenF Vocal tract modeling and synthesizing speech oD speeh synthesis ugmented with rtiultory informtion tries to stik to the nturl speeh mehnism s lose s possileF xeverthelessD there is vriility in how lose tht isF fiomeE hnil pproh solves the xvierEtokes equtions to estimte deformtion of ll @prominentA vol trt musles nd orgns nd estimte the orresponding eroEousti phenomenF st hs ene(t of full ontrol over the rtiultorsD ut su'ers from hevy omputtionl lod nd lk of required ntomil nd physiologil dt vpIPD erw + ISF wenwhileD rtiultory speeh synthesis is sed on using simpli(ed rtiultory model to imitte humn speehD ontrolling the rtiultors in fesile wyF st is le to replite the ntomil nd physiologil phenomen without delving deep into the strutures tht re involved"it onerns only temporl evolution of the vol trt geometryD whih turns out to e enough to synthesize speeh of mple qulityF o llevite the omputtionl lod nd mke the known theoretil knowledge pplile in this seD the diret numeril simultion of speeh phenomen is still limited to individul ses tht often re overly simpli(edD nd even in suh ses the omplexity is so high tht the omputtions tke dozens of hours weWHY it n tke dys to ompute the )ow dynmis t the vol folds trde + IQF oD nother pproh is to develop models only for the most essentil physil phenomenF g + WT nd i + II provide theoretil foundtions tht llow us to de(ne wht pheE nomen re most importnt in phontionY they re supported y experiments of r + HU nd h + HIF he onstruted lrynx models re lolly inompressile @with low relmholtz numerAD qusiEsttionry @with low trouhl numerA nd with n insigni(nt visosity @with n intermedite eynolds numerAF st is not neessry to speify ll detils out the )ow motion to produe turuleneEindued sound rwHSD urHSD xIWF he vol trt on(gurtionD from the glottis to the lips openingD n e enoded y mens of re funtions tht pproximte the vol trt y ousti tues of vrying sizeF he re funtion n e estimted without ny knowledge of seprte rtiultors pnUIF toIQ genE ertes re funtions s trnsition from one voli re funtion to notherD with onsonnts superimposed s onstritions ttined during this vowelEtoEvowel trnsitionF his pprohD roughly sed on Öhmn9s ÖhmTT vowel sustrtes nd onsonntl perturtionD n e used to synthesize some (xed phrses ut is di0ult to generlize for use in textEtoEspeeh @A systemsF wfIV pproximtes re funtions s oneEdimensionl model of the vol trt sed on just six pointsF enother pproh would e to onstrut the sgittl setion of the vol trt @the twoE dimensionl seD suh s in the work y werUQA or the vol trt proper @the threeEdimensionl seD suh s in the work y ftHQA s the re ounded y unh of primitivesF foth of the mentioned studies do not (t the form of the vol trt too preiselyF snsted of geometril primitivesD the vol trt n e shped out from medil imges y mens of n rtiultory modelF wny rtiultory models hve ppered in reserhD rnging from s simple s threeE prmeter ones pnTH to s sophistited s hundredsEprmeter ones qHTF yne of the most fmous models is the one y weWHD omprising model for the lipsD model for QT the tongueD nd model for the lrynxF ell three rtiultors9 on(gurtions re deorrelted from the in)uene of the lower jw position nd enoded y mens of the prinipl omponent nlysis into vetors of vrying lengthsF o keep trk of the enodingD one stores the vertil opening nd protrusion of the jwF weWH indites tht three tongue prmeters n deE srie WT7 of the vrine in the test imgesD nd ll three rtiultors re enoded in seven prmeters in totl @see pigure PFWAF pigure PFWX he seven pE rmeters of wed9s rtiuE ltory modelX P7 is the larynx heightY P2D P3D nd P4 enode the tongue "its odyD shpeD nd tip respeE tivelyY P1 stores the jaw openingY P6 nd P5 de(ne the position of the lips " their protrusion nd opening respetivelyF pirst pplied to vowels onlyD this model ws followed y modE els to ount for onsonnts too"in two dimensions vfIID vgIRD wpIW s well s in three dimensions ff + HPF @yE viouslyD the twoEdimensionl models hve to e ompnied y some kind of sptil estimtion t further stge in speeh synE thesisY sineD ginD fullEsle Qh lultions tke exessively long omputtion time"often hours to lulte IHEmsElong speeh signl"the more fesile lterntive ppers in limiting the numer of dimensionsFA es for nother rtiultor tht ws left untreted y wedD the velumD there re not mny models for itF fHV mde threeEdimensionl model of the velum sed on stti Qh ws nd g imgesD whih rises the question of eing tully le to pture the high vriility in the dynmi proess of speehF viIS propose geEsed velum model uilt on n Ery (lm of IS short prenh sentenesY this model is le to pture UH7 of the totl vrine y mens of two omponentsF hen the shpe of the vol trt is estlishedD the vol trt hs to e divided into nrrow tues tht re 4strung4 on the vol trt9s entrl line wvIQ whih n e omputed y vrious lgorithmsF he res A @in m 2 A n e omputed from the heights @dD in mA of the estimted tues nd two spekerE spei( prmeters tht re to e determined empirillyX A = αd β @PFQA o ontrol the dynmis of the modelled vol trtD one n use the notions of gestures @motions of formtion of prtiulr onstrition over timeA nd targets @the vol trt onE (gurtions tht the speker ims to rehAF ixmples of studies tht im for simulting the underlying mehnisms of speeh prodution involve the tsk dynmi model y wVW @E ompnied y the work on how to operte itX xw + IPAD the gestureEsed dynmi model y fqWP nd gesturl dominne model y ftuHTD nd the trgetsEsed model y firHUF purthermoreD the dynmi hrteristis of the system n e enoded together with the prE metri representtion of the vol trtD suh s in work of rIWF QU Introduction yne of the prinipl notions tht one hs to fe when deling with speeh synthesis nd proessing is ortiultionX the in)uene of neighoring sounds on eh otherD stemming from the ft tht we nnot simply ppend one shpe of the vol trt fter the other nd lwys hve to try to (nd the most e0ient rtiultory trjetory tht would still leve our messge omprehensileF por exmpleD when we pronoune oth GbaG nd GbyGD wht is de(nitive is tht the lips need to lose to produe the stop GbGF roweverD there is no restrition on where they need to do thtF oD nturl trnsition is to give the lips the shpe tht is required for the oming vowelX to protrude them for GyG nd not for GaGF his wyD the moment the urst of GbG is relesedD prodution of the vowel strtsF his in)uene is most pronouned in the diretion of ntiiptionX when the sounds plnned in the ner future quire some fetures of those tht urrently re in produtionF he other diretion of ortiultionD rryoverD where the pst sounds hve lingering e'ets on the present onesD is less prominent nd is mostly ttriuted to pssive inerti uxWWF he gretest impt is typilly rought y vowelsD whih eome syllle nuleiF his is espeilly true in the se of prenhD where vowels need to e rtiulted quite preisely ndD unless it is G@GD nnot e reduedD unlike some other lnguges like ussinF his lrgely outlined the pproh of this studyX to produe speeh nd rtiultory moveE ments from phoneti trnsriptionD relying on set of si vol trt on(gurtions nd fousing on the wy how vowels in)uene onsonnts efore themF e were interested in exploring the potentil in using quite littleD nd yet su0ientD stti mgneti resonne imging @wsA dt nd implementing one of the few existing ttempts t reting fullE)edged rtiultory speeh synthesizer with omprehensive ontrol over the rtiultorsD tht would e ple of reproduing the vst diversity of speeh soundsF his led us to deling with these uildingElok vol trt on(gurtions y following the steps of firIQD pplying tht methodology to prenh nd extending the sope to over prtilly the entire set of prenh phonologyF QW Chapter 3. Articulatory speech synthesis from static MRI data

3.2

Objectives he ojetives of this study were s followsX o represent the ville ws dt with n existing rtiultory model @joint work with ves vprieAY o extrpolte this limited dtset so s to estimte the missing smplesD generlly folE lowing the pproh of firIQ @rried out y myselfAY o develop set of rules de(ning howD given n utterne to produeD to pik neessry smples from the extended lirry of si rtiultory on(gurtionsD how to djust them to set up n e0ient trnsition oth t the level of eh of the rtiultors nd the entirety of the vol trtD nd how to trnsition etween them in set time while lso mnging the prmeters of the soure nd pressure in the suglottl nd suprglottl vities @rried out y myself"the min ontriution of this prtAY o postEproess the otined vol trt on(gurtions to mke sure tht they follow the phoneti rules relevnt to the prenh lnguge @rried out y myselfAY o proess those on(gurtions with n existing ousti simultion unit @joint work with fenjmin ilie nd ves vprieAY o nlyze nd evlute the resulting utternes @joint work with ves vprie nd fenE jmin ilieAF

3.3

Building an articulatory speech synthesis system es set up y the ojetivesD the system is mde up of three mjor omponentsX the dtse with the uilding loks4 for rtiulting utternesD the joint ontrol lgorithm for the vol trt nd the glottl soureD nd ousti simultionF he primry onern of this work re the (rst two omponentsF 3.3.1 Dataset he dt were sttiEws suset"the suset tht ws ville t the time of the study"of the ertpeehwsfr dtset hpp + IW reorded t xny gentrl egionl niversity rosE pitlD prneD under the pproved medil protool wiryhy @glinilrilsFgov sdentiE (erX xgHPVVUHSQAF he snner in use ws qenerl iletri ign rhxt Q @qi helthreD ghigoD sllinoisD nited ttesAF e used Qh pqi @ = 3.12 msD i a 1.084D py = 26×26 mD )ip ngle = 10 degreesA for the quisitionF n slie thikness is 2 mmD sping etween slies is 1 mm nd pixel ndwidth is 488 rzGpixelF eelertion ftor is 2F he imge resE olution is 256 × 256 with 76 sliesF hurtion of one quisition is 12.7 seondsF he sujet @sujet AD susequently referred to s S A A is mleD 35 yers oldD 182 m tll nd 74 kgF yverll it mde for 97 imgesF ixmples of midEsgittl uts of these dt re shown in (gures QFI nd QFPF RH pigure QFPX widEsgittl slie of the stti Qh imges of sujet S A for some of the prenh onsonntsF gonsonnt ws pronouned in ontext of the following vowelF s seleted the midEsgittl slie in those imgesF hese dt ptured rtiultion without phontionX the speker ws instruted to show the position tht he would hve to ttin to produe prtiulr soundF por vowelsD tht is the position when the vowel would e t its lerest if the sujet were phontingF por onsonntEvowel @gA sylllesD tht is the loked on(gurtion of the vol trtD s if the sujet were out to strt pronouning itF he ssumption is tht suh rtiultion shows the ntiiptory ortiultion e'ets of the vowel on the onsonnt g preeding itF here were 13 vowelsD 72 g syllles nd 2 semiEvowels in the (nl dtsetF his overs ll min phonemes of the prenh lngugeD ut not in ll ontextsF ih onsonnt ws reorded in the ontext of the three rdinl vowels nd GyGD whih is strongly protruded in prenhF ome intermedite voli ontexts were dded so s to enle the vowel ontext expnsion lgorithm to e hekedF pigure QFQX en exmple of dtset imge nnottion @GaGAF RI pigure QFRX he geEsed rtiultory modelX urve hnge diretions enoded in the (rst three ftors of eh rtiultor @the jwD the tongueD the lipsD the epiglottisD the lrynxAF Expanding the dataset ine the olleted prenh phonemi dtset ws limitedD s needed to expnd it to over other ontexts s wellF e used the notion of the rdinl vowels"GaGD GiGD GuG nd GyGD"ssuming tht GaGD GiGD GuG nd GyG represent the most extreme ples of vowel rtiultionD nd sine then ny other vowel rtiultion n e expnded s omintion of its GaGD GiG GuG nd GyG omponents4F rving ptured the gCGaGD gCGiGD gCGuG nd gCGuGontext for ll onsonnts g nd ll nonErdinl vowels on their ownD s ws le to estimte the missing gC smplesX s projeted the vowel rtiultory vetor @from R 29 A onto the onvex hull of the GaGD GiGD GuG nd GyG vetorsF essuming tht the liner reltionship etween the gC vetor nd the gCGaGD gCGiGD gCGuG nd gCGyG vetors is the sme s the one etween nd GaGD GiGD GuG nd GyGD s estimted gC from gCGaGD gCGiGD gCGuG nd gCGyG using the oe0ients from the projetion of onto the onvex hull of GaGD GiGD GuG nd GyGF s lso estimted the neutrl g on(gurtionD the one without ny ntiiptory e'etsD s the verge of gCGaGD gCGiGD gCGuG nd gCGyGF RP pinllyD s ssumed tht the voied nd unvoied onsonnts did not hve ny di'erenes in the rtiultionF Articulatory model efter mnully nnotting the ptures s shown in pigure QFQ we pplied priniplEomponentE nlysis @geAEsed model on the rtiultor ontours vfIID vgIRD viISF e pid speE il ttention to the intertion etween rtiultors nd the relevne of deformtion modesF woreoverD rtiultors other thn the jwD tongue nd lips re often negleted nd modeled with insu0ient preisionD wheres they n strongly in)uene oustis t ertin points in the vol trtF rere re two exmplesF he position of the epiglottisD whih is essentilly rtilgeD is likely to modify the geometry of the lower prt of the vol trt y dding n rti(il onstriE tion disturing ll the oustisF st is therefore importnt to model its deformtion modes nd intertions with other rtiultors orretlyF sn the sme wyD the velum plys n importnt role oth in ontrolling the opening of the velophryngel portD nd in slightly modifying the orl vity to otin resonnt vities tht give the expeted formnts of vowelsF he ousti tests we hve rried out show in prtiulr tht the velum mkes it possile to etter ontrol the lne etween the two vities neessry for the reliztion of GuG nd GiGF egrding the tongueD ge ws pplied on the ontours delineted from imgesF heformE tion modes re likely to e impted y delinetion errorsF sn the se of the tongueD these errors re mrginlD or t lest give rise to deformtion modes oming fter the genuine deformtions whose mplitude is iggerF yn the other hndD the width of epiglottis ndGor velum is smll on the imgesD nd the errors of delinetionD whether mnul or utomtiD re of the sme order of mgnitude s genuine deformtionsF gonsequentlyD ge pplied without preution will mix oth types of deformtionF o prevent the pprition of these spurious deformtion omponents the epiglottis ws pproximted s thik urveD nd only the enterline of epiglotE tis ws nlyzedF es mtter of ftD the enterline ws determined fter delinetion of ll the epiglottis ontoursD nd the width ws set s the verge width of ll these ontours in the upper prt where the two epiglottis edges re lerly visile @see pigure QFSAF he height of the upper prt @where oth ontours re visileA is djusted y hnd to (t the ontours extrted from imgesF he enterline is pproximted s fEspline nd represented y its ontrol points P l @0 ≤ l < M where M is the numer of ontrol pointsA in the form of twoEoordinte vetorD nd the reonstrution of the epiglottis from the enterline mounts to drw line t distne of hlf the width from the enterlineF he in)uene of delinetion errors is very similr for the velumD whih is firly (ne struture not lwys well mrked on ws imges euse it moves quiklyF es for epiglottis we used the enterline nd firly simple reonstrution lgorithmF roweverD ge ws not pplied diretly on the ontrol point of the splines euse the velum n roll up on itselfF his prtiulrity does not lend itself well to the diret use of geD whih results in the emergene of liner omponents not pproprite in this seF he enterline is therefore roken down into series of segments of the sme lengthF ih segment rtiultes with its predeessor nd the (rst point is (xedF he enterline is then de(ned s the vetor of ngles etween two onseutive segmentsF sn this wy ge n e pplied e'etively to velum nd gives rise to relevnt deformtion modesF he rhiteture nd generl orgniztion of the rtiultory model re sed on the depenE deny links etween the rtiultorsF he min rtiultor is the jw whih is represented y RQ pigure QFSX ipiglottis nd velum enterlines reonstruted y the model @solid lue linesAF eonstruted vol trt is represented y solid red linesF he vol trt input nd output re represented in solid green linesF ell these ontours re superimposed onto the ontours @represented s solid yellow linesA delineted from the imgeF 3 prmeters to get omplete nd urte ontrolF sts geometril ontriution is sutrted from the tongue ontours efore the pplition of ge euse tongue is diretly tthed to the mndileF he tongue is represented y 12 prmeters in order to otin su0ient preE ision for the relistion of onsonnt onstritionsF he lips re represented y 3 prmetersF nlike the tongueD the intertions etween lips nd jw re more omplexF por this reson we sutrt the orreltion etween jw nd lips efore pplying geF he lrynx is onsidered to e independent of the jw nd is represented y 3 prmeters to ontrol ist orienttion nd vertil positionF sn the sme wy the velum is onsidered s n rtiultor independent of the othersF st is nlyzed s explined ove nd is represented y 5 prmetersF he epiglottis is the rtiultor tht is sujet to the gretest numer of in)uenesX the jw vi the tongueD the tongue itself nd the lrynxF hese in)uenes re sutrted y pplying multiple regression to the epiglottis entreline efore pplying geF enlysis of the vrine shows tht the vrious in)uenes on the epiglottis ount for most of its deformtionsF sts intrinsi deformtion re represented y 3 prmetersF sn totl these prmeters form vetor from R 29 @see pigure QFR for mjor prmeter onE triutions to the rtiultor shpeAF ine the model uses geD the zero on(gurtion should orrespond to the entrl position s identi(ed in the dtsetD nd smll hnges in the prmeE ter spe within ertin neighorhood of zero should orrespond to smll hnges @in terms of distne nd shpeD not in terms of the resulting oustisA in the urvesF e lipping lgorithm is used to solve prolems of ollision etween rtiultorsD iF eF essentilly etween the tongue nd plteF o the model9s ehvior is not entirely linerF RR 3.3. Building an articulatory speech synthesis system

3.3.2

Strategies for transitioning between the articulatory targets he dtset provided stti imges pturing idelistiD possily overErtiultedD trgets for onsonnts ntiipting prtiulr vowelsD wheres the gol ws to e lso le to del with onsonnt lusters nd onsonnts tht would not ntiipte ny vowel t ll"for exmpleD due to their ultimte position in rhythmi phrseF oD in our ontextD to estlish trnsitioning strtegy would men three thingsX ghoose the uilding loksX identify the rtiultory trget for eh phoneme in phrseF st n either e wht ws ptured in the dtset @ vowel or onsonnt ssuming voli ntiiptionAD n estimtion of wht the dtset ws missing @missing phonemesD suh s voied onsonntsD missing ontexts or the sene of ny ontextAD or only suset of rtiultory prmeters orresponding to the ritil rtiultors for the prtiulr trgetF e onsonnt nnot ntiipte multiple phonemesD nor n vowels ntiipte nything due to the restritions of the dtset t my disposlF heide when " nd whether " the rtiultory trget should e ttinedF heide how to generte the rtiultory positions etween the trget onesF yur si ssumption ws tht y defultD onsonnts ntiipte the next oming vowelF roweverD it would e unrelisti to ssume it hppens in ll sesF his is why s imposed severl restritions on the ntiiptory e'etX emporlX no ortiultory e'et if the ntiipted phoneme is more thn 200 ms hedY ptilX if there is ny movement sheduled etween the ntiipted vowelD the phoneme in question negtes the e'etF por exmpleD onsider suh sequene s GlkiGX fter GlGD the tongue needs to move kwrd to produe GkG efore oming k forwrd for GiGF sn this situtionD my lgorithm does not llow the GlG to ntiipte the oming GiGF elgorithE millyD it is done with ssoiting every ple of rtiultion to numer nd heking the di'erene etween those numersY gtegorilX it is not possile to ntiipte vowel more thn 5 phonemes hedD nd this restrition eomes striter if it pplies ross syllle oundriesF por vowelsD there is lso model of trget undershootF rving estlished the rtiultory trgetsD the question is how to trnsition etween themF e hve tested out three strtegies for interpoltion etween the trget vetorsX vinerX the interpoltion etween the trget vetors is linerD with shrp turns t the knotsY gosineX smooth trnsitionsY ieewise 1Ed monotoni ui rermite interpoltion tht hs smooth trnsitionsD the mgnitude of eh trnsition setion ounded y its orresponding interpoltion knotsY RS gomplexX trnsitions re done with the previous ui rermite interpoltionD ut the timing vries y the rtiultorsF he ritil ones reh for their trget position fster thn the othersD while those rtiultors whose ontriution to the resulting sound inE telligiility is not s lrge move slower @for exmpleD the tongue n e in numer of positions for the sound GbGD ut the lips hve to ome into onttAF purthermoreD the rtiultors omposed of hevier tissues @suh s the tongue kA move slower thn the light nd highly moile ones @suh s the lipsAF hih of the strtegies is more relisti from the rtiultory stndpointD is question tht n only e nswered with the help of dynmi dtF sn their seneD s hd to restrit myself to the nlysis of the resulting oustisF he hoie of the strtegy did notD howeverD seem to mke di'erene in the qulity of the udio outputF

3.3.3

Obtaining the sound ih vol trt position ws enoded in n re funtionF hey were otined y the lgorithm of rTS with oe0ients dpted y hinji wed nd ves vprieF hese prmeters only depend on the position in the vol trt etween the glottis nd the lipsF he trnsition from the sgittl view to the re funtion hs given rise to severl works whih ontrdit eh other slightly @vwhHP nd wtfIPA nd it is therefore ler tht the determintion of the re funtion will hve to tke into ount the dynmi position of the rtiultors in the futureF henD hving otined the trnsltion into re funtionsD the onstritions were orreted with the knowledge of the phoneme in produtionX stopD fritive or vowelF his wy s ws le to ensure tht ll stops ttined losure t the ple of their rtiultionD ll fritives did not lose too muh or open too wideD nd ll vowels hd enough spe for the ir to pssF prom the development perspetiveD eh ple of rtiultion ws ssoited to setion in the 40Etue re funtion representtionX the lips t [37, 39]D the teeth t [36, 38]D the lveolr ridge t [34, 36]D the plte t [32, 35]D the velum t [17, 33]D nd the uvul t [21, 29]F henD whenever the ple of rtiultion needed to hve onstrition t tht time orD on the ontrryD ould not hve too lose onstrition ording to the timing rulesD the re funtions were orretedX losed vowels were not llowed to hve onstrition of less thn 0.25 m 2 D midElose less thn 0.3 m 2 D midEopen less thn 0.35 m 2 D open less thn 0.4 m 2 Y orl nd nsl stops were enfored to hve omplete losureD 0D t their ple of rtiultionY fritives were not llowed to hve losure of less thn 0.1 m 2 F elophryngel opening ws veri(ed nd orreted s wellX if it ws n orl soundD velophryngel opening ws orreted to 0Y if it ws nsl sound with n opening less thn 0.5 m 2 D the opening ws reset to tht miniml vlueF hen we used n ousti simultion system implemented y ivIT to otin sound from the re funtions nd supplementry ontrol (lesX glottl opening nd pith ontrolF qlottl opening ws modeled y using externl lighting nd sensing photoEglottogrphy @eqqA mesurements rwHVF ithin the model s implementedD glottl opening is reltive vlue from 0 to 1D 0 orresponding to most losed @s in vowelsA nd 1 orresponding to s open s possileF he key vlue to ttin during the prodution is 0.0 for vowelD 1.0 for voieless fritive or stopD nd 0.7 for voied fritive or stopF he other phoneme lsses do not hve key trget vluesF henD empirillyD for every senrio @the opening vlue inresing from one pekD suh s 0.7D to notherD or deresing from one pek to notherD or moving towrd or from RT 0AD s dded uxiliry points to mth the shpe nd temporl ehvior of the few exmples in the dt tht were t my disposlF here ws no need to model voiing @highEfrequeny osilltions of low mplitude superimE posed onto the glottl opening wvesA sine the vol folds operted y the glottl hink model ivITD ivIT re selfEosilltingF 3.4 Evaluation ih step in the system ws evluted on its ownD nd fterwrds the synthesis results were evluted visullyD oustilly nd pereptullyF ine the ojetive of the work ws rther to hve fully funtionl lgorithm tht produes resonly relisti movements nd sounds rther thn to otin highEqulity speehD more rigorous evlutionD suh s quntittive omprison to the dynmi dt on rtiultory trjetoriesD is still n venue of future workF

3.4.1

The articulatory model and the trajectories yne peulirity of the dtset nd therefore of the model ws the ft tht it used only the sgitE tl setion of the speker9s vol trtF hile full threeEdimensionl models n pture the full geometry of the vol trt with suh phenomen s lterl phonemes @eFgF GlGAD twoEdimensionl models get the ene(t of fster omputtion time nd overll simpliityD ut irreversily lose the sptil informtionF sn generlD the rtiultory model ptured vol trt positions orretly or with no ritil errorsD though some prt of its suess is de(nitely owed to the postEproessing stge where re funtions re orretedD sine on its own the model did not impose muh ontrol over onstritionsF his wyD ontrol eme twoEfoldX the rtiultory model operted t the level of rtiultorsD nd the postEproessing set of rules on the resulting vol trt geometryF es for the movementsD we n sy tht they were resonle nd the ortiultionE'eted trgets guided the rtiultors to the positions neessry to produe prtiulr utterneF yne key point here is the timing strtegyF uleEsed timing strtegy seems to e very rigid for the dynmi nture of speehY it would e more nturl to follow speeh prodution proesses in humns nd to guide the synthesis with the eliited sound or the speker9s expettion"sed on their experiene"on wht this sound will eF

3.4.2

Glottal opening control he lgorithm for the glottis opening suessfully llowed to distinguish etween vowels nd onsonntsF histinguishing etween voied nd voieless onsonntsD thoughD stys point for improvementD s well s wellEoordinted ontrol over the glottis nd the vol trt to void ousti rtiftsF

3.4.3

The synthesized sound owels nd stops were the most identi(le nd orretD lthough sometimes some minor djustE ments in the originl dt were neessry to otin formnts lose to the referene vluesF hen ompred to humn speehD the formnt trnsitions within the suggested strtegies sometimes RU ourred too fst nd sometimes too slowlyY ginD this highlights the utmost importne of relisti timing strtegiesF pigure QFT shows n exmple of the synthesis when it is guided y rel timingX GaSaG s produed y the system nd s uttered y humnF he highEfrequeny ontriutions in GSGD not ppering in the humn smpleD re due to the ousti simultionF he noise of GSG is t the orret frequeniesD ut with it di'erent energy distriutionD proE ly euse of di'erenes in rtiultion or in the re funtionsF here is lso n ousti rtift etween GSG nd GaGD whih mens tht more work is neessry on liising the vol trt nd the soure ontrolF pigure QFTX en exmple of humn9s utterne of GaSaG @leftA nd its synthesis @rightA long with the glottl losure ontrol s opied from the iqq dt @elowAF honeme durtions re lignedF 3.5 Conclusion

Overview of results

egrding speeh s proess of trnsitioning etween ontextEwre trgets is n interesting pproh tht n e onneted with the mentl proesses of speeh produtionX to llow the others to pereive the neessry ousti ueD the speker needs to ome lose enough to the ssoited rtiultory golF he importnt di'erene etween rel speker nd the lgorithm is the ft tht the lgorithm solves stti prolemD lid out in fullY it needs to hit prtiulr trgets in given orderF es for humnsD we solve dynmi prolemD nd ortiultion is not something we put in its de(nitionY rtherD ortiultion is our mens to mke the prolem of rehing too mny trgets in too short period of time solvleF he sttistilly derived rtiultory model enodes omplex shpes of the rtiultors in only 29 prmetersD sometimes struggling t the onstritions euse of the inherent"nd intentionl"lk of ontrol over the resulting geometry of the vol trtF his drwk is ddressed with postEproessing stgeD whih is ompromise @though rutlA solutionF he shpes of the rtiultors hnge in time ording to the produed trjetories of the vol trtD nd those re phonetilly soundF hey evolve in synhrony with other speeh RV prodution prmetersX pHD suglottl nd suprglottl pressureF ogetherD the timing nd vlues of the system need to e in delite lne so s not to produe ny rtifts down the synthesis pipelineF e (nd tht they re su0iently well tuned for vowels nd stopsY s for fritivesD the interply etween the ple of rtiultionD pressure ontrol nd the temporl evolution is so intrite tht it essentilly oils down to modeling eh fritive seprtelyF s onlude thtD within the lid out setting"omprehensive ut stti nd little in quntity dt of the vol trtD sttistil model to enode itD nd set of rules to mnipulte it" this work represents thorough explortion of the pprohF he results show tht suh dt nd methods re not unsuitle for uilding n rtiultory speeh synthesizerY howeverD given the limittions of the pprohD more promising ourse of rrying on with this work would e inorporting ptterns from some tully reorded dynmi dt rther thn ontinued serh for etter set of rules through theoretil modelingD tril nd errorF wore on this in the following setionF 3.5.2

Future work es ws sid oveD for the pproh to work wellD loserD intertwined nd wellEligned interE tion etween ll its omponents"vol trt on(gurtionD voiing nd pressure ontrol"is neessryF gonsidering the unountle mount of timing strtegies nd possile trnsitionsD it is resonle to lern them from dynmi dtY for exmpleD pprohes y ivyIT ould e used for thtF imploying nnotted dynmi dt suh s iwe or Ews nd ugmenting the rtiultory model used in this study with temporl ontrolD ligned with voiing ontrol s extrted from simultneous udioD should onsiderly improve the resultsF edditionllyD exploring onnetion etween dynmi dt nd the present system would llow to properly evlute the generted trjetories nd the soundF enother pproh to oosting the performne of the system would e to utomte the serh for etter rulesX set ost funtion for how good generted rtiultion smple nd its udio reD hve severl initil sets of synthesis prmetersD generte the output with themD evlute the resultD djust the prmetersD reEgenerte the outputD mke onlusions out the new djustment of the prmetersD etFD until the improvement disppers or is negligile" omprle to how this relting to the result ws treted in fIUF hisD like tking ptterns from tully oserved dynmi dtD is lso very produtive ideD ut di0ult to impleE ment in setting with multiple disjoint omponents rther thn single environment s in olrtv firIQF RW Articulatory speech synthesis from real-time MRI data he previous hpter explored rtiultory speeh synthesis where the spet of rtiultion ws treted s n extrpoltion of the ville stti dt with no referene of its dynmisF he ovious inherent pitfll of tht pproh ws the temporl spetX mnging the durtionD the speed nd the trjetory of rtiultory gesturesF o ddress thtD this hpter is dedited to rtiultory speeh synthesis with ll its omE ponents derived from relEtime dtF 4.1 Objectives he ojetive ws to ddress oth rtiultory nd ousti spets of speeh y tking into ount ll the e'ets tully oserved in use y humn speker during normlD nturl nd unrestrined speehY more spei(llyD to uild joint model to synthesize oth rtiultory nd ousti prmeters tht would orrespond to given phonetized textul inputF ht inluded piking pproprite dt nd methods with onsidertion for the di0ulties of treting rtiultionD pplying those methods to the dt nd evluting nd nlyzing the resultsF pei(llyD the ojetive ws to prepre Ews dt s rih soure of rtiultory informtion tht n e olleted t good enough rte without tmpering with the nturl proess of speeh nd use oth its visul nd udio omponents jointly to drive n rtiultory speeh synthesizer derived from ptterns in the rel ehvior of the speker rther thn set of developed rulesF qiven the longEterm ojetive to synthesize speeh sed on relEtimeD tully oserved ousti nd informtionErih rtiultory phenomenD it ws neessry to get good geometri desripE tion of the whole vol trt nd to get running speeh whih would exhiit how the glol geometry of the vol trt evolves over time during speeh produtionY Ews rtiultory dt seemed to e good (t for thtF hile it is reltively esy to (nd udioErtiultory dtses in inglish xfq + IID + IUD there re no similr free dt in prenhF his prompted o' the retion of ertpeehwsfr hpp + IWD the dtset used for this workF Dataset conceptualization and creation e dtset neessry for n rtiultory speeh synthesizer needs to meet severl riteriX por the synthesizer to e roustD the orpus must hve good overge of ll phoneti ontexts whih n pper in prenhF goverge is of higher priority thn the numer of instnes per eh phoneti omintionF por the synthesizer to e nturlD nd for us to e le to evlute the nturlnessD the dt needs to e nturlF por the eventul rmi(tions of rtiultory speeh synthesis for speeh sieneD one my dd utternes with some phenomen of theoretil interestD suh s the rtiultion of sounds with omplited physis of produtionD rtiultory djustment phenomen nd intrEspeker vriilityF he eventul results tht ome from using the dtset need to e in form tht is omE prle to previously existing workF yn top of thtD the nture of Ews quisitions imposes few more restritionsX he orpus plnned for reording nnot e too longD oth for the omfort of the speker who is sujeted to long period of immoility in tight spe with loud noiseD nd for the ost nd limited vilility to use the mhineF he solution to split the orpus into severl prts to e reorded over severl sessions is suoptimlD s it ould led to gret interEsession vriility in terms of the speker9s hed position nd speking styleY he speking tsks need to e esily understndle y the spekerD sine ommunition etween the ontrol sttion nd the person in the mhine is noisy nd unomfortleF SP egrding phoneti overgeD let us onsider the lssil wy of onstruting orpus for speeh synthesisD whih is to ollet sentenes from vst written orpusD for instneD of newspperF he greter the olletion of sentenesD the etter its overgeF he prolem with this pprohD howeverD is tht eh sentene ontriutes only very limited numer of new phoneti ontextsF e orpus vst enough to over enough speeh phenomen to serve s sis for speeh synthesizer would hve thus required very long reordingD more thn dozen hours of speehD whihD s explined oveD is not n option for wsF e good solution for our se ws to spei(lly prepre set of sentenes for the sujet to red outD so tht the overge nd lne of phoneti omintions eme extly s neessryF he set of sentenes for ertpeehwsfrD generted y ves vprie from phonetized version of prenh worphlou lexion whih provides TPHFHHH )exed forms tweHTD vqHRF ves vprie used severl levels of riteri to guide the mnul omposition of sentenes with the ville phonetized lexil entriesF yn inserting new senteneD the (rst level is to ount its numer of doule vowels V 1 V 2 for ll vowelsD the numer of CV with the onsonnt C eing one of Gp, t, k, f, s, S, l, KG nd V of Gi, a, uG or"dditionlly"GyGD the numer of VC where C is in the list Gl, K, n, mG s od nd V in Gi, a, u, y, e, E, o, OGD onsonnt lusters C 1 C 2 V with C 1 in Gp, t, k, b, d, g, fGD C 2 in GK, lG nd V in Ga, i, u, yGD nd the numer of instnes of IS spei( omplex onsonnt lusters @t lest sequene of Q onsonntsD etween two vowelsAF ixept for those omplex lustersD ll the ontexts in question lmost lwys pper within words to void the e'et of prosodi oundriesF uh orpus omposition overed the ore of mndtory phoneti ontextsF henD new dditionl ontexts were introduedD in prtiulrD ontexts with vowels not present in the set of the rdinl vowels Gi, a, uG extended with GyGF he instnes of VCVs were roken down y grouping vowels into lsses from the open to the lose @Gi, eGD GE, aGD Gu, o, OGD Gy, øGD Goe, @G nd nsl vowels Gã, õ, Ẽ, oeGAF his provided seond level of phoneti overge evlutionF sn totl this orpus ws mde up of UW sentenes o'ering very good overge of ll possile phoneti ontexts in prenhF he disdvntge of the sentenes eing rti(illy onstruted is their highly odd semntis @eFgF ous dtylogrphiez s soupe sirupeuse u lit"Fr. ou type his syrupy soup in edA ndD in the se of two wordsD nonEprenh inlusions @rtoonsD squwAF roweverD synttilly they re orretD nd ntive speker should not hve ny prolem reding themF edditionllyD for omprison purposesD n dditionl set of syllles nd sentenes oming from previously existing orpor @the frequently used in phonetis 4v ise et le soleilF F F 4 sequene sntWW nd rndomly seleted suprt of the orpus in wgyIIA ws ddedD s well s sequene of syllles mirroring nd extending our stti orpus @ghpter QFQFIAF elsoD to study the omplex rtiultion of trills @tht re not present in stndrd prenhAD experimentl sequenes of GaBa, uBu, iBiG nd Gara, uru, iriG were ddedF es for speeh nturlnessD one ould see how reding out rti(illy onstruted sentenes tht would never e voied in ny rel ontext is proly not the est representtive exmple of nturl speeh dtF por this resonD we wnted to reord some spontneous speehD whih is less ontrolled thn reding out loud nd lso mnifests more e'ets like overlpping gestures nd phoneme elisionF s reted prompts for eh sujet to follow for minute per promptD in rndom orderF hey overed everydy topisX ht do you like in your workcD pek out your lst trip nywhereD pek out (lm or ook tht hd lsting impression on you @PH topis in SQ totlD see le eFIAF sn the endD t the reording timeD despite hving hesittions in speehD oth sujets hd enough to sy to (ll the llowed minuteF pinllyD during the quisitionD we hd n opportunity to reord dditionl dt with S B F his led to the retion of utternes reorded in nonEsgittl slies t n ngleF his prtD howeverD ws (ltered out in my workF Technical settings he dt were reorded on iemens rismE(t Q snner @iemensD irlngenD qermnyA t wx lnk snstitute in qöttingenD qermnyF e used rdil pEspoiled pver sequene + IH with = 2.02 msD i = 1.28 msD py = 19.2 × 19.2 mD )ip ngle = 5 degreesD nd slie thikness is 8 mmF ixel ndwidth is 1600 rzGpixelF smge resolution is 136 × 136F he quisition time vried from 34 se to 90 seD mostly out 60 seF e followed the protool desried in xu + IQF smges re reorded t frme rte of SS frmes per seond with the lgorithm presented in + IH @more on the frme rte will follow in ghpter RFPFIAF he sujets re two dult mle prenh ntive spekers speking prenhF ujet AD S A D is the sme s in the stti dtset from ghpter QD now with the following mesurementsX mleD QS yers oldD IVP m tll nd UR kgF ujet BD S B D is mleD QP yers oldD IVH m tll nd TS kgF eudio is reorded t smpling frequeny of 16 krz inside the ws snner y using pyws sss optooustis (reEopti mirophoneF he sujet wers erplugs to e proteted from the noise of the snnerD ut is still le to ommunite orlly with the experimenters vi n inEsnner interom systemF ine the sound is reorded t the sme time with the ws quisitionD there is dditionl noise of the mhine in the udio signlF sn order to denoise itD we used the lgorithm proposed in yfIPF ine the noise ws so strongD disruptive nd present lso in the frequeny nds of speehD the denoising lgorithm ended up in removing some energy from speeh formnts tooD whih ws espeilly notiele in nsl soundsF he resulting signl is onsiderly more intelligile thn the originlD ut the noise is still presentF Transcribing the corpus er eh wv reordingD s produed text (le with the text of wht the speker sys thereF he tokens with nonEstndrd pronunition were stored in seprte simple text (le long with their phoneti trnsription in ewe + WU @suh s jpense d p ~n s re)eting the ft thtD s it is ommon mong ntive spekers of prenhD S B devoied his GZG from je GZ@G efore voieless fritive GSGA"GS@GF he trnsription proedure ws sed on the guidelines for wgHQX xumers re written in wordsY eronyms @tht re pronouned letter y letterA hve to e written letter y letterX Je suis allé en prépa scientique, en P C S I 4Y £ usesX mrked with the plus sign @+ or ++ if it is very longAY pusionsX mrked with n undersore @ch_pratique to men je prtique ut relE ized s GSpKa.tikGD ça va_êt' Internet to men ç v être snternet ut relized s Gsa.vE.tẼ.tER.nEtGAY rtil wordsX mrked with n postropheD if it seems to e n intentionl wy to inrese the speed of speeh while retining intelligiilityD or hyphenD if the word is just ndonedD whih is often followed y restrt " see elow @bah je rent' assez tard D dans l---dans la jungle AY estrtsX mrked with doule hyphens @pas de petit-déjeuner mis à part, euh mis à part du café AY wispronunitionD nonEstndrd wordsX sterisk efore the wordD no spe @je me demande si ça sert à *queqchose AY nintelligile speehX the losest guessD if there is ny possileD put inside doule prenE theses @((mais bon)) or en tant qu'un amateur, ((cvraiqu)) j_vais : dans des musées AY snterjetionsX treted s ny other lexemes @bahD hmD etFAY etionsX {BR} for rethD {CG} for oughD {LS} for lip smkD {LG} for lughE terD {NS} for loud kground noise @side from the noise of the mhine whih is lwys presentAF his proedure posed some limittionsF sn prtiulrD sine the ilil trill /B/ nd the lveolr trill /r/ do not elong to the prenh lngugeD they re not reognized y prenh eweF hereforeD they hd to e misleledX suh sequenes s /aBa/ nd /ara/ hd to e trnsried s /aba/ or /ava/ nd /aKa/F e phenomenon tht turned out to e impossile to represent within the frmework of ewe ws stops with long lok phseD ourring in sequenes like crabes bagarreurs 4D GkKab^.ba.ga.KoeKGX the (rst GbG hs no udile relese nd is followed y the next GbGF heir trnsriptions were deided upon on n individul sisF Phonetic labeling o trnslte the trnsried text into its phoneti trnsriptionD s tried out two toolsX estli pwtIS nd evite r ffhIRF he ene(t of estli is the ft tht its phoneti leling is more )exileD giving me wy to tke into ount the nonEstndrd phoneti tokens s took note of in the supplementry phonetizing ditionry (lesF sts disdvntgeD howeverD is the ft tht its output ws rt extqrid (le fIV with no linguisti informtionD while s needed more omplete overge to trin n rtiultoryEousti modelF SS he output of evite r is in the r formt enHTD nd it gives every phoneme riher piee of informtionX it inludes the phoneme identity itselfD phoneti ontext @two phonemes efore nd two fterD thus forming quinphone togetherAD position in the syllle @forwrd nd kwrdAD stressD ent nd the numer of phonemes in the syllle @nowD in the previous syllle nd in the next oneAD the vowel in the urrent syllle s well s text prsing round the phoneme nd utterneElevel informtionF ell in llD it ws more resonle to keep using evite rF roweverD upon loser look it turned out tht the tool hd ugX despite pssing to it input texts in orret enodingD it proessed them in egssD resulting in n inorret tretment of ll prenh diritisX whenever hrter ws not reognized"let us syD é tht should e phonetized s GeGD"it ws phonetized s GGF fesidesD the tool ould not del with postrophes or other symolsD thus mishndling even more sesF o (x phoneti leling of evite rD s employed set of text replement rules so tht the instnes tht were known to impede evite r were repled with hrter sequenes tht it would hndle orretly @see le RFIAF xturllyD these sustitutions hindered the prsing unit of evite rD so for ny given text s rn evite r twieX one to otin the orret phoneti informtion ut inorret syntti one nd one vie versF o omine the outputsD there ws n issue tht the orret phoneti lels did not hve to hve oneEtoEone orrespondene to the syntti onesF ometimes it would e expeted to insertD delete or sustitute elements in one sequene of lels or the otherF hus s mthed the sequenes through vevenshtein9s distne vevTSF s put the ost of sustitution to I nd the osts of deletion nd insertion to PD so tht orretion of the lel is more likely to hppen thn opy or n omissionF his proessing (xed ll mistkes identi(ed when reviewing lelsD ut lso reted some newF por exmpleD the hrter sequene ai n e phonetized orretlyX faire Gf E KGY howeverD when treting the word système"normlly phonetized s Gs i s t E mG"nd repling the hrter è with the sequene ai @systaimeAD whih voids using entution ut should e phonetized the smeD evite r produes n inexplile phonetiztion Gs i s t E i mGD iFeF the sound GEG is orret nd should over oth letters a nd i ut is followed y n extr GiGF rving no open soure ode of the toolD it is di0ult to determine the use of this errorF pinllyD eEvite r ssumed tht every (le strted with phonemeD while tully the eginning of every reordingD leit shortD ws sileneF o s prepended every output of eEvite r with n extr phoneme silF he next step ws to perform stte fore lignment to estimte the oundries of the phoneti lelsF s used rite from ru iq + HP with werlin s frontend uITF por eh speeh (le"in my (rst run whole singleEquisition reording @one minuteD typillyA"rite loded the orresponding lel (leD expnded it to rete n lignment network nd produed phoneti trnsription ording to the output proilities of the rwws in the networkF henD to improve phoneme lignment nd redue susequent omputtionl lodD s split eh reording into sentenes ording to the timing of stte foreEligned lel (les nd eh textul trnsription ording to the prenh sentene tokenizer sent_tokenize of xvu vfHP nd mnully heked the utEo' points for the udioY it turned out tht oftenD espeilly with sponE tneous speeh nd syllle y syllle enunitionD the foreElignment lgorithm put sentene ST 4.2. Methods ghrter omintion eplement BD @D AD empty string E t the end of word @mrks the speker utting their )ow of speehA empty string X @mrks long syllleA empty string cD 3 F 9D D 9 empty string or lnk speY if possileD reE ple the word efore the postrophe with its omplete form @queD lorsqueD jusqueD presqueD leD laD jeD meD teD seD ce @espeilly importntD to void phonetizing it s GkGAD de A rés or dés t the eginning of the word C vowel rer or der C white spe C z C vowel other instnes of és C vowel er C white spe @to void pronouning the r A C s C vowel éD possily followed y silent e or ominE tion of dD s nd t t the end of the word er C white spe @to void pronouning the r A è ai @to otin the sound GEGA àD ù aD u respetively ê ai âD ûD ôD î aD uD oD i respetively ë e t the end of the word @eFgF ambiguë AD otherE wise ai @NoëlA ï i C white spe @eFgF naïve → na ive A üD possily followed y n e t the end of the word u ö or ü @words of qermn origin retining the originl orthogrphyD not present in the orpusA oe nd ai respetively ç s ñ @words of pnish origin retining the originl orthogrphyD not present in the orpusA gn ae ae ÷ e le RFIX epling hrters in evite r inputs so tht they get phonetized orretlyF oundries inorretlyF sn suh ses the text ws orreted to orrespond to the udioD even though this did not represent full sentene thenF his wy the entire orpus gets trnsformed into olletion of short text nd udio phrsesD rring the (les where phonetiztion of eEvite filed due to internl errorsF hen s regenerted phoneti lels from the sentene trnsriptions nd performed nother run of stte fore ligning the sentene udios with those lelsF efter ll the improvementsD the qulity of phoneti nnottion eme quite resonle provided tht the lel (le listed the orret phonemes for the sound sequeneF nfortuntelyD this ws not lwys the se euse of the sene of proessing nonEstndrd pronunitions @eFgF the elision in Bqueqchose4 GkEk.SozG insted of quelque chose 4 GkEl.k@.SozGAD errors in SU splitting into singleEsentene reordings or errors in phonetiztionF Synchronizing audio, video and phonetic labels hespite the ft tht ws quisitions re onduted in wy tht is s ontrolled s possileD with stle settings nd (xed position of the sujetD there re ftors tht will inevitly vry from one quisition to notherF he most imptful of them is the tempertureF he mhine itself nd the ir inside it"s well s the ody of the sujet"n het up nd ool down long the quisition timeD ording to the periods of ontinuous use or reksF his n use vrition in imge quisition rte tht n go up to I7F sn our seD the expeted rte is SS rzD whih mens n imge every IVFIV msD nd oneEminute long quisition should hve QQHH imgesF e vrition of I7 would men etween frequeny etween SRFRS nd SSFSS rzD tht isD tking from IVFHH to IVFQT ms to mke one ptureF eloneD shift of ±HFIV ms is negligile for rtiultory events tht re ptured y EwsF he only ple where suh short period of time would e sensitive is the vol foldsD ut their tivity is not registered nywyF roweverD when this vrying time per imge umultes over the ourse of the quisitionD ligning the udio trkD the durtion of whih stys s set in dvne efore the quisitionD with the resulting video n led to onsiderle shift up to round THH ms per minuteD sine we ould e deling with up to round QQ imges tht go missing or re unounted forF his durtion n orrespond to round ten phonemes in strem of speehD thus mispling the phoneti lelsD espeilly t the sequene endF o ommodte to thtD one s split the udio nd the trnsriptions of the orpus into sentenesD s nlyze the orresponding sequenes of imges per eh quisitionF hue to the setupD the (rst few imges do not orrespond to ny sound due to the gp of 70 ms from the strt of reording the imge to the strt of reording the udioF es for the imges in totlD just s preditedD their numer vries from one tke to notherF sn the treted sesD 95 quisitions hd more imges thn ntiipted @8.39 more imges on vergeD stndrd devition of 5.70AD 11 hd fewer @5.64 fewer imges on vergeD stndrd devition of 6.09AD nd 3 hd extly the ntiipted numer of imgesF sn totl it mkes 6.74 more imges on vergeD stndrd devition of 7.15F e filproof method to ligning imges to udio timing is to (nd phoneti lel with very short nd reognizle visul ueD for exmpleD GpG requiring the lips to loseD t the eginning nd the end of the sequeneF he imges orresponding to this lel will then e identi(ed unmiguouslyD nd hving ler orrespondene etween time stmps t the two ends of the sequene nd the imges should extrpolte to other imges quite orretlyF his isD howeverD n pproh tht is etter to pply mnully rther thn through utomti detetion of lip losure euse of the mny possiilities how it n go wrong if one does not lso hek the neighoring vol trt shpes nd how ptly they re)et neighoring phoneti lelsX for exmpleD the lips my or my not lso lose for periods of sileneD or the lips my e reognized s open euse of the ontt eing too )eeting or euse of proessing errorF husD to ssoite every imge to n r lelD s mde the ssumed frme rte )exileD orresponding to the numer of ville imges for the given udioD tking re to hek tht the estimted frequeny stys within the predited limits of 17 vritionF his wy s identi(ed 81811 msElong sequene with 326 imges missing @tht isD no ptures for some 6000 msAF sn the end s otined orrespondene etween ny frme of video from the orpusD the SV 4.2. Methods time in the udio nd its ssoited phoneti lelF es foreEligned lels re provided every 5 msD tehnilly this mens skipping few lels with eh new frmeF sn relityD howeverD phonemes do not hnge s frequentlyD so lels repetD nd skipping few does not rete prolemF he next stepD whih is going to e overed elowD ws to proess the frmes nd extrt rtiultionErelevnt informtion from themF he timing of these rtiultory prmeters reE veled ertin issues with the synhroniztion tht ws presented in the urrent setionF Processing the frames en Ews ptureD s shown in pigure RFID hs lot of res tht re irrelevnt for studying rtiultionF o perform rigorous utomti proessing of the imgesD we need to proess the imge so tht it only ontins the re of the vol trt nd we re le to extrt rtiultorily relevnt informtion from itF hen hoosing methods to do soD sine the purpose of working with Ews dt ws to hve omplete informtion on the position of the rtiultors of the spekerD the preferene ws given to those tht lose s little preision in the shpes of the rtiultors s possile" rther thnD for exmpleD preferene for the speed of omputtion or the interpretility of the resulting imges y humnF pigure RFIX en exmple of n originl Ews frmeD efore proessingF pigure RFIX moothing the frme in pigure RFI with ilterl (lterF pigure RFIX epplying dptive thresholding to the smoothed frme in pigure RFIF Ews frmes re rther noisyD so the (rst step tken y sonnis houros ws to redue noise through smoothing zeIHD n opertion where eh pixel is sustituted y the result of some kind of opertion on its neighorhoodD most ommonly y weighted sum of the pixels round itF his ide works well due to the ft tht typillyD imges hnge their pixel vlues grdully over speD nd verging neighorhood n help get rid of the outliersD the pixels tht re less orrelted with the (lter9s enterF roweverD prolem here is tht oundries of ojets @in our seD the outlines of the rtiultorsA represent edgesD nd the ssumption of slow hnge filsF e my verge wy the line tht ws previously shrp nd lose some essentil rtiultory informtionF his is why smoothing ws hosen to e performed through pplying ilterl (lter wWVD whih ws developed so s to preserve edgesX in the regions where pixel vlues re quite similr SW to eh otherD it ts s stndrd domin (lterD ut it will not ross shrp oundry in this vergingF he dimeter of eh pixel9s neighorhood ws d = 9 @the originl imge size eing 136 × 136 pixelsY suh vlue of d orresponds to very lrge (lterA nd oth (lter sigms in the olor nd oordinte spes equl to USD whih is moderte hngeF his prmeter setup is eptle for o1ine pplitions tht need hevy denoising rther thn the speed of omputtionF en exmple of smoothing typil Ews frme from the dtse @pigure RFIA is given in pigure RFIF yne the imge ws smoothedD we wnted to simplify it for further tretmentF his n e done with thresholdingF he ide of thresholding grysle imge is to reple ll pixel vlues ove ertin threshold with one olor @whiteAD nd ll pixel vlues elow with nother @lkAF roweverD peulirity of ws ptures is tht pixel intensity is not the sme in di'erent regions of the frmeD depending on the ws oils"loops of ondutive wire round the oreD used to rete mgneti (eldF he nose nd the frontl regions of the vol trt"the lipsD the tongue tip"re visulized with muh greter intensityD s lmost white shpesD thn the k regions @the phrynxD the lrynxAD where we n only see lurred gry outlinesF por suh imgesD simple thresholding with glolly ssigned threshold vlue is not suitle methodY it is est to hve lol threshold vlue per eh luminosity regionF his pprohD hosen y sonnis hourosD is lled dptive thresholdingF e blockSize×blockSizeEregion n e hrterized y its verge vlueD ut to use it s threshold mens to keep in too muh noiseF e etter solution is to set the threshold vlue T (x, y) sed on the neighorhood9s @ 11×11aretngulr round point (x, y) in our seA rossEorreltion with qussin windowD minus n djustment onstnt C = 2F plooding the spe outside of the hed of the speker with the white olorD we otined imges s shown in pigure RFIF o filitte rtiultory informtion extrtionD it ws importnt to limit the study to single re of interest t timeF e used the tip of the nose s one point of refereneD whih is the middle of the leftmost points on the speker9s hedF sn ses when the nose tip stryed too fr from where it ws on verge in its sequene @more thn Q pixels in either of the dimensionsAD sine the speker9s hed ws not supposed to e movingD s ssumed it to e n error nd reset it to sequeneEspei( defult vlueF prom thereD s identi(ed the vlues to hrdEode windows ontining @A the vol trtD pigure RFP @used for the work in the next hpterAD @A the velum @used for the work in this hpterAD pigure RFP nd @A the lipsD pigure RFP @used for the work in this hpterAF heoretilly the reltive lotion of these windows with regrds to the nose tip ould depend only on the speker9s ntomy nd e therefore spekerEspei(D ut sine the hed ngle vried from one quisition session to notherD it hd to e sessionE spei(F edditionllyD in the individul imges where s ould utomtilly identify tht the lgorithm ws filing to extrt the fetures @see the detils in the following susetionsA due to slightly mispled windowD the window ws shifted in the diretion tht ws expeted to solve the prolemF Extracting features sdellyD n rtiultory speeh synthesizer should work with rtiultory informtion tht is s rih s possileF here lso is very strong requirement for this informtion to e highly preise so tht it remins interpretle nd sound from the point of view of the physis of speeh produtionF he pprohes to get this informtion n e mnulD semiEutomti nd pigure RFPX ol trt @RFPAD velum @RFPA nd lips @RFPA windows for the Ews frmesF utomtiF por given midEsgittl ws frmeD stti or dynmiD trined humn with the knowledge of phonetis nd of the ntomy of speeh orgns n drw the ontours of lmost ll the orgns tht re known to ply prt in speeh produtionF he trining is neessry to lern to e onsistentD for exmpleD in the hoie where the lips egin nd end or how the ottom prt of the tongue is nnotted when it is visileD nd solutely essentil to orretly hndle di0ult sesD suh sX he tongue tip is one of the fstest rtiultorsD prtiipting in rpidD preisely ontrolled wideErnge motions to produe numerous phonemesD suh s the voieless lveolr stop GtGF he tongue dorsum is less soD ut thnks to jw displement it n e gile tooF hese movements result in ghostingD for exmpleD n imge with two outlines of the tongue or physilly impossile shpeD or in merging the oundries of the tongue with neighoring rtiultor @see pigure RFQ for n exmpleAF st my pper tht the uvul eomes in)ted nd inreses in volumeD whih relly is either lurring @pigure RFRA or the uvul rolling up onto itself @pigure RFRAF fesidesD the velum is represented with lurry shpe of only few pixels9 widthF ometimes the phoneme tht is eing produed needs to e tken into ount to orretly identify whether the velum touhes the phryngel wll or notX if it is nsl soundD there should e ontt to lose the velophryngel portD nd if it is n orl oneD noF he epiglottis is thinF feing set deep in the vol trtD fr wy from the frontl side tht ws visulizes with greter intensityD the preise shpe of the epiglottis n e di0ult to disern @pigure RFSAF fesidesD it n press up to the tongueD thus hiding its edge @pigure RFSAF es resonting systemD the vol trt is hrterized not only y its width t ny pointD ut lso y its lengthF yne end is ler to seeD the lipsY menwhileD the other endD t TI the vol foldsD n only e guessed upon @pigure RFTAD sine the position of the lrynx is extremely unstle in visuliztionD oth due to its ntomil struture @very little tissue to see in the midEsgittl frmeA nd due to its position t the rer of the vol trtF /e/ /s/ /s/ /s/ /s/ /@/ /k/ /k/ pigure RFQX he tongue in frgment of n Ews sequene Gs(@)kGX note the doule lurred outline of the tongue tip to produe the lveolr fritive GsG nd the lurred shpe of the tongue dorsum when mking onstrition for the velr stop GkGY in the ltter seD it is lso di0ult to see where the plte ends nd the tongue eginsF pigure RFRX wo min uses of the velum ontour to e di0ult to nnotteX rpid motions nd n intrite geometryF eside from eing this hllengingD mnul nnottion lso is quite txing nd extremely timeEonsumingF goupled with the opious mounts of dt in the Ews orpus @round two hours of speehAD ll the points ove ruled out the option of mking nnottions y hndF enother solution would e semiEutomtiX to hve seline nnottor nd use humn to orret the urvesF he prolem with this pproh ws tht we needed the dt to e up to very high stndrdD sound from the perspetive of physis nd with gret degree of onsistenyF his led to the need to perform t lest minor orretions in every single imgeD nd evluting the nnottor9s performne showed tht orretions did not get to e done ny esier or fster thn drwing the ontours oneselfF e more utomti pproh would e to perform mnul preprtory work on some Ews imges nd run n existing ontour extrtion nd identi(tion methodD for exmpleD nnotte pigure RFSX wo min uses of the epiglottis ontour to e di0ult to nnotteX merging with the tongue nd eing rely visileF pigure RFTX he outlines of the lrynx n e very di0ult to disernD whih n led to wrong estimted position of the vol folds nd wrong vol trt lengthX S A pronouning GkGF prt of the frmes nd use it to trin sttistil or deep lerning modelF hespite the hllenges posed y high vriility in the vol trt shpe nd on(gurtion when produing the extensive vriety of speeh soundsD di'erenes in spekers9 ntomy nd the onnetivity of the trt irwy to other hnnels of ir through lip openingD velum opening or lrynx gIQD this hs een proli( diretion of reserhD oth with rtiultory dt tht ws more in use in pst dedes nd those tht re eing explored nowD in prtiulrD Ews pturesF ome notle exmples re methods tht strt o' prede(ned urve for n rtiultor or prede(ned vol trt shpe nd try to void deforming it too muh when trnsitioning etween frmesD suh s geometrilly onstrined snke model tht lso relies on the informtion from lndmrksD ontt points nd pronouned urvture res p + IVD or models relying on the vol trt pperne eiIU or the pperne of some vol trt omponents xISY unsupervised trking methodsD usully providing less informtive outlinesD suh s fxHVD v + IQD fuxIHY nd supervised methodsD relying on nnotted exmples ndGor lndmrksD suh s ifIID gIQD uqIVD qr + IWF sn this diretion s explored the imge segmenttion method of x + IRF sts ide is to rete templtes"orretly nnotted frmesD"run the segmenttion lgorithmD identify frmes with signi(nt delinetion errorsD orret their ontoursD sve the new ontours s templte nd rerun the lgorithmF en exmple of templte tht s (tted to our dt is shown in pigure RFUF xeverthelessD imge segmenttion proved to e hrd prolemD prtiulrly exerted in our se y the need to hve results tht would e very preiseF rving weighed in the expertise nd the estimted long hours of work neessry to identify imges to serve s templtes nd to tret themD it ws deided to proeed with fully utomti pprohesD limiting ourselves to less rih informtion while retining preisionF TQ pigure RFUX en exmple of segmenttion templte (tted to the ertpeehwsfr Ews dt @S A D G@GAF he rtiultor tht ws the esiest to proess ws the lipsF por speeh prodution in prenhD the most signi(nt ttriutes out them re the following vluesX hether the lips re open or losedD iFeF whether there is lil ostrution of the ir pssge in the vol trtF sf they re openD how wideD iFeF how lrge is the re t the outer end of the vol trtF sf they re losedD how tightly they re pressed togetherD iFeF whether there is ny pressure uilding up ehind their losureF hether they re protruded or notD iFeF whether the vol trt is mde longer due to their lotion on the nteroposterior xisF hether they re rounded or notD iFeF the shpe of the vol trt t the outer endF ell fetures ut the lst one re re)eted in the midEsgittl frmesD nd this informtion n e relily extrted from window ontining the lipsF enother signi(nt rtiultor tht still n give us some useful informtion even without the omplete knowledge of its entire ontour is the velumF st ontrols whether the ir goes through the orl ndGor the nsl vity when produing speehX when the velum is downD the velophryngel port is open nd the ir)ow n go through the nose produing nsl soundD nd when it presses upD this pssge is loked nd the ir is ontined in the orl vityD mking the sound orlF purthermoreD the velum n ome in ontt with the tongueF rere we should mention two relevnt ples of rtiultionD the one involving the lrge ody of the velum nd the other the smller uvul dngling t the end of the ltterD nd three mnners of rtiultionX pproximnt TR 4.2. Methods @eFgF the voied lioEvelr pproximnt GwGAD fritive @eFgF the voied uvulr fritive GKGA nd stop @eFgF the voieless velr stop GkGAF hereforeD the most importnt ttriutes of the velum reX hether there is pssge etween the velum nd the phryngel wllY sf there isD how wide it is @determines the intensity of nsliztionAY here the ontt etween the velum nd the tongue is @whether it is full ontt etween the velum nd the tongue or only the uvul nd the tongue"determines the ple of rtiultionAY he temporl hrteristis of this ontt @determining the mnner of rtiultionX prolonged nrrowing of the vol trt mkes n pproximntD onstrition thtD euse of turuleneD rpidly hnges etween wide nd nrrow one or even onttD mkes for fritiveD nd prolonged ontt with susequent urst mkes stopAY he sptil rrngement of the ir)owX whether it is ompletely ostrutedD or there is onstritionD or the ir)ow is loked t the enter of the vol trt y the uvulD ut there still is pssge for it t its sidesD thus reting lterl soundF wost of these spets reD t lest to some degreeD represented in Ews dtD the limiting ftors eing the sene of other plnes ut the midEsgittl one @no informtion regrding the potentil lterl wys for the ir)owA nd reltively low frme rte @lurring the frition eE tween the rtiultors togetherAF por the purposes of rtiultory speeh synthesis the ville mount of informtion should still e usefulD so we proeeded with extrting itF he proedure we followed ws result of ollortive e'ort of sonnis houros nd myselfX sonnis houros oneived the generl ide to nrrow down our view to spei( windows nd to use seed pointsD olor (lls nd ontour detetion tools to look for ontts etween the orgns in those windows @see detils elowAY then s worked ompletely on my own on produing rtiultory prmeters @piking nd developing methods for proessing ontoursD redjusting the windows for etter resultsD treting errorsD unforeseen ses nd minor noise in the imgeD giving informtive feedk from the omputtion proessD treting the lulted vlues for onsistent temporl ehviorD nlyzing nd evluting the resultsAF Extracting the articulatory parameters for the lips qiven frme window ontining the lips s in pigure RFVD (rst we disrd @iFeF (ll with lkA the leftmost white re of the imge tht orresponds to the spe outside the hed of the spekerF hen we look for seed points of the lips @pigure RFVAX the upper lip pixel is white (x, 0) pixel tht is losest to the expeted (15, 0) nd elongs to n re of t lest V pixelsY the lower lip seed ould e the (rst white pixel from the leftEottom orner of the imge rightwrdsD exept tht the (rst white pixel normlly orresponds to the white spe outside the speker9s hedD so it should e the (rst white pixel seen fter spn of lk pixels tht surround the speker9s hedD with the sme ondition on the re sizeF sf the lgorithm fils to (nd seed point for the upper lipD it is ssumed tht the window ws pled too lowD nd the proess is repeted with the window one pixel higherY the symmetri proess is done for the seed point of the lower lip nd the window shift y one pixel downF TS henD whenever there were two ontoursD we ould (nd the distne etween them @the smllest euliden distne etween their pixels orders"see iqution RFI elowAD whih served s the vlue of their opening"the seond rtiultory prmeter extrted from this windowX iqution RFQF

eucl(A, B) = (displ x (A, B)) 2 + (displ y (A, B)) 2 , @RFIA
where displ axis is displement long given xisX

displ axis (A, B) = |A axis -B axis | if A axis = B axis |A axis -B axis | -1 otherwise @RFPA lsdist = min(eucl(U, L)),
U eing pixel on the order of the upper lip ontourD L the lower lip @RFQA pigures RFIH!RFIH mrks how the distne etween the lips is omputedF he mening of hving distne etween pixel borders rther thn pixel oordintes themE selves n e shown on the following exmpleX if the lowest point of the upper lip is (10, 10)D nd the highest point of the lower lip is (10, 11)D the trditionl euliden distne etween them will e 1D while the ontours re oviously in onttD shring ommon pixel orderF rving distnes in pixels is solution tht is onvenient for omputtionsY nturllyD ll the vlues n e trnslted k into mmD sine the distne etween ny two djent pixel enters @iFeF the vlue of I pixel in our implementtionA is 1.412 mmF es for the se when the lips were in onttD the lsdist distne ws set to HD nd we lulted the surfe of their ontt lscont s follows"see pigure RFIHeX O y , I y < 0.8 × h, h eing the height of the window @RFRA he vertil limit of the ontt points eing neessrily in the upper VH7 of the imge mens to mke sure tht the thinning of the lower lip t the ottom of the imgeD the one tht needs to e inluded in the window in se the lips open wideD does not get ounted s the ontt surfeF hen the ontt surfe ws used to divide the joint re of the lips in ontt into two seprte lips touhingD thus providing us with full lip ontours in ll sesF hese (rst three rtiultory prmeters re not independentX whenever the lips re openD the distne etween the lips lsdist is positive nd the ontt surfe lscont is zeroY other timesD when the lips re losedD the distne etween them is utomtilly zeroD nd their ontt surfe is not negtiveF hen s lulted lips protrusion vlues uplpr nd lwl p r"see pigures RFIHfX pigure RFIHX ixmples mrking how the lip prmeters re omputedF he upper lip is greenD the lower lip is orngeD the key elements re highlighted in red or lueF

Extracting the articulatory parameters for the velopharyngeal area he next step ws to extrt the rtiultory informtion of the velumF es disussed oveD the velophryngel re of the frmes hs less onstrst nd rightnessD whih mens tht smoothing nd thresholding it is less relile nd my ring in errorsX feele shpes nd outlines n get disrdedD nd spes n e reognized s tissueF fesidesD the re hs more rtiultors moving thn it ws in the se of lipsD whih mkes it more hllenging to relily reognize TV 4.2. Methods them utomtillyF pigure RFII shows wht proessed window my ontinD nd pigure RFIP shows how the presene of movement in the sequene n ring in some odd shpesF pigure RFIIX en exmple of proessed velum windowX the initil window @the tongue nd the velum in ontt to produe GkGAD the smoothed nd (ltered imgeD nd its nnottionF pigure RFIPX hue to noise nd movementD the extrted ontours my e of unpreditle shpesX sequene of frmes where the loss of ontt etween the tongue @greenA nd the velum @violet rther thn lue elsewhere in the hpterD to indite tht it is joint ontour of the velum nd the phryngel wll tht lone is mrked orngeA retes highly irregulr ontour of the tongue @one frme efore the lstAF he ontour nnottion error in the middle imge @ssigning the lel tongue to the joint tongueEvelum ontour pst the point of onttAD howeverD is not relted to this disformtion ut rther to the filure to reognize the two rtiultors s eing in onttF he tongueEvelum distne rising from this inorret nnottion ws disrded t the susequent onsisteny veri(tion stgeF he iggest shpe is the tongueD ut even its reognition is not strightforwrdD sine the lyout of the musle (ers retes drker res in the ody of the tongue tht get proessed s vitiesF heir numerD ext lotion nd size n vry etween one imge nd notherF et the right we hve the order of the vol trt tht is the losest to the kX the velophryngel wllF hepending on the windowD prtly on the movement of the speker ut espeilly on whether the nose tip ws deteted orretlyD the window my or my not lso ontin some of the vities ehind it tht tully re (lled with the vertereF his sets up hoie to e mde when setting the limits to the windowX inluding more spe to the right TW will gurntee tht the entirety of the phryngel wll is present in the window t the expense of introduing there lot more irregulr nd unstle shpes tht will require proessingD while mking sure tht the vertere do not get into the window he mjor fous of our ttentionD the velumD is n rtiultor whose shpe is prtiulrly prolemti to pture utomtillyF st my pper untthed to the more front re of the vol trtY its shpeD due to it eing very thin nd moileD n e distortedD whih n e even more ggrvted y the hevy proessing of the imgeF es for the window oundriesD depending on the speker9s hed preise positionD sometimes shpes from the nsl vity re fully visile in the originl lrge imgeD nd sometimes notF goupled with the ft tht the velum my pper disjoint from the more frontl prt of the vol trtD it de(nitely rules out hving sfety mrgin in the window to mke sure tht the entirety of the velum is ontined in itD sine the shpes in the nsl vity would eome indistinguishle from the shpe of the velumF es the velum n urlD thisD in turnD mens tht it my ross the upper oundry of the window multiple times @potentillyD enter the windowD leve it nd reenterAF his explins why it my e di0ult to reognize nd proess the outlines of these rtiulE tors even when they re lerly prt s in pigure RFIQF he tsk eomes even more di0ult when they ome into onttF hue to the velum urlingD the velum nd the tongue n touh multiple times within the sme imgeD reting very irregulr outlinesF st the onttD loser to the lipsD there my or my not e spe etween the velum nd the tongueY sme goes for the spe efore the ontt etween the tongue nd the phryngel wllD loser to the glottisF e sitution when ll three rtiultors re in ontt n pper in mny di'erent wysD with spes nd notF pigure RFIQX en exmple of utomtilly extrted ontours in the window of the velumF qreen stnds for the tongueD lue for the velumD nd ornge for the phryngel wllF pinllyD ontour detetionD suh s the hosen orderEfollowing lgorithm of + VSD is relile when the entirety of the outline is ontined within the imgeF hen prt of n ojet touhes the order multiple timesD oupled with inonsistenies suh s the velum ppering to hng in UH 4.2. Methods the ir it eomes slightly unpreditle whether shpe will e onsidered prt of one ojet or multiple onesF oD s hd to rek down ll lrge orderErossing ontours into smll hunks tht entered the imge on the imge orderD followed oundry nd left the imgeF his ws neessry to void flsely joined orgns when they re notD tullyD in onttD ut reted mny more ndidte ontours to identify s the velumD the tongue or the phryngel wllF pirstD s found seed points of the three rtiultors in questionX he initil seed point of the velum ws the losest white pixel @iFeF with tissueA to the upper left orner of the imge s long s it elonged to su0iently lrge white reD nd then it ws shifted t most U pixels down s long s the olor styed whiteF he initil seed point of the tongue ws the losest white pixel to the middle point of the left order of the imge s long s it elonged to su0iently lrge white reD nd then it ws shifted up to T pixels to the right s long s the olor styed whiteF he initil seed point of the phryngel wll ws the losest white pixel to the down right orner of the imge if it elonged to lrge enough n reD nd then if ws shifted to the left s muh s possile to (nd the order of the phryngel wllD the mximum shift eing up to the middle of the imgeF xo diretion of serh ws preferredD s in se the entirety of the wll ws in the imgeD it ws possile to lso (nd prts of spes orresponding to the vertereD nd they sometimes ould pper disjoint from the rest of the wllF sf the seeds were found outside the res where s roughly expeted them to eD the orreE sponding imges were sved s potentilly prolemtiF henD to tret ses of onttD the following proedure ws used to identify referene points for spes etween the orgnsX he point etween the tongue nd the velumX the (rst lk pixel enountered on the line from the tongue seed point to the velum oneY if it filsD try IS pixels to the rightF his wy the point should e loser to the tongue surfe thn to the velumF xormlly the tongue seed point is high enough tht there re no lk res reted y the musle (ers on the wy from this point to its velum ounterprtY he point etween the tongue nd the phryngel wllX the lst pixel in the (rst spn of lk pixels enountered on the line from the phryngel wll seed point to the tongue oneF he point etween the velum nd the phryngel wllX (rstD new phryngel wll point is found s the leftmost white pixel shring the sme white re with the phryngel wll seed point when looking from (w -1, 8)Y thenD the point is the lst pixel in the (rst spn of lk pixels enountered on the line from the new phryngel wll seed point to the velum one"soD it would e loser to the velum thn to the wllD ut will not enter ny smll pokets reted y the velum9s urlingF sn similr fshionD the newly onstruted upper phryngel wll seed point ould serve to (nd point etween the tongue nd the phryngel wll tht would e in the upper prt of the imgeD loser to the velumF UI henD using the orgn seed pointsD s identi(ed orgn ontts y oloring the white re round one seed point nd heking whether the other seed point in question hnged its olor tooF sf noD the rtiultors were not in onttF sf yesD there were two possiilitiesX either there is true ontt etween themD or oth of them touh the third rtiultorD ut not eh otherF sn the ltter seD s identi(ed suEwindow with the potentil ple of ontt nd repeted the proess of oloring point from one rtiultor nd heking the olor of the point of the otherF sf it ppered tht the tongue ws in ontt with the phryngel wll nd the phryngel wll seed point ws not t the very right of the imgeD most likely it ws n error of the phryngel wll seed identi(tion"see pigure RFIRF pigure RFIRX en exmple of n erroneous phryngel wll seed using the tongue to e reogE nized s in ontt with the phryngel wllX initil wrong guess @leftAD shifted orret window @rightAF he mrked points re the seed points for the tongueD the phryngel wll nd the velumF hen s suppressed ll shpes tht did not ontin single seed point nd pplied the sme orderEfollowing lgorithm of + VS to (nd the ontours present in the imgeF s ut them so tht every ontour would either e entirely ontined within the imge proper or egin t the oundryD go through the imge nd end somewhere else t the oundryF here typilly were numerous ontours with irregulr shpesF sf n orgn ws known to stnd prt from the othersD s ws looking for one ontour for itF sf n orgn ws known to e in ontt with nother oneD s ws looking for two ontoursX one for the prt efore the ple of ontt @this ple hving n a priori unknown lotionAD the other for the prt pst the ple of onttD loser to the lipsF por exmpleD if the tongue nd the phryngel wll re in onttD the (rst ontour would inlude the lower prt of the phryngel wllD ner the glottisD up until the ontt pointD followed y the lower prt of the k of the tongueD nd the seond ontour would inlude ll the upper prt of the phryngel wll until the ple of ontt elowD followed y the outline of tongue dorsumF sf there ws hin of rtiultory onttsD for exmpleD the tongue touhing the velum nd UP 4.2. Methods the velum touhing the phryngel wllD ut not the tongue touhing the phryngel wll @s in pigure RFII when produing Gk(u)GAD s looked for the ontour of the hinX tongueEvelumEwllD velumEwllEtongueD velumEtongueEwllF sf ll orgns were touhing eh other @s in pigure RFISAD there ws nothing to ompute s tking note of the presene of ontt ws enoughF pigure RFISX ell rtiultors in ontt with eh other"no need to lulte the distnesF flue mrks the ontour tht ws leled s the velumY purple the ontour tht ws leled s the velum nd the phryngel wllY ottle greenD the tongue nd the phryngel wllF pirst s mde ll possile guesses for whih orgn eh ontour ould eF Special case sn the speil se when rtiultor ws in ontt with D with D ut not with @see pigure RFITAD s serhed for the joint EE hin ontour s the ontour tht pssed lose to ll referene points @the upper prt of the phryngel wllD the velum seed point nd referene point for the spe next to the k of the tongue nd the end of the velumA or ws so lrge tht it oupied more thn 757 of the imge nd then roke down tht hin into three ontoursX the tongueD the velum nd the phryngel wllF he splitting points were found s the extreme points of the ontour nd the losest points to the referene nd seed pointsF xo other se needed the ontours to e splitF Pharyngeal wall he ontour of the phryngel wllD when it is not in ontt with ny other orgnD needed to e su0iently rightD su0iently tllD nd its ounding retngulr needed to ontin the phryngel wll seed @ginD s used eucl distne from iqution RFIAF sf the phryngel wll ws in ontt with the tongueD it hd two optionsF yne option for it ws to e the ontour whose ounding retngulr ontined the identiE (ed point in the spe etween the phryngel wll nd the tongue @the down prtD loser to the glottisD efore the ontt pointAF his ontour lso needed to e ontined in the lower right prt of the imgeF UQ pigure RFITX essigning ontours in the speil se of two rtiultory pirs in onttD ut not the third pirF o extrt orret distne etween the two rtiultors tht do not touhD it is neessry to ut the joint ontour in three prtsF es usul in this hpterD green stnds for wht ws leled s the tongueD lue s the velum nd ornge s the phryngel wllF he other option ws to e the ontour whose ounding retngulr ontined the sme point s ove ut shifted QH pixels up nd S pixels left @the upper prt of the wll from the velophryngel port downwrds to the ontt point nd the rest of the tongue from thereAF sf the velum touhed the phryngel wllD the ontour of interest ould e the ontour whose ounding retngulr ontined the spe point etween the velum nd the phryngel wll @see oveAF Tongue he ontour of the tongueD when it ws not in ontt with ny other orgnD needed to enompss lrge enough reD nd tht re hd to ontin the seed point of the tongueF sfD howeverD the tongue touhed the velumD there were three options to onsiderF sf the velum ly on the tongue with no spe etween themD then their joint ontour ws the one tht we would hve found for the tongue hd it een stndloneF sf the ontour9s ounding retngulr ontined the seed point of the spe etween the velum nd the tongueD it ould e the joint velumEtongue ontour pst the ontt pointD loser to the lipsF o qulifyD this ontour lso needed not to streth too fr to the rightF sf the ontour9s ounding retngulr ontined the seed point of the spe etween the tongue nd the upper prt of the phryngel wllD it ws the joint velumEtongue ontour efore the ontt pointD loser to the glottisF o qulifyD this ontour lso needed not to hve ny points lose to the left order of the imgeF UR Velum sf the velum did not touh ny other rtiultorD ny ontour tht ws su0iently lrge nd whose ounding retngulr ontined the velum seed point ws to e leled s the velumF henD for every pitureD the set of ontours with their potentil guesses were exmined together to mke sure they did not ontrdit eh otherX every rtiultor hd to hve ontourX one nd only one for n isolted rtiultorD t lest one for n rtiultor in onttY dditionllyD ontour ould hve severl lels only for rtiultors in onttF sf the leling ws found to e inonsistentD s proeeded to ressigning lels until they eme onsistent or the numer of repeted ttempts to do tht got lrger thn the numer of rtiultors in question @threeAF o (nd onsistent lelingD s heked rtiultors one y oneD ll the while keeping trk of the ontours whose leling hd lredy een determined firly ertinlyF sf there ws no inonsisteny found for given rtiultorD its ontour or ontours got preliminrily pproved to e leled s the rtiultor ndD when pplileD the rtiultor it touhedF sf there ws prolemD the lgorithm tried to (x it in the wys desried elowF sf n rtiultor ws not ssoited to ny ontourD (rst s found the ontour tht ws the losest to the rtiultor9s seedF his ontour ould not e smll nd loted in the down left orner of the imge euse tht ws the hrteristis of lk res flsely reognized s empty spes in the tongueF he sme pproh ws ttempted if n rtiultor ws touhing nother one ut hd only one ontourF sf it did not yield new ontour to ssoite to the rtiultor in questionD it should theoretilly hve hppened only for the joint prt of the tongue nd the velum loser to the glottisF o identify thtD s onsidered the ontour losest to the seed point etween the tongue nd the upper prt of the phryngel wllD sine y design it is very lose to the velumF sf n rtiultor ws not found to e in ontt with ny other ut got ssoited to multiple ontoursD the ontour tht ws the losest to the seed point of the rtiultor eme the one tht got to keep the lelF henever suh prolem (xing reted on)it with n lredy preliminrily pproved lel or the new leling turned out to e inonsistent @for exmpleD if the tongue ws not touhing ny other rtiultor nd hd its single ontourD ut then this ontour got to e ssigned to the velumAD it triggered new round of serh for onsistent leling of the ontoursD this time pplied to the new wy of leling the ontoursF sf fter this proess the leling ws still inonsistentD the entire proess of ontour serh nd leling ws repeted for the potentilly noisy ontours extrted from the originl smoothed nd (ltered imge @rther thn the imge where ll white res not ontining single rtiE ultor9s seed point were (lled with lkAF sf the leling styed inonsistent even fter thtD rtiultory prmeters extrted from the velum window were set to xxsF ytherwiseD hving mde sure tht the shpes present in the frme were ll properly leled nd the understnding of the piture seemed to e orretD s ws le to lulte the following vlues s the miniml distnes etween pixels on orders of the respetive ontours"see pigure RFIUX pigure RFIUX vtdistD vwdist nd twdist re omputed s the miniml distnes etween the respetive rtiultorsF e ompliting issue ws the se when there ws n rtiultory onttX for exmpleD the tongue nd the velum touhing @see pigure RFIVAF hen s would only hve ess to the joint ontour of oth rtiultorsD nd there will e only one miniml distne etween this joint rtiultor nd the phryngel wllD insted of twoX one for the distne etween the velum nd the phryngel wllD nd the other etween the tongue nd the phryngel wllF sn this seD s msked the imge to hve the distne to e omputed only in the pproximte expeted re @in the se of the velum nd the tongue in onttD the distne etween the tongue nd the phryngel wll needed to only e in the lower prt of the imgeD nd the one etween the velum nd the phryngel wll only in the upper prtAF henever the msk did not pture single point of the ontourD the distne ws set to xx @not numerAF hese two vlues er the piees of informtion we were serhing forX how fr the tongue is from the velum @to investigte their ehvior when produing suh sounds s rhotisA nd how open the velophryngel port is @to mesure nslityAF roweverD there is one more onstrition present in the windowX the spe etween the k of the tongue nd the phryngel wllF sn the se of prenhD this ple of rtiultion is not used onsiouslyD ut still is quite informtive for the oustis of the resulting soundD so s extrted it s well @see pigure RFIUAF twdist = min(eucl(T, W)), T for the tongueDW the phryngel wll if no ontt 0 otherwise @RFVA edditionllyD the tegoril informtionD whether there is ontt or notD ws stored s he cont vlues ould e derived without preise ontour ssignmentD so those vlues ould e set even when there ws no dist vlues villeF st should e noted tht while the rtiultory shpe hnges grdully nd smoothlyD the funtion of minimum etween two urves does not hve to e smoothD sine the points produing the miniml distne do not hve sty lose ross neighoring imgesF st ws less of n issue with the lipsD sine most of their midEsgittl ehvior n e explined in two dimensionsX x for protrusion nd y for openingF he lips do not url nd do not hve ny options for how to ome into onttF his is not the seD howeverD with the more omplex orgns in the k of the vol trtF por exmpleD pigure RFIW showses three very di'erent ses for how the tongue UU nd the velum n touhX either euse of the tongue going up while the velum is stright nd presses up t the phryngel wll @s in GkGAD or euse the velum lowers nd t lest slightly urls to touh the tongue while opening up @s in GÕGAD or euse the velum flls )t on the k of the tongueD nd the surfe of their ontt virtes in order to produe the rhoti GKGF pigure RFIWX he tongue nd the velum n ome in ontt in vrious wysD ut these on(guE rtions will led to the sme vlues tvdist = 0, tvcont = 1F he lueEgreen olor stnds for joint ontours of the velum nd the tongueD green for the tongue loneD lue for the velumD ornge for the phryngel wllF purthermoreD in the se of the lips we were lulting the surfe of the ontt etween the upper nd the lower lipsD whih gve us the possiility to di'erentite etween )eeting ontt nd the lips eing (rmly pressed togetherF ine the sme omputtions for the ontts tht n our in the window of the velum would e muh more prone to errorsD it ws deidedD s iqutions RFW!RFII showedD to only store the tegoril informtion out the presene or the sene of the ontt @see pigure RFPH to see how the two types of ontt s treted s smeAY this e'etively rules out ny wy to foresee ny hnge in the tvcontD vwcont nd twcont vluesX in one frmeD they n e 1D nd in the next one lredy 0D even though most likely these trnsitionry frmes re not ler se of eitherF Creating articulatory parameter sequences hile hving the ig dvntge of eing fully utomtiD this pproh ove hs its shortE omingsF he mjor wek point is the roustness of the lgorithmF pirstD whenever there is n error in identifying point of referene @suh s the nose tip or seed pointA used to lulte one prmeter or the otherD ll the susequent lultions eome inorretF iven when the nose tip is found orretlyD it still does not gurntee orret proessing of the ontours in the imgeD sine ontour leling relies on treting numerous sesD nd while the results show tht most of the ses were tretedD there is possiility tht some of them were notF his n use drmti errors down the pipelineX let us onsiderD for exmpleD the following sequene pigure RFPHX he rtiultory prmeters extrted from the window of the velum do not llow to trk the di'erene etween wellEplnted ontt etween the rtiultors nd )eeting oneX two exmples of the velum touhing the phryngel wllF of vwdist vluesX 6.00, 6.00, 6.71, 6.09F hey re ll quite onsistentF sf we dd the next vlue tht ws lulted to e 18.00D it will rete drmti e'et in the estimtion of the derivtive nd will e piked up y the mhine lerning lgorithmF e possile solution to tht is to ompre the dynmis of the prmeter vlues in (xed numer of djent imges nd keep out those vlues tht were proly wrongF husD hving extrted the vlues lsdist @iqution RFQAD lscont @iqution RFRAD uplpr nd lwlpr @iqution RFSA from the window of the lips nd the vlues vtdist @iquE tion RFTAD vwdist @iqution RFUAD twdist @iqution RFVA nd vtcont @iqution RFWAD vwcont @iqution RFIHAD twcont @iqution RFIIA from the window of the velumD if the window of given frme did not produe vlues onsistent with the reent historyD its group of prmeters were repled with xxsF he ext hek for onsisteny ws s followsF he size of the history ws hosen through nlyzing the intervls of erroneously proessed imges nd onluding tht it rrely ws more thn three imges t one nd hving found no instnes of more thn four imgesF husD the vlues hd to e onsistent with their history of hist = 4 most reent imgesD whih orresponds to looking k y pproximtely UQ ms nd ounting for WI ms in totlF sf there ws not enough reent relile history in the sequene yetD s heked whether the lulted prmeter t lest fell into the intervl of its resonle vluesF sf yesD it ould e storedD nd if notD if it ws the (rst vlue fter sequene of xxsD it ws repled y the mximlly eptle vlueD nd otherwise it ws disrdedF hen there were enough reent vluesD s ould nlyze the dynmis of their ehviorF por this nlysis to e informtiveD the history needed to e onsistentX to qulifyD ll vlues ould not ontin ny outliersD whih mens tht they needed to sty lose enough to their men over the ourse of hist frmesF he following two prgrphs re going to quntify this enoughF UW st ws surmised tht the informtion out the presene or sene of ontt ws more relile thn the vlues lulted s distne etween two potentilly mistken urvesF his led me to empirilly set two thresholds for the di'erene etween the urrent vlue nd its men over the ourse of the reent historyX weker thresholdD T l w = 2.5 for the lips nd T v w = 3.5 for the velumD nd stronger thresholdD T l s = 3.75 for the lips nd T v s = 6.3 for the velumF he motivtion for the ext vlues ws rought y the rnge of vlues for orret prmeter sequene nd how muh it should e le to hnge over the (xed numer hist of frmesF husD hunk of hist reent prmeter vlues ws onsidered s relile enough to serve s ground for keeping or disrding the next vlue if ll its present elements @iFeF ll elements ut xxsA did not devite from their men vlue y more thn T w @T l w or T v w D depending on whether the rtiultor ws studied in the window of the lips or of the velumAF rving ess to relile verge vlue over the ourse of the lst hist frmesD if the hnge etween the verge nd the new vlue implied no hnge in how we onsidered the rtiultors in ontt in the frme or notD it ws ompred to T w D nd if it didD to T s F e hnge greter thn the threshold vlue drove the new vlue out of onsidertionD nd we mrked it s implusile nd disrded itF henever window hd even single vlue tht ws disrded s implusileD either due to too drsti di'erene from the pst verge vlue or euse of going eyond the expeted limitsD ll other prmeters extrted from this window were lso repled y xxsD sine it ws likely tht wht ws hppening in the frme ws proessed inorretly @inorret seed pointsD inorret leling of the ontoursD etFA ine fterwrds s ws going to pply interpoltionD s lso needed to set some nonExx vlues t the oundries of the sequeneF sf the prmeter vlues were inresing from the (rst nonExx vlue in the sequeneD s set the sequene to egin with the miniml meningful vlue for this prmeterY if the vlues were deresingD with the mximl oneF he se of the end of the sequene went in reverseX if t the end the vlues were deresingD s dded the miniml vlueD nd if they were inresingD the mximl oneF sf there ws no dynmi ner the sequene oundry @stle vluesD no inrese or dereseAD the sme nonExx vlue ws opied to the oundryF efter ll these steps s (nlly otined sequene of rtiultory prmetersD vlue or xx per eh frme per eh rtiultory prmeterF ueeping in only the vlues the lgorithm ws on(dent out ssured tht we would not dedue ny errti trnsitions from one frme to notherF hus s ws le to remove xxs nd reEestimte them through upsmpling the signl to mth the ousti rte of PHH rz @ vlue one in S msA tht ws neessry for prmetri speeh synthesisF psmpling ws done y pieewise IEd monotoni ui rermite interpoltion in order to hve smooth trnsitionsD the mgnitude of eh trnsition setion ounded y its orresponding interpoltion knotsF Verifying the articulatory parameters sn the susequent prts of the workD the phoneti lels styed ssoited to the extrted rtiultory prmetersF he lels were oftentimes ssigned not preisely in greement with wht would e the judgment of humn nnottorF ome of these impreisions ould still mintin the onsisteny etween the lel nd the rtiultory prmetersD thus not 'eting the interpretility of the resultsD nd some of them ould severely disrupt itF rving no ground truth nd hving to reson only with the lulted rtiultory prmetersD s ould not VH verify the (rst seY for the seondD s rried out the following tests on those frmes tht ould potentilly rete n inonsistenyX por every phoneti lel ssoited with neessrily open lips @vowelsAD if the lips never losed during the phoneme produtionD s leled the instne s onsistentD nd if they didD s n errorF por every phoneme tht would e impossile to produe without losing the lips @ilil stops nd nsls Gb, p, mGAD if the lips losed t lest one during the phoneme produtionD it ws mrked s onsistent lelingD nd if notD s n errorF s extrted ll instnes mrked sX e rounded vowelD tht is expeted to e rtiulted with the orners of the lips drwn together nd the lips protruded forwrdY en unrounded vowelD where the lips re not protrudedF he ssoited uplpr, lwl p r vlues were ompred s smples for eing di'erent for vowels requiring protrusion nd not @hpiroEilk9sD h9egostino9s nd enderson9s tests for heking tht the smples did not follow norml distriutionD uolmogorovEmirnov nd wnnEhitney tests to ompre the smplesAF st would e expeted tht overllD the vlues for protruded vowels should e lrgerF por ny phoneme requiring ontt etween the tongue nd the velum @GK, k, g, ñGAD if these two rtiultors me into ontt t lest one during the phoneme produtionD it ws mrked s onsistent lelingD nd if notD s n errorF por ny phoneme tht would e impossile to produe with ontt etween the tongue nd the velum @the vowels nd more frontl phonemes Gj, Z, S, 4, l, d, t, z, s, v, f, w, m, pGAD if there ws suh ontt @tvdist vlues equl to zeroD positive tvcont vluesAD it ws mrked s n errorF por ny nsl phoneme @ vowel or onsonntAD if it ws produed with losed velophE ryngel port @vwdist = 0, vwcont = 1 " ontt etween the velum nd the phryngel wllAD it ws mrked s n errorF por ny orl phoneme @ onsonnt or vowelAD if it ws produed nslized t ny moment @without ontt etween the velum nd the phryngel wllX vwdist > 0, vwcont = 0AD it ws mrked s n errorF his mens tht when the lels were found to e inonsistentD there were two possile error typesX either n rtiultory ontt tht should not hve hppened for this phoneme @this type is further referred to in (gures s uneptleD for n uneptle onttAD or n sent ontt tht should hve een there @referred to s sentAF por every errorD it ws lso importnt not only to mrk its rtio in the entire distriution of onsistent nd inonsistent phoneti lelsD ut lso to mrk how mny times it ould hve ourredD s n error ourring 25 times out of 26 times possile is very di'erent from n error ourring 25 times out of 2500F he results of ll the heks ove were ggregted y spekers @S A D S B or othA nd y speeh styles @spontneousD not spontneous or othAF VI he losure ws onsistent with the lel VVFWP7 for S A @when reking down y spontneous nd nonEspontneous speehD WIFHW7 nd VVFHS7 respetivelyAD WPFSP7 for S B @WRFQU7 nd WIFUS7D respetivelyAF sn totl it mkes in WHFVS7 ses @WPFVU7 nd WHFHP7AF he mjority of the errorsD SQFPI7D were the se of n sent ontt etween the rtiuE ltorsF ht represents PWFQI7 of ll ses where the presene of ontt ws ritilF he rest of the errorsD RTFUW7D were the se of n extr losure during phoneme tht prohiit itF ht represents IHFVH7 of the ses when this error ould hve ourredF pigures from RFPQ to sRFPR visulize the possile omprisonsF VP pigure RFPIX @gontFA ertiultory prmeter onsisteny in the originl orpusD overll pigure RFPPX @gontFA ertiultory prmeter onsisteny in the originl orpusD roken down y spekers @ontF on the next pgeA VV For extra closures, there is a greater impact of who the speaker is; for absent closures, whether it is spontaneous or non-spontaneous speech.

Considering the total distribution of error types, precision is more dened by the kind of speech rather than by the identity of the speaker. pigure RFPRX @gontFA ertiultory prmeter onsisteny in the originl orpusD roken down y spekers nd spontneityF WV es pigures from RFPQ to RFPR showD eh pir of rtiultors @the lipsD the tongue nd the velumD the phryngel wll nd the velum nd the tongue nd the phryngel wllA exhiit their own prtiulritiesF lsdist nd lscont were the pir with one of the highest preision rtesD 92.877 nd 90.717F hile the mjority of the errors were extr losures for lsdist nd sent losures for lscontD sent losures @no lip ontt when supposedly produing Gb, p, mGA ourred muh more often with respet to how often they ould hve in oth prmetersF S A ws more likely to hve extr losure errors thn S B wsF xonEspontneous speeh hd lower rte of sent losure errorsD whih ould e relted to more reful rtiultionF he lst two points re lso re)eted in the onlusion when ompring the impt of the speker identity @S A or S B A nd the kind of speeh @spontneous or notAX for extr losuresD the impt ws greter from who the speker wsY for sent losuresD whether it ws spontneous or nonEspontneous speehF gonsidering the totl distriution of error typesD preision ws more de(ned y the kind of speeh rther thn y the identity of the spekerF tvdist nd tvcont hd slightly lower preision @87.397 nd 90.287AF he rte of extr losures @iFeF hving velumEtongue ontt when produingD for exmpleD GiGA ws prevlent s n error type in oth prmetersF roweverD it ws relted to the overll numer of phonemes prohiiting suh ontt rther thn to the frequeny with whih this error ourred with respet to how mny times it ws hekedD sine sent losure errors ourred muh more often @UTFTI7 nd VUFVH7 of the ses when they ould hve ourredAF S A ws more likely to hve extr losure errors thn S B wsY in ftD they were the reson for the drop in preision for S A F hose extr losures were lso the reson why the totl preision for nonEspontneous speeh ws lower thn tht of spontneous speehF he prole explntion is proly tht in generl spontneous speeh struggled more t pturing the ontt etween the tongue nd the velumY heneD fewer extr losure errors in spontneous speeh nd fewer sent losure errors in nonEspontneous speehF enywyD the mjor de(ning ftor for the distriution of onsistent nd wrong lels ws the identity of the speker rther thn the kind of speehF vwdist nd vwcont exhiit the lowest preision out of ll pirs @86.937 nd 81.597AF hespite the ft tht there were muh fewer extr losures thn sent onttsD with respet to the numer of times tht extr losures were heked they ourred muh more frequentlyF sn vwdistD S A ws more likely to hve sent losure errors @iFeF to produe supposedly orl sounds nslizedA thn S B wsD nd this error rought down S A 9s preisionF sn vwcontD oth spekers9 preision su'ered from n inrese in sent losure errorsF pontneous speeh hd greter preision thn nonEspontneous due to the lower ounts nd lower reltive frequeny of sent losureF eginD the prole explntion is proly tht in generl spontneous speeh struggled more t pturing the ontt etween these two rtiultorsF es for the de(ning ftor for the distriution of onsistent lels nd error typesD there ws di'erene for the two prmetersX in vwdistD for oth types of errors nd the resulting preisionD the mjor de(ning ftor is the identity of the spekerD nd in vwcontD y the kind of speehF twdist nd twcont re speil se sine no losure is neessry to produe ny soundD so the only error type tht ws heked ws extr losureD nd preision went s high s 97.707 nd 98.637F here ws no onsiderle di'erene in lel onsisteny ross spekers nd spontneityF he nlysis of error ptterns showed the following prolemsX WW ime shifts nd other errors in leling @missingD extr or wrong phonemesAD ommonly using multiple errors in single (leF por exmpleD if + mens losure ndmens noneD with time shift frme sequene leled s Gb, b, b, b, i, i, i, i, i, i, i, i, iG ould produe ----+ + -------D reveling tht GbG tully strted only when the phoneti lel lredy hnged to GiG @or this ws no GbiG in the (rst pleAY snorret identi(tion of the lipsX rong nose tip identi(tion due to noise eing too strong in the imge or due to identl (ltering out of the nose reD leding the lgorithm to misple the leftmost point reognized s the speker9s feF s improved this y orreting the nose tip vlues tht get too fr from the other ones tht re oserved in the sequene @for exmpleD if ll previous frmes hd the nose tip t round (9, 20) nd the next frme me with nose tip t (20, 19)D s orreted itAY ghoosing wrong seed @ white pixel pproximtely where the lipsD the velumD the tongue or the phryngel wll re expeted to eA rought y slightly mispled window due to n unexpeted speker9s movement or noiseY yther shpes getting involvedX the tongue touhing the lips @ompromising the esE timtion whether the lips touh or not"see pigure RFPS to get n ide for how the lgorithm interpreted the upper nd lower lip thenAD or in noise nelingD the lips getting identlly merged with the re outside the vol trtY mishndling the omplited geometry of the ontours in the k of the vol trtF hen the ontt ws )eetingD the pixels where the rtiultors touhed ould sty too drk for the lgorithm to pik up on their ontt @see pigure RFPT for n exmple of the lipsAF Articulatory-acoustic and acoustic models por omprison purposesD s mde two setupsX full art nd no artX with rtiultion nd withoutD respetivelyF eoustilly signl is hrterized y sequene of fHD the smooth spetrl envelope nd periodi energyF hen they produe the following prmeters"oth for full art nd no artX IHP he mgc prmetersX welEqenerlized gepstrl oe0ientsD extrted from the signl9s spetrl envelopeD its dimensionlity of 60D nd the orresponding dmgc prmeters estiE mting the derivtive of mgcD the dimensionlity of 3 × 60 = 180F he lf0 prmetersX log of fH vlues in ses where there is voiing identi(edD (xed lrge negtive vlue otherwise"its dimensionlity of 1D nd the orresponding dlf0 for the derivtiveD 3 × 1 = 3F he bap prmetersX nd periodiitiesD the periodi energy of the signl"its dimenE sionlity of 1D nd the orresponding dbap of 3F he vuv prmetersX voied or unvoiedD oolen vlue @no derivtiveAF eousti prmeters were treted y the yvh vooderF ertiultorily @full art onlyA s enhned the signl with the upsmpled sequenes of the rtiultory prmeters lulted oveX

lsdist ∈ [0, h window ] @efore normliztionAD lscont ∈ [0, w window ]D uplprotr ∈ [0, w window ]D lwlprotr ∈ [0, w window ]D tvdist ∈ [0, h 2 window + w 2 window ]D tvcont ∈ [0, 1]D vwdist ∈ [0, h 2 window + w 2 window ]D vwcont ∈ [0, 1]D twdist ∈ [0, h 2 window + w 2 window ]D twcont ∈ [0, 1]F
me s the durtion modelD the network setup strted o' the stndrd werlin reipeX feedEforwrd network with 6 lyersD eh lyer with hyperoli tngent tivtion @tanhAF he th size ws lrger thn in the durtion modelD 256D sine the ousti serh spe onstituted more prmeters nd therefore needed to ene(t from more urte grdient omputtionF he lerning rte ws 0.002 with n exponentil deyY the numer of trining epohs ws set to 25F he output of the network ws set of prmetersX the ousti prmeters used to generte udio @mgcD lf0D bapA nd ll the rtiultory prmetersF IHQ 4.3 Evaluation 4.3.1 Evaluation components and criteria he system is joint rtiultory nd ousti synthesizerY heneD it should e evluted s suhF pirstD s would like to know how the dding of the rtiultion in)uened the resulting qulity of speehX intelligiility nd nturlnessF ithout going into detils there ould e three senriosX @A the qulity improves euse rtiultion is helpfulY @A the qulity redues euse rtiultory informtion does not ehve onsistently with the ousti nd linguisti trining dtY @A the qulity stys the smeF et loser srutinyD the results ould e mixedX n improvement in some spets while deteriortion in othersF eondD s needed to go deeper into the developmentGtest evlution nd evlute spei(E lly the ehvior of the rtiultory prmetersF he synthesized sequenes should sty interE pretleD whih is not gurnteed onsidering the network does not hve the knowledge out the mening of eh rtiultor9s vluesD nd they should not devite fr from wht we oserve in the orpusD whih is not gurnteed either s the network ould try to keep the ost funtion low y working primrily on the ousti prmeters nd lrgely disregrding the rtiultory onesF

Evaluation data and methods

Large-scale articulatory-acoustic evaluation he (rst dtset ws the rest of the wgyII orpusX 281 sentenes tht were not inluded in the originl orpusF hey were synthesized oth within no art nd full art setupsF heir durtion models were nturlly the smeF ht ws di'erent ws their ousti modelsD whih in the se of full art ws n rtiultoryEousti modelF pirstD the results were ompred ojetivelyX with wgh @men melEepstrum distortion etween the generted prmeter sequene nd the trget oneAD fe @nd periodiity preE dition errorAD nd three mesures for pHX wi @root men squre errorAD gy @orreltion oe0ientA nd G @frmeElevel voiedGunvoied errorAF hen they were evluted pereptullyF es for rtiultionD sine no art did not tret itD it ws not possile to ompre it for the two setupsF roweverD one of my evlution riteri ws to keep the generted rtiultory prmeter sequenes interpretleD nd the numer of sentenes ws high enough to provide meningful omprison of prmeter interpretility on the generted set to the one tht ws in the originl orpusF One-sentence-out articulatory evaluation o evlute spei(lly the ehvior of the generted rtiultory prmeter sequenesD s trined ten instnes of the model with the sme input dtD ut only one sentene tken out @iFeF ll smples tht ontined this senteneAF he sentenes were hosen rndomly ording to the following riteriX IHR st needed to hve no fewer thn 3 instnes in the dtse of eh speker nd no more thn 10F hose instnes were found s sentenes with trd similrity tHI greter thn 0.6Y st ould not hve dislned representtion etween the spekersX the numers of its ourrenes in the sets of S A nd S B ould not di'er y more thn 407Y st ould not e shorter thn the shortest sentene tht mde sense to nlyzeX sl ps ml4 @thus ruling outD for exmpleD /aBa/ nd /ara/AY st ould not e only prt of n tul senteneD strting in the middle of it due to the impreise orpus split into sentenes or ending midwy or with the speker9s hesittionF hen the generted sequenes were ompred to the originl ones with dynmi time wrpE ingD or hF issentilly it onsiders the soure signl s one tht needs to e morphed into the trget signl t the lowest expenseF qiven grph of pttern vlues over timeD it uses dynmi progrmming to (nd the shortest pth in this grphF he (nl ost of this pth n serve s mesure of similrity etween two temporl ptternsF pinllyD s ompred the rtes of interpretility with respet to the phoneti lel of the originl nd the generted sequenesF

Evaluation results

Synthesis example

vet us onsider the synthesis of fonjour4 @Fr. rello4AF sts phonetized form is GbÕZuKGF pigure RFPU shows the spetrogrm of the generted signl with the voie of S A F his sound ws generted ompnied y the sequenes of eh of the rtiultory prmE etersF vip opening nd ontt prmetersD lsdist nd lscont @pigure RFPVAD show tht the lips orretly losed to produe GbG @losure is mrked with redA nd orretly remined open throughout the rest of the utterneF vip protrusion prmeters uplprotr nd lwlprotr @pigure RFPWA were generted with the lips eing more protruded on the vowels GÕG nd GuG nd their neighorhoodD whih is orretF ongueEvelum distne nd ontt prmeters tvdist nd tvcont @pigure RFQHA orE retly showsD s the velum lowers to produe the nsl GÕGD the derese in the tongueEvelum distneY there is no ontt etween the velum nd the tongue to produe the rhoti GKGD ut this is something we oserve in the originl dt s well @pigure RFQIAD sine the lgorithm reogE nizes the spe reted y the velr nd uvulr virtion s n sene of onttF purthermoreD the inrese of the tvcont vlues right efore the end of the utterne ould indite tht the prolem ws us getting too open distne t GuG nd hving no time to lose it for GKG without drmti e'et on the derivtiveF elumEphryngel wll distne nd ontt prmeters vwdist nd vwcont @pigE ure RFQPA generlly exhiit orret division etween orl nd nsl sounds over the ourse of ll phonemes ut GbG in the se of vwdistX in this on(gurtionD GbG would e produed s IHS pigure RFPUX he full art synthesis of onjour4 GbÕZuKG with the voie of S A F he trnsition of formnts orresponds to the hnge of phonemes in produtionF GmGF his should e used y the sme e'et s in tvdist nd tvcontX the silent posiE tion for the velum is to e loweredD nd the velum ws generted to hve suh high vwdist tht the vlue did not hve the time to get s low s neessry for GbG without 'eting the derivtive @while it should e noted tht the derese from pau to GbG is quite steepAD nd then it ws lredy time to prepre for the nsl GÕG oming nextF ytherwiseD oth GbG nd the (nl phonemes orretly register s orl soundsF ongueEphryngel wll distne nd ontt prmeters twdist nd twcont @pigE ure RFQQA show orret sene of the ontt throughout the entire sequeneF Large-scale articulatory-acoustic evaluation es mentioned oveD two setupsD no art nd full artD were trinedF hey were ojetively evluted on their ommon prtD oustis"see le RFQF por the refereneD the durtion model9s evlution @in oth setupsA ws s followsX

S

A has an increase the velum and the tongue touching during phonemes that are impossible to produce like that. st ws deided ginst rrying out multiEuser pereptul test euse oth these ousti evlution vlues nd n informl preliminry test suggested tht while not eing identilD the generted udio smples were indistinguishle to the humn erF One-sentence-out articulatory evaluation he min instrument to ompre how well the synthesized rtiultory sequenes modeled the originl ones ws the distne indued y dynmi time wrpingF le RFR ompres the h distne etween the generted rtiultory sequenes @10 per eh prmeter per spekerA nd their orresponding sequenes from eh speker9s prt of the orpus tht were left out in the model9s dtseF hile in generl the distnes etween the synthesized sequenes nd the originl ones re higher thn those etween the originl ones themselvesD sine the vrine is highD they still fll within the rnge of the eptleF pigures RFQT show exmples of ligned rtiultory prmeter sequenesF rmeter peker pigure RFQTX h lignment etween the originl @oveA nd generted @elowA rtiultory prmeter sequenes"ontF on the next pgeF pigure RFQTX @gontFA h lignment etween the losest mthing originl @oveA nd generted @elowA rtiultory prmeter sequenes"ontF on the next pgeF

M synth to corp [dtw] ± σ synth to corp [dtw] M intracorp [dtw] ± σ intracorp [

IPS

IPT 4.3. Evaluation (c) v_w_dist, S
A ,Il a pourri, DTW = 9.07. The generated sequence is not as stable (uctuations around 0 while the original sequence was just constant 0), but its approximate values and the timing are correct.

pigure RFQTX @gontFA h lignment etween the losest mthing originl @oveA nd generted @elowA rtiultory prmeter sequenes"ontF on the next pgeF IPU 3.43. The generated sequence is not as stable, but its approximate value and the timing are correct.

pigure RFQTX @gontFA h lignment etween the losest mthing originl @oveA nd generted @elowA rtiultory prmeter sequenesF IPW edditionllyD s ompred phoneti lel onsisteny with the prmeter lels for the generE ted sequenes nd their originl ounterprts tken out from the orpusF he onlusion is the sme s in the lrgeEsle evlutionX tht it is more omplited to generte onttD therefore the numer of uneptle losure drops in synthesisD nd the rte of sent losures inresesF es for the frequenies of errorsD they ome out to e relted to wht ws in the originl dt not used to trin the modelF IQH pigure RFQUX @gontFA ertiultory prmeter onsisteny ompred on the originl sentenes tht were tken out nd the generted onesD roken down y spekers"ontF on the next pgeF pigure RFQUX @gontFA ertiultory prmeter onsisteny ompred on the originl sentenes tht were tken out nd the generted onesD roken down y spekers"ontF on the next pgeF IQQ pigure RFQUX @gontFA ertiultory prmeter onsisteny ompred on the originl sentenes tht were tken out nd the generted onesD roken down y spekersF IQR pinllyD s looked into lip protrusion in the generted sequenes nd in the originl dtF eginD the upper lip prmeter uplprotr ws etter t di'erentiting etween protruded nd nonEprotruded vowelsY when verifying whether the vlues orresponding to protruded vowels were on verge lrger thn those to nonEprotruded onesD this ondition ws true 61.407 of the times in synthesis nd 59.657 in the orresponding originl dtF here re few experiments nd improvements tht ould still help us gin some insight reE grding this line of reserhF por instneD it would e useful to study the ext ehvior of the distortion nd error funE tionsD espeilly of the rtiultory prmeters errorD over the time of ousti model triningF ine the full art setup introdued new prmetersD it ould e the se tht it would e justi(ed to modify the numer of trining epohs or to dpt the lyers in the networkF henD the study would e muh more omplete with experimenting with other types of neurl networksD espeilly vws nd fvws known to exel in speeh synthesis thnks to their improved mngement of temporl reltionsD nd djusting their prmetersF elsoD it would mke sense to proess the generted rtiultory prmeters t the (nl lyer so tht they sty within the interpretle rngesD for exmpleD without rehing negtive vluesF prom the purely speeh prodution perspetiveD it is lso interesting to study the ontriution of eh of the prmetersF por exmpleD the pir of the tongue nd the phryngel wll is not prt of the ple of rtiultion of ny phonemes of prenhD nd throughout the nlysis it oftentimes ppered seondry despite its lluded ontriution to estimting the kness of the tongueF his rings out the question of whether this piee of informtion ws useful to the network or notF st ould turn outD for exmpleD tht the lips re muh more meningful in the resulting sound genertion thn the distne etween the tongue nd the velumY this is n open reserh question tht merits eing studiedF gontinuing the line of thought of seprting di'erent ontriutionsD we sw some di'erenes etween spontneous nd nonEspontneous speehF hile spontneous speeh is more di0ult to lign due to more irregulrities with respet to the stndrd refully rtiulted prodution of the delivered phrsesD it nnot e denied tht it is immensely more nturlD for exmpleD through exhiiting the speker9s unertinty tht is ssoited s n ttriute of nturl speeh nd n e piked up y rtiultory speeh synthesis fp + F sn order to lern more out onstruting rtiultory speeh dtses in the futureD it would e useful to evlute the importne of spontneous speehF yne of the mjor issues of the study ws the prolem of phoneti lignmentF ith the reording eing quite noisy nd the video eing in synhrony with the udioD it is of no dout tht otining orret orrespondene etween the timing of the udio reordingD the ws frme nd the linguisti nnottion lel is tll orderF xeverthelessD s it ws ommented upon in the setion out the nlysis of phoneti lel inonsisteniesD oftentimes rtiultion mkes it quite ler tht there is time shift @the exmple of Gb, b,b,b,i,i,i,i,i,i,i,i,where + stnds for the lips losureAF st would e expeted to improve phoneti lignment drmtilly if we integrted rtiultory informtion in the lignment proessF rving seprte len reording of prompted speehD devoid of the noise of the ws mhineD nd trnsferring its phoneti leling onto our denoised smplesD ould lso e of helpD provided tht the noise does not hinder the lignment etween the two versions of the utterneF pinllyD very promising"nd more mitious"diretion of followEup reserh would e to represent rtiultory informtion of the whole vol trt rther thn few seleted resF he hllenge in tht is tht we would hve most likely to do without preise outlines of the rtiultorsD sine they ould e too prone to mistkesF por thisD potentil solution ould e IQT to represent the vol trt on(gurtion with methods suh s vxIV nd then to enode it with sttistil methods suh s ge or deep lerning methods suh s utoenodersF here isD howeverD growing ody of reserh to trk full rtiultor outlines with vrious mount of detil with supervisedD semiEsupervised nd unsupervised mhine lerningD suh s qr + IWD so eventully it should e possile to hve preisely segmented imges s wellF IQU Static targets versus running speech for articulatory speech synthesis sn the previous hptersD we hve seen wht n e done for rtiultory speeh synthesis using stti or dynmi wsD with quite di'erent methodsF he di'erene etween the two dt types is inherentF yn the one hndD ws n pture the position of vol trt tht ws held stle over the quisition time @typilly dozen or more seondsAF he threeEdimensionl spe is represented s numer of imgesD eh ollpsing together the informtion of its respetive slieF his wy we n otin omprehensive piture of the vol trt with high resolutionD ut due to the extended quisition timeD this piture is frozenF yn the other hndD the protool of Ews selets only one piee of Qh volumeY for speeh prodution reserhD typilly the midEsgittl oneF st ptures the tissues within tht slie in rel time vv + IWD whih enles us to nlyze rpidEped speeh movementsF he speeh oserved with suh method is unrestrited nd therefore highly nturlD llowing for deep understnding of the dynmis of the rtiultors x + IRD xITD + IV ndD s ghpter R showedD for speeh synthesis sed on themF roweverD it nnot e denied tht in the ttempt to gin enough temporl overge in Ews we lose lot of imge shrpness nd lrityF sf the slie is not thin enoughD the intrite geometry of the rtiultors gets projeted on single plne @there re phonemes with quite omplex threeEdimensionl ehviorD suh s the lterl GlGAY if the speker moves too fstD no position will e mnifested for long enough to e ptured y the mhine @sine eh elementry quisition lsts 2 or 3 ms nd the lst rdius gets quired round 18 ms fter the (rstAF foth of these points n result in ghosting @for exmpleD the presene of two outlines of the tongue tipD whih is n espeilly rpid rtiultorAD imge lurring or other rtiftsD susequently 'eting the nlysis nd rendering imge segmenttion espeilly di0ultF es shown in the previous hptersD our se ws not n exeption to the generl ruleF E le SFI summrized tht while eing n optiml solution for pturing onsiously ontrolled vol trt positions with high resolutionD stti ws ptures hd the disdvntge of demonE strting unrelistiD proly overrtiulted vol trt shpesF he speker hd pprent troule voiding exessive nsliztion s there is no onsious ontrol of the velum when it is xo @entirelyA onsious ontrol over rtiultion le SFIX edvntges nd disdvntges in regulr @sttiA ws nd EwsD summrizedF not tively used in speeh produtionY lsoD t showing n pproprite ontext when rtiultE ing prtiulr phonemes @for exmpleD not rising the tongue dorsum enough for ntiipting GiG"see pigF SFID"while GiG is dened s lose front unrounded vowelY using this trget to guide n rtiultory speeh synthesizer would nturlly led to misinterprettion of ntiE iptory ortiultion for GiGD mking it more similr to the e'ets of GaGD n open entrl unrounded vowel in prenhAF here is evidene tht phonemes tht re intrinsilly dynmi in rtiultionD suh s liquidsD were lso misrepresented y the stti protool of ws quisitionsD leding gin to unrelisti shpes viIVF pigure SFIX tti @leftA nd typil dynmi @rightA reording of Gp(i)G rtiultion y speker S A F xote how the tongue dorsum is not rised high enough in the stti dtsetY this tongue position would e muh etter explined y ntiipting more open sound like GaGF his ould hve een helped with speil voie nd ent trining preliminry to the quisitionsF his rings us to the need to uild ridge etween the two diretions of workF wore preise detils follow elowF 5.1 Objectives sn our two pprohes to treting the mehnis of speeh prodution D the pivoting point ws mnifesting or irumventing the notion of n elementry unit of rtiultionF ithin the stti pprohD we expliitly uilt the system round some key rtiultory positions reveling ortiultion nd serving s rtiultory trgetsD every rtiultory motion generted s trnsition from the previous trget to the next in given mount of timeF IRH 5.1. Objectives es for the dynmi oneD there rtiultory prmeter sequenes were generted similrly to ousti prmetersD sttistilly driven y the reltion identi(ed in the dt etween linguisti spei(tion nd rtiultory prmeter vluesF st does imply tht the speker9s intended utE terne guides their rtiultionD ut does not speify howF eny generted vlue is result of the ft tht the system hs hd enough exmples of when this linguisti spei(tion ws seen together with this vlue nd its derivtive estimtionY there is no strtegy involvedF o further illustrte the lst pointD let us explore the following exmpleX if we look t the prmeter of lips opening in the synthesis of the syllle GbaGD the prodution of the stop GG n only hppen if the lips ollide with foreD involving the musles of the lipsD the pressure in the orl vity uilds up until the ertin point t whih the lips prtD oth euse of the ir foring them to do so nd euse of the jw movement tht proeeds to the open vowel GaGF sf the ontt is )eetingD with no foreD then wht will e produed is ilil )pX G baGY if there is no ontt t llD just the lips oming loser to eh otherD it is ilil fritive @or pproximntAX GBaGY if there is onttD ut it does not get relesed during GaGD no GaG will e produedD sine GaG is vowel nd not n ostruentF oD whether the lips lose or notD is something tht de(nes whether the syntheti sound will e reognized s wht ws intended or notF his mens tht from the rtiultory perspetive lips losure is less of question of preision @whether the opening is P m 2 or PFP m 2 D or HFH m 2 versus HFP m 2 A nd rther whether it is present or notF sn other wordsD the losure is tegoril vlueD whih the resulting speeh intelligiility entirely hinges uponF hus it would idelly require some kind of plnningD nd tht would led us to go k to modeling the orgniztion of rtiultory movementsF his motivtes us to see whether we n (nd ny tre of suh ortiultionEwre trgets s we hd in the stti se present in the dynmi dtF xot only will this serve s vlidtion to the sttiEtrget pproh"show whether it is possile to use this kind of stti rtiultory representtions s points of referene for the dynmis of speeh"nd serve s stepping stone to potentil hyrid methodsD it will lso dvne us in Ews dt tretment in generlX for exmpleD it n serve s se for the deision on whih Ews ptures to nnotte mnullyF he im of the prt of work is to look for suh sttiD frozen rtiultory trgets tht were onstruted in the stti ws orpus in Ews dt nd give n interprettion either of their preseneD possily to some extent or only in some sesD or seneD e'etively drwing link etween the two types of dtF he ojetive is to employ mesures tht re proven to e e0ient in imge proessing nd omputer vision feture extrtion to ompre the stti nd dynmi ws dtsetsD nd to drw onlusions from the dynmis nd distriutions of these similrity mesuresF his work ws joint e'ort of sonnis hourosD enstsi himorin nd myselfX the methodE ology ws preliminrily vlidted y sonnis houros nd meD sonnis houros prepred the proE essing protool of the initil set of input imges @tht were lter repled y the vol trt windows proessed nd ut s desried in the previous hpterAD the omputtion ws set up y me nd run with the help of sonnis houros nd enstsi himorinD the results were gE gregted y enstsi himorin nd meD nd evlution nd nlysis were done y meD though enstsi himorin provided helpful disussions out their methodologyF (2) he imges do not depit extly the sme res of the sujets9 vol trtsD nor do the sujets tke extly sme positions or postureF woreoverD three yers pssed etween quiring the stti nd dynmi dtsetsD resulting in some minor physil hnges in speker S A Y nd nturllyD there re di'erenes etween spekers S A nd S B F

(3) tti quisitions my produe shpes tht will never e oserved in dynmi dt sine they involve no phontion nd there re phonemes whose sustined imittion of rtiultion is either di0ult or impossile @liquids due to their dynmi nture" point rised in viIVY stopsD whose urst is result of pressure uilding up in the vol trtY it is di0ult to ontrol nslityAF (4) he stti dtset is rther smll nd should not e expeted to e le to over ll the imges in the dynmi dtsetF

(5) hile eing lrgerD the dynmi dtset still remins reltively smll s fr s speeh resoures goF hen reking down into spei( ontextsD phoneme lssesD syntti struturesD or speking stylesD dt sprsity quikly eomes n issueF king ll the points ove into ountD we reted retngulr windows tht only ontined the vol trt informtion @pigF RFPAD from the lryngel region to the lipsD s ws explined in ghpter RFPFIF his ropping relied on the referene points of the tip of the nose nd the orner of one of the vertereD whih is less relile thn (tting the rtiultory model derived from stti imges to the dynmi onesD ut with the upside of eing usle in the sene of rtiultory ontoursF hese windows were pplied oth to the stti nd dynmi dtF hen the smller windows were resized to imges of size 84 × 82 pixelsF IRP

Data and methods

Static MRI e pplied histogrm mthing to eh stti imge so tht its imge hrteristis follow the ones of smple dynmi imgeF es n sideD sine the hed position of speker S A ws quite di'erent in the stti ws dtset ompred to the Ews one ndD out of ll hosen tehniquesD only SIFT is rottionE nd lotionEinvrintD we turned the stti imges y 12.3 • so tht the vol trt ngles were ligned with those in the dynmi onesF es ws explined in ghpter QFQFID eh of the 95 ws ptures orresponded either to vowel V or to loked onsonntEvowel C(V) rtiultionF RT-MRI here were some Ews imges where the ropping lgorithm for leving out ll rtiultorily irrelevnt informtion filed due to n inorretly determined nose tip @this prt of the work ws done efore the spekerE nd sequeneEspei( improvements to seleting windows tht were desried in ghpter RAF hese sequenes were kept outF por the purpose of nlysisD eh ropped Ews frme needed to orrespond to phoneti lel whih ws foreEligned iq + HP uIT orreted output of evite r ffhIR @see ghpter RFPFI for detilsAF ivery sequene where the phonetiztion lgorithm filed ws thus exluded tooF sn the end it left us with 368, 848 Ews frmesF he sme riteri were then pplied when developing method to evlute the results of this workF e hose to sty s rigorous in our pproh s possile nd to hve oherent mesure etween eh of the stti imges nd eh of the dynmi imgesF e ut out the retngulr of IRQ the vol trt nd resized the resulting imges to 84 × 82 pixelsF hen three tehniques whih re known to perform well in imge proessing nd omputer vision feture extrtion were usedX struturl similrity @swA fHRD evHWD irth mover9s distne @iwhA qHH @sserstein distne on the histogrms of the imgesAD nd sleEinvrint feture trnsform @spA vowWWD vowHRF iwh mesures the di'erene etween two proility distriutionsD lulting the work it would tke to trnsform one of them into the otherF hen pplied to pixel intensity histogrmsD it produes mesure of imge similrityF sf f i,j is the optil )ow etween lusters p i nd q j from P nd Q respetively nd d i,j is the ground distne etween themD

EMD(P, Q) = m i=1 n j=1 f i,j d i,j m i=1 n j=1 f i,j
@SFIA vower vlues of EMD men more similr imgesF sw fHRD evHW is mesure tht originlly qunti(ed pereived imge degrdtion when given n originl imge nd its ompressed versionD ut n e used to quntify similrity etween ny two imgesF st is lulted on windows of the imgeF SSIM etween two windows x nd y"in our implementtionD windows of size 7"is rtio tht depends on the windows9 vergesD vrines nd the ovrine fHQX

SSIM(x, y) = (2µ x µ y + c 1 )(2σ xy + c 2 ) (µ 2 x + µ 2 y + c 1 )(σ 2 x + σ 2 y + c 2 ) , @SFPA
where µ x nd σ 2 x re the verge nd the vrine respetively of window xD µ y nd σ 2 y of yD σ xy is the ovrine of x nd yD nd onsonnts c 1 nd c 2 re omputed s followsX

c s = (k s × R) 2 ,
R eing the dynmi rnge of the pixelEvluesF he (nl SSIM index is the verge over ll the windowsF sts vlues rnge from -1 to 1D 1 stnding for identil imgesD nd -1 for n inverted lol struture fHVF sp vowWWD vowHR is feture detetion lgorithmF st desries fetures s histogrms of oriented grdient in neighorhoodF pirstD ertin numer of fetures re deteted nd omputedF he lrger this numer isD the lower the threshold lolEontrst sore is for feture to e retinedF wost wek fetures in lowEontrst regions re (ltered outD nd edgeElike fetures re preferredF hen the omputed fetures in the soure nd trget imges re ompred nd mthedF e used feture mthing with 5 kEd trees fenVH using pvexx @pst virry for epproximte xerest xeighorsA wvIUD whih is suitle for nerest neighor serh in lrge dtsets nd highEdimensionl feturesF ih mth ws ssoited with distneX the lower this vlue wsD the etter the mthF sf two imges depit similr ojetsD their mthes re supposed to e loseD whih rings in the ide to mesure imge similrity y the proportion of lose mthes out of ll mthes onsidered"iqution SFQX SIFT (x, y) = |matches(x, y) : match.dist < threshold| |matches(x, y)| @SFQA fefore running the omprisons on the entire dtsetD s took smll suset of di'erent vol trt on(gurtions nd studied the qulity of mthes identi(ed y sp when djusting the IRR imge preproessing fetures @ilterl (lter wWV prmetersD s explined in ghpter RFPFIAD the numer of fetures nd the mth distne thresholdF he on(gurtion with the fewest noisy mthes tht still yielded enough informtive ones ws when the ilterl (lter hd dimE eter d of eh pixel neighorhood set to 9 @ lrge (lter"s heked vlues 5, 6, 7, 8, 9X lrger (lter ws more sensitive to the glol struture of the ojets in the imgeA nd when the (lter9s σ prmeter oth in the olor nd oordinte spe ws redued to 20 in ontrst to 75 from the setup in ghpter RFPFI" muh more moderte hnge @s heked σ vlues 20, 40, 60 nd 80X too muh lurring redued the qulity of mthesAF es for the numer of fetures nd their threshold distneD it proved to e good ide to lulte moderte numer of feturesD 80 @s heked 5, 20,35,50,65,95,110,125 nd 140X too few fetures were not informtive enoughD nd too mny mde them noisierAF ith this in mindD s set the limit threshold to e 80 @s heked 60, 70, 80, 85, 90 nd 95X eing too lenient with the distne threshold led to flse mthesD nd too strit used the mthes to eome trivil or dispper ltogetherAF ine SIFT mesure is proportion of ll mthesD its vlues rnge from H to ID nd the greter the vlueD the loser two imges reF sp trnsformtion is invrint to rottion nd feture lotion in the imgeF o disount the mthes drwn etween di'erent rtiultorsD s experimented with modi(ed version of this mesure s wellD SIFT l D whih is simpli(ed version of n dditionl geometri testX

SIFT l (x, y) = 1 M n=1 dist(keypoint (x) n , keypoint (y) n ) × 1 M , M = |matches(x, y) : match.dist < 70|,
@SFRA the ide eing tht the smller the displement is etween two mthed feturesD the more prole it is tht it orresponds to the sme rtiultor @hene the (rst rtioAD nd tht imge pirs with lot of strong mthes should not e penlized for hving too mny omponents in the sum @hene the seond rtioAF 5.3 Experiments

Temporal behavior

pirst the hosen mesures needed to e vlidted through studying their temporl ehviorX where their extrem were found nd whether there were ny ptterns in the reltions etween the extremF pigures SFRD SFS nd SFT show n exmple of the hnge of the EMDD SSIMD SIFT nd SIFT l vlues over sequene m ville ntle Gma.vil.na.talG @see pigure SFQ for the spetrogrmAF sn prtiulrD pigure SFT shows SIFT nd SIFT l D tht re sed on the sme method spD next to eh otherF eording to our riteriD n idel similrity mesure would hve mjor peks when omprE ing the phonemes with the sme rtiultion @Gm(a)G to Gm(a)GD GaG to GaGD Gf(i)G to Gv(i)GF F F AD smller peks for the sme ple of rtiultion @shred etween Gt(a)GD Gn(a)G nd lso Gl(a)GAD moderte inreses when ompring ny two vowelsD nd very minor inreses when ompring vowel to onsonnt ntiipting itF ht n e identi(ed from these (gures is tht EMD omprisons @pigure SFRA do not seem to e informtiveD s they re highly orrelted nd do not depend on rtiultionF IRS SSIM @pigure SFSA onsistently peks on GaGEtoEGaG omprisonsY the sme holds for GiGD Gn(a)G nd Gt(a)GF hpes resemling Gf(i)G re identi(ed t GiGD nd lso in trnsition periodsD suh s from GmG to GaG nd GlG to GnG to GaGF yne n lso remrk tht quntittive nlysis of SSIM vluesD just y treting the minimum nd mximum vluesD is less promising thn nlysis of lol extremD s the reltive positioning of eh urve rrely hngesF sn our exmpleD it usully is Gt(a)G on the top nd Gn(a)G t the ottomF wenwhileD the lol temporl ehvior of eh urve seems to e more informtiveF he peks of SIFT (Gm(a)G, x)D SIFT (GaG, x)D SIFT (Gf(i)G, x) re onsistent with the ourE renes of Gm(a)GD GaGD Gv(i)G respetivelyF here is ertin onfusion of vowelsX the stti GiG resemles the olletion of frmes of the dynmi GaGY dditionllyD the shpes of Gl(a)G nd Gn(a)G re tivted on Gt(a)G s wellF e shpe resemling Gt(a)G is enountered not only in GtGD ut lso during the trnsitions etween Gm(a)G nd GaGD GiG nd GlG nd GaG nd GlGF he hnge of SIFT l @iqution SFRA over time revels tht generlly the vlues do not diverge muh from 0F eginD there is ertin onfusion regrding vowelsX shpes resemling GaG re ssoited with instnes of GiGY of nsl onsonntsX GmG is onfused for GnGY shpes similr to Gl(a)G pper t GaGY Gt(a)G is orretly identi(ed t GtGD ut lso during the trnsition from GvG to GiGF IRT (c) Coarticulatory eects are a ground to increase the similarity value: static /t(i)/ (left) and a frame of dynamic /i/ (center), whereas a dierence in them is not: consider static /t(E)/ (right).

pigure SFPX honeme omprison riteriX wht rtiultion fetures should ontriute s ftor of similrity @left third of the pgeEenterAD nd how not eing shred should led us to the onlusion tht the phonemes re di'erent @right hlf of the pgeEenterAF IRU pigure SFQX he leled spetrogrm of the denoised sequene tht ws used for the temporl nlysis of EMDD SSIMD SIFT nd SIFT l in pigures SFR!SFTD m ville ntle Gma.vil.na.talGF IRV pigure SFRX he hnge of EMD"irth mover9s distne"vlues over time when ompring the stti frmes Gm(a)GD GaGD Gf(i)GD GiGD Gl(a)G nd Gt(a)G to the sme sequene s in pigure SFQD m ville ntle Gma.vil.na.talGF he lower the vlueD the more similr the frmes re onsidered to eF he signls re strongly orrelted nd do not seem to e informtiveF pigure SFSX he hnge of SSIM"struturl similrity"vlues over time when ompring the stti frmes Gm(a)GD GaGD Gf(i)GD GiGD Gl(a)G nd Gt(a)G to the sme sequene s in pigure SFQD m ville ntle Gma.vil.na.talGF he greter the vlueD the more similr the frmes re onsidered to eF he peks of SSIM(GaG, x) re onsistent with the ourrenes of GaGY the sme holds for GiGD Gn(a)G nd Gt(a)GF Gm(a)GD Gf(i)G nd Gl(a)G do not follow the lelingF pigure SFU shows the similrity mesures etween seven stti imges of GfG @Gf@iAD f@EAD f@AD f@oAD f@uAD f@yAD f@øAGA nd the onseutive imges of the dynmi sequene G(D feD fED fD fOKD foD fuD fyD føD foeKD fãD fÕD fẼGF yne n see thtD just like in pigure SFRD the vlues of EMD in pigure SFU re strongly orrelted nd do not seem to pture ny rtiultorily relevnt informtionF me s in pigure SFSD pigure SFU demonstrtes tht the rnking of SSIM vlues when ompring prtiulr set of stti imges to the relEtime sequene typilly stys quite stleF sn our exmpleD the vlues of SSIM(Gf(a)G, x) re lmost lwys the lowestD nd SSIM(Gf(o)G, x) the highestF he shpes most resemling the stti smples of GfG pper in the trnsitions from GfG to vowelsF histinguishing the voli ntiiptory ortiultion is impossileF pigure SFU shows tht SIFT vlues ntiipte the eginning of GfGD the peks ourring right efore the eginning of the phonemeF eginD distinguishing the voli ntiiptory ortiulE IRW tion is impossileF pigure SFUdD like pigure SFTD shows tht most SIFT l vlues re very lose to 0F he spikes do not seem to e very informtiveD though it n e sid tht they generlly our t the end of the fritive nd n lst through the vowel nd the susequent eginning of sileneF winor peks follow the ehvior of SIFT X they preEempt the eginning of GfGF histinguishing the voli ntiiptory ortiultion is impossileF yverllD the nlysis of the mesure9s temporl ehvior in the exmples ove leds to the following onlusionsX EMD does re)et some hnges in the imge sequenesD ut they re not spei( to ny rtiultory on(gurtionF vikelyD the stti nd dynmi imges needed to e muh more similr to eh other for EMD to e informtive due to the optil )ow ssumptions tht ontour orresponding to n ojet in the soure imge n only slightly move in the trget imgeD while our imges re oth dissimilr in terms of imge qulity nd representing di'erently shped ontoursF sn terms of interpretilityD the performne of SSIM nd SIFT is quite similrF foth do enode some rtiultory informtionX we n identify oth vowels nd onsonntsF eove we disussed mismthes sD for exmpleD ording to SIFT stti Gl(a)G nd Gn(a)G were similr not only to their dynmi ounterprts @tken s entire olletions of frmes spnning over the ourrenes of prtiulr phoneti lelD Gl(a)G nd Gn(a)G in this seA ut lso to Gt(a)GF uh mismthes nd similr demonstrte tht in the se of onsonnts SIFT is rther sensitive to the ple of rtiultion @where the onstrition oursAD whih mkes it etter t getting onsonnts right ut retes di0ulties to identify how open or lose nd front or k vowel isD nd SSIM is etter t pturing the generl positioning of the rtiultorsD whih mkes it etter t deling with vowels nd ntiipting ertin voli ontextF he onerns regrding how well the stti dtset models the relEtime one seem to e justi(edX the espeilly prolemti e'ets re those tht we expeted in point (3) t the eginning of this hpter @ghpter SFPFIAX the prodution of nsls nd liquids nd relisti ntiiptory ortiultionF vikewiseD point (4) out the poor overge of the vriety of shpes ourring in speeh prodution is vlid onernD onsidering the frequent mismthes t the trnsition periods etween ny two phonemesF edding the feture mth lotion into SIFT to rete SIFT l @iqution SFRA does hve some meritD ut the interpretility of this mesure is severely 'eted y the unevenness of its peksX in omprison to the very strong spikes tht do not tully ontin muh rtiultory informtionD the moderte peks hve the potentil to e useful when merged with other vluesF st should e noted tht the jerks in the mesures9 plots our more frequently nd re greter in mgnitude thn the hnges in tul rtiultion n ourF herefore we (nd it imprudent to look for ptterns one imge t timeD for exmpleD t the imge or imges in the enter ISH of given phonemeD like in the pproh of vxIVF snstedD we resolve ourselves to using verging nd looking for generl ptternsF 5.3.2

Distributions and correlations ell the mesures re)eted tht the stti dtsetD eing done with speker S A D resemled S A 9s prt of the relEtime one etter thn tht of S B @pigure SFVAX in se of EMDD the mss of histogrm for S A ompred to S B is shifted to the leftD nd in se of the rest of the mesuresD to the rightF he only exeption to this is SSIM nd its S A ES B distriutions @pigure SFVD middle pirAX in the frmes of S B D SSIM identi(ed more imges tht hd similrity vlue higher thn the mode thn in those of S A F hen reking down y phonemesD this pttern of S A 9s frmes eing loser to the stti dtset thn those of S B re holdsF he shpes of the distriutionsD howeverD sty the sme"see pigure SFWF he lulted distnes were ggregted y spekersD y phonemesD y phonemes in voli ontext @wht vowel V is ntiipted in the dynmi dtset ording to the phoneti lelingAD y the phoneme9s phoneti lsses nd y speking styles @spontneous speeh or notAF o redue the memory lodD it ws done in IH rndomly split loksD T times overF le SFP shows the men nd stndrd devition of eh metri ross the entire volume of speeh y spekers S A nd S B F yverll the reltionship etween the mesures is not strongX see orreltions in le SFQF husD in order to determine whether the oserved di'erenes etween phonemes re sttistiE lly signi(ntD s needed to use nonEprmetri testsF he smples mde y di'erent mesures were ssumed independentF

E(EMD) ± SD(EMD) E(SSIM) ± SD(SSIM) E(SIFT ) ± SD(SIFT ) E(SIFT l ) ± SD(SIFT l ) S A 0.

5.3.3

Analyzing the speakers por every mesureD its distriution for speker S A is di'erent from the one for S B re di'erentY furthermoreD quite oviouslyD the distriutions of ny two mesures for given speker re di'erentX see le SFR @uolmogorovEmirnov nd wnnEhitney testsAF

ISP (a)
The change of SIFT (Equation 5.3) over time. The peaks of SIFT (/m(a)/, x), SIFT (/a/, x), SIFT (/f(i)/, x) are consistent with the occurrences of /m(a)/, /a/, /v(i)/ respectively. The static /i/ resembles the collection of frames of the dynamic /a/. /l(a)/ and /n(a)/ are activated on /t(a)/ as well.

A shape resembling /t(a)/ is encountered not only in /t/, but also during the transitions between /m(a)/ and /a/, /i/ and /l/ and /a/ and /l/.

(b) The change of SIFT l (Equation 5.4) over time. Most of the time the values are close to 0. Shapes resembling /a/ are associated with instances of /i/; /m/ is confused for /n/; shapes similar to /l(a)/ appear at /a/; /t(a)/ is correctly identied at /t/, but also during the transition from /v/ to /i/.

(c) The comparison of SIFT and SIFT l measures: SIFT identies the similarity between /f(i)/ and /v(i)/. SIFT l does, too, but it demonstrates a higher peak already in the section of /i/. Both SIFT and SIFT l nd that the static /i/ resembles the collection of frames of the dynamic /a/ (/i/-like shapes are also found during the transitioning time, for example, between /a/ and /t/; the top peak of SIFT l (/i/, x) is, indeed, at /i/). pigure SFTX he hnge of sp @sleEinvrint feture trnsformAEsed SIFT nd SIFT l vlues in time when ompring the stti frmes Gm(a)GD GaGD Gf(i)GD GiGD Gl(a)G nd Gt(a)G to the sme sequene s in pigure SFQD m ville ntle Gma.vil.na.talGF en idel similrity mesure would hve mjor peks when ompring the phonemes with the sme rtiultion @Gm(a)G to Gm(a)GD GaG to GaGD Gf(i)G to Gv(i)GF F F AD smller peks for the sme ple of rtiultion @shred etween Gt(a)GD Gn(a)G nd lso Gl(a)GAD moderte inreses when ompring ny two vowelsD nd very minor inreses when ompring vowel to onsonnt ntiipting itF ISQ Chapter 5. Static targets versus running speech for articulatory speech synthesis pigure SFUX he glol view of the plots of the similrity mesures etween seven stti imges of GfG @Gf@iAD f@EAD f@AD f@oAD f@uAD f@yAD f@øAGA nd the onseutive imges of the dynmi sequene G(D feD fED fD fOKD foD fuD fyD føD foeKD fãD fÕD fẼGF ISR pigure SFVX he overll distriution of the EMD mesure for the distne etween ll the dynmi nd ll the stti ptures for spekers S A @leftA nd S B @rightA @the pir t the topAY SSIM @middle pirAY nd SIFT @ottomAF yne n see tht the stti dtset ws loser to S A 9s prt of the dynmi oneF he overll shpes of the distriutions re similrD exept tht SSIM hs more imge pirs whose similrity ws greter thn the mode for S B thn for S A F ISS pigure SFWX he overll distriution of the EMD mesure for the distne etween ll the dynmi ptures leled s GiG nd the ulk of stti ptures for spekers S A @leftA nd S B @rightA @the pir t the topAY SSIMD the dynmi Gf(a)G nd the entire orpus of stti ptures @middle pirAY nd SIFT D Gf(y)G @the ottomAF yne n see tht the stti dtset ws loser to A nd for S B D eh mesure9s distriution is di'erent from the othersD nd for eh mesureD the distriutions for S A nd S B re di'erentX uolmogorovEmirnov @KSA nd wnnEhitney @MWA equlity testsF pEvlue is 0.000F le SFSX erifying whether the distriution of eh mesure is di'erent or not when ompring the stti imges to the spontneousE speeh relEtime sequenes or the other equlity testsX uolmogorovEmirnov @KSA nd wnnEhitney @MWAF pEvlue is 0.000F ISU peking styles 'et the distriutionsX spontneous speeh is found to e di'erent from nonE spontneous @le SFSD uolmogorovEmirnov @KSA nd wnnEhitney @MWA equlity testsD α = 0.05AF

X X X X X X X X X X X X

5.3.4

Phoneme comparisons o repitulteD s hd the similrity vlues for eh of the 95 ws ptures ompred to eh of the 368, 848 Ews frmesD ording to four similrity mesuresX EMDD SSIMD SIFT nd SIFT l F ih ptureD ws nd Ews likeD orresponded to phoneti lelD from whih s ould extrt the phoneme in prodution s well s its ntiipted vowelD where pplileF oD for ny phoneme pir enounteredD s ggregted ll instnes of their omprisons in the orpusF por exmpleD when studying mesure sim on the stti ws pture of GaG nd ny instnes of GOG in the Ews orpusD if we hve twenty frmes of GOG in the (rst reordingD (ve in the third nd (ve in the tenthD it would provide thirty vlues of simF yrD when studying phonemes in ontextD s ould (nd the omprison vlues for the stti ws pture ofD for exmpleD Gk(i)G nd ll the dynmi instnes ofD syD Gt(a)GF ivery phoneme ws ssoited to list of its phoneti lssesD suh s vowel nd consonantD ndD more spei(llyD oral vowelD nasal vowelD back vowelD central vowelD etF or labialD alveolarD etF nd plosiveD fricativeD etF his wy the ggregtion ould e rried out not only over ll seprte phonemesD ut lso over the phoneti lssesF o sty within the rnge of omprle feturesD the phoneti lsses were divided into the tegories of ple nd mnner of rtiultionD nslity nd lip roundednessF yne ll omprison vlues for prtiulr pir of phonemesD ortiulted phonemes or phoneti lsses were gtheredD nd s lulted their verge nd stndrd devitionF xote tht it ws done in IH rndomly seleted loksD s in IHEfold rossEvlidtion testsD to redue the memory lod nd repeted T times to mke sure the onlusions were not due to the rndomness of the dt splitF edditionllyD s tested V @vowelA nd C(V) @onsonnt C ntiipting vowel VA distriutions for eing di'erentF he verges nd stndrd devitions were then used to identify whih stti phonemesD phonemes in ontext or phoneti lsses were the losest to eh phonemeD phoneme in ontext or phoneti lss enountered in the dynmi orpusD nd vie versD to wht dynmi phonemesD etF the stti ones me the losestF his prt of the work is presented elow in setion SFQFR" wthing stti nd relEtime phonemesD phonemes in ontext nd phoneti lssesF he distriutions were used to verify the presene of ortiultory e'etsF his prt of the workD tooD is presented elowD in setion SFQFRX esting for phonemeEspei( distriution equlityF Testing for phoneme-specic distribution equality oD in order to do the followingX o see whether we n see the sme distriutions for phonemes di'ering only in voiingD eFgF GpG nd GbGY o identify di'erenes used y ortiultionD eFgF Gp(a)G nd Gp(ã)GY ISV o tell the distriutions of vowels prtD s tested the distriutions formed y ll omprisons of V9s nd C(V)9s with sme onsonnt C for eing di'erent with uolmogorovEmirnov @KSA nd wnnEhitney @MWA equlity testsD α = 0.05F hen ompring the distriutions tht pered from the omprison etween ll rtws frmes leled s given vowel nd the ulk of stti ws frmes to the similr distriution ut for nother vowelD ll mesures were le to distinguish the distriutions of lmost ll vowE elsF here were instnes where vowel pir would produe two indistinguishle mesure distriutionsF his does not disqulify the result s most of these vowel pirs re rther lose phonetilly @for exmpleD GoG nd GuGAY howeverD sine our idel mesure would pik up on phonologil di'erenes s wellD suh indistinguishle results re importnt to onsiderF peifE illyD EMD ould not distinguish GaG nd GOG for S A D ut ws le to distinguish ll vowels for S B Y SSIM ould not distinguish GoG nd GuG for S B Y SIFT GeG nd GÕGD GẼG nd GoeGD GẼG nd GEG for S A D GãG nd GOGD GoG nd GEGD GoG nd G@GD GoeG nd G oeG for S B F SIFT l ws the most indisriminte of them llX the distriutions of GãG nd GoeGD GaG nd GoeGD GÕG nd G@GD GuG nd GyG for S A nd GeG nd GyGD GẼG nd GoGD GẼG nd G oeG were found equlF ome onfusions were frequent ross dt splitsD some were notF st should lso e noted tht lmost ll of these pirs re ses when t lest one vowel of the pir is reltively infrequent in the dtF hen ompring distriutions ppering from lulting similrity mesure etween stti imge depiting ertin vowel or phoneme nd the entire relEtime dtsetD there were lmost no instnes when they would e equlF he ext list of indistinguishle distriutions vried with dt splitsD ut usully there were fewer thn (ve pirs per mesure per spekerF por exmpleD EMD ould not distinguish GK(a)G from GK(ã)G for S A D GK(ã)G from GK(E)G for S B D SSIM Gk(E)G from Gk(Õ)GD Gk(a)G from Gk(u)G for S B D SIFT GS(o)G from GS(u)GD GS(a)G from GS(u)GD GK(ã)G from GK(i)GD Gt(E)G from Gt(i)G for S A G nd Gn(E)G from Gn(i)GD GK(u)G from GK(y)G for S B D ndD (nllyD SIFT l Gs(ø)G from Gs(u)GD Gs(i)G from Gs(u)GD GS(a)G from GS(E)GD GS(E)G from GS(y)GD Gl(e)G from Gl(o)G for S A nd Gk(i)G from Gk(y)GD Gk(i)G from Gk(Õ)GD GS(a)G from GS(E)GD GK(E)G from GK(Õ)G for S B D etF o summrizeD typilly the indistinguishle distriutions onern the phonemes GsGD GSGD GKGD GlGD whih together with the nsls re on the list of the phonemes tht were suspeted to e prolemti to pture in stti setting sine the very eginningD s the erodynmis of produing fritive is quite ruil to djust its rtiultion nd therefore the stti quisition with its silent rtiultion ws t disdvntgeF his is n indition of the ft thtD (rstD for most of the stti imges there is no evidene for them to e produed inorretly or inonsistentlyD sine their distriutions pper to e unequlY nd seondD some sounds re ertinly di0ult to sustin for n extended time in n ws mhineF Matching static and real-time phonemes and phonemes in context es eh of our mesures hs reltion of orderD iFeF ny two phonemes n e onsidered more or less similr ording to it @in SIFT D SIFT l nd SSIMD the greter the vlueD the more similr the phonemesY in EMDD the reverseAD tht ment tht for ny stti pture s hd the mens to see wht phonemes were onsidered to e the most similr to itD on vergeD ording to eh of the mesuresF his diretion of nlysis is enoded s stti to dynmi in the (gures nd text of this hpterF end vie versD for ny lel enountered in the dynmi orpusD s ould identify the stti pture tht me to e the losest on verge @dynmi to sttiAF ISW he ext list of suggestions vried with every progrm runD ut repetitions ppered onE sistentlyF s nlyzed the two most frequent top mthes ross ll the dt splitsD onsidering the following ftors to e of in)ueneX hether the most frequent top mthes ross the rossEfold heks were orretY row lrge ws the vrine of the prtiulr phoneme pir omprison"with respet to the verge vlueY row vrious the top mthes ross the rossEfold heks wereX ifD ording to mesure simD the losest phoneme to ph hnges with every dt splitD then hving mth of ph 1 nd ph 2 does not er the sme importne s onsistently (nding ph 2 to e the losest to ph 1 F he ehvior of similrity mesures di'ered in the following two senriosX (a) stat-to-dyn X ompre given stti ws imge depiting onsonntEvowel GC(V)G or vowel GVG loked rtiultion to ll the Ews imgesD roken down y their ortiulted phoneme lels @GC(V)G or GVG ginAD or ompre set of stti ws imges depiting the sme onsonnt GCG in di'erent voli ontexts to ll Ews imges s per their phoneme lelsD with no ortiultion informtionY nd (b) the other diretion dyn-to-stat X ompre the unh of dynmi imges leled s prtiulr phoneme 17 D with ortiultion or notD to stti imgesD with ortiultion or notF hen performing imge omprisons in the dyn-to-stat diretionD EMD onsistently lists GoG nd GOG s the est ndidtes for ny dynmi imge when disrding ortiultory informtionD nd GoGD GOG or Gm(a)G when keeping it inF his holds for oth of the spekersF sts implition is thtD on vergeD the sttistis of the Ews frme dtset mthed most losely those of these three stti ws pturesF sn the reverse diretionD stat-to-dynD EMD is very noisy nd inonsistent when deling with ortiulted phonemes @GC(V)GAF hen disregrding ortiultionD the est mthes strt repeting over di'erent dt splitsD nd some of them eome rightD ut this improvement is not drmti nd the mthes sty spordiF e n thus onlude tht there is little hope to pturing rtiultory informtion with the EMD mesureD euse of the dtsets eing too di'erentF SSIM lso hs prolem of giving sme est mthes for ll dynmi phonemes with no ortiultionX in the dyn-to-stat diretionD in the se of sounds with little or no ostrution @vowelsD pproximntsA GEGD GOG nd GaG re the most frequent mthes for S A D nd GuGD GwG nd GeG for S B F hen dditionlly reking down y ntiipted vowelsD s dditionlly get GKG nd GfG in di'erent voli ontexts tht indisrimintely pper s top frequent mthes for S A D nd GsGD GSGD GpG for S B F his ould e due to the ft tht in smller groupingsD roken down y ortiultionD the impt of rtiultory trnsitions ppering in Ews ut leled just the smeD s vowels or onsonnts ntiipting vowelD eomes lrge enough to get high SSIM sore on verge @or it ould e inditive of inorret timing in the phoneti lelsAF es further evidene of thisD when ompring Ews vowel frmes to stti imgesD s n identify some reltion etween the originl vowels nd the mthes9 ntiipted vowelsX for exmpleD 17 I will call such image sets dynamic images and similarly hereon, even though I do not mean any particular RT-MRI frame; the approach is to generalize over all the frames that received a particular label ITH the most frequent est mthes for GEG re GK(E)GD Gl(e)GY for GãG the est mthes re GK(E)GD GK(ã)GY for Gk(i)G it is Gk(i)G or Gl(i)GF his mens tht the openness of the ntiipted vowel is usully orretF he feture of nslity is not ptured in the omprisons in this diretionX SSIM mthes Gf(E)G nd Gp(o)G to the dynmi imges of Gm(E)GY orD nother exmpleD Gp(y)GD Gl(y)GD Gp(i)G to the dynmi imges of Gm(y)GF es for stat-to-dyn SSIMD in most ses its est mth for sounds with little or no ostruE tion will indisrimintely e GHGD GøG or GuG for S A nd GHGD GoG nd GwG for S B "sounds tht fll into the sme tegoryD ut re no preise mthes @oth GuG nd GoG re k rounded vowelsD nd the di'erene etween them is tht GuG is lose nd GoG is loseEmidY GwG is lil pproximntD the losest onsonntl equivlent of GuGAF st oth disquli(es SSIM from eing used s evidene of the stti frmes vlidity in this seD ndD regrding the onsisteny nd similrity ross spekers @lil pproximnts nd loseEmid rounded vowelsAD does show tht some rtiultory informtion is indeed pturedF he preision inreses for fritivesD oth with voli ontext nd not @Gf, v, z, s, Z, S, KG"ppernes of orret mthes re numerous nd frequent over di'erent dt splitsAD nd even more so in the se of plosivesX oth GkG nd GgG get mthes of GkG nd GgGD Gp, bG re mostly mthed to GbG in se of S A or GmG in se of S B D whih ptures the lil rtiultionD ut rises questions out voiing nd nslityF Gt, dG get mthes of fritives with lose ples of rtiultionX Gf, vGD Gs, zGD GS, ZGF hen pplied to nsl stopsD howeverD lmost ll stat-to-dyn SSIM9s top frequent mthes involve nslityX for exmpleD for Gm(Õ)/D they re Gb(e) @sme ple of rtiultionA or Gv(Õ)G for S A D or Gn(e) or Gm(e)G for S B D or for Gn(u)GD it is Gj(u)G @lose ple of rtiultionD sme voli ntiiptionA or Gb(Ẽ)G @nslityA for S A nd Gj(u)G or Gm(u)G @nslityD sme voli ntiiptionA for S B F ht n e onluded is tht with SSIM it is esier to (nd the Ews imges tht re most similr to given stti imge or set of them rther thn vie versD to (nd the est stti imge to explin olletion of Ews imgesF hen the rtiultors re lose enoughD SSIM egins to pture voli ontext of onsonntsD nd when there is ontt suh s in plosivesD SSIM is ple of identifying the ple of rtiultionF he symmetry etween SSIM9s tretment of nslity"no hndling nslity in the dyn-to-stat diretionD pturing it in the stat-to-dyn diretion nd even fvoring nsl mthes for some orl onsonnts"is n inditor tht the onern regrding the orretness of nsliztion in the stti dtset my e quite justi(edX the orl sounds my e produed s if slightly nslized euse of the lk of tension in the rtiultorsD nd in the simultion of nsl sounds the velophryngel port my e not open enough euse there is no tul speeh prodution ourring during the stti ws quisitionF sn the dyn-to-stat diretionD SIFT 9s mthes of vowels with no regrd of ortiultion re rther loseX its est ndidtes for GaG re GaG nd GeG @oth re front nd unroundedA for S A nd only GaG for S B Y for GeGD GeG nd GiG @oth re front nd unroundedA for S A nd GaG nd GeG for S B Y for GiGD GeG nd GiG @oth re front nd unroundedA for S A nd GyG @lso lose nd front s GiGD ut roundedA for S B Y for GuGD GwG for oth spekersD GwG eing the losest onsonnt to GuGY for GyGD GeG nd GwG for S A D GyG nd GwG for S B F hen s rek the ses of Ews nd stti ws onsonnts y the vowels they ntiipteD the top frequent est mthes of vowels strt inluding onsonntsD with the sme e'ets s in the similr senrio of SSIMX there is reltion etween the @dynmiA vowel nd the vowel ntiipted y the @sttiA onsonntF por exmpleD SIFT mthes GaG nd Gk(a)G to the dynmi GaGD or Gw(a)GD Gt(o)G nd Gp(o)G to the dynmi GoGF he explntion ould e the sme s in SSIMX either this is the in)uene of the ITI trnsition timesD or it is n inditor of temporl shifts in the phoneti lelingF sn the stat-to-dyn diretionD SIFT 9s mthes of vowels with no regrd of ortiultion re rther oftentimes nslX the stti GaG is mthed to GøG or GẼG of S A or to G oeG or GøG of S B Y GøG to GøG or GoeG for S A nd GøG for S B Y GEG to G oeG or GẼG for S A nd G oeG for S B Y GoeG to GøG or G oeG of S A or G oeG or GãG of S B F tust like it ws with SSIM of the sme senrioD GHGD GwG nd GuG re unusully frequent nd indisriminte mthes s wellF hen pplied to nsl vowels nd onsonntsD dyn-to-stat SIFT is not ple of mthing them to the nsl vowels in the stti dtsetX ll its mthes re orlF es for the st-to-dyn diretionD SIFT does pik either nsl Ews phonemes or phonemes with velr rtiultionD eFgF ording to SIFT D if ortiultion is tken into ountD the losest mthes to the stti GãG re Gb(ã)G or Gv(ã)G for S A nd Gr(o)G for S B Y nd with no ortiultionD no onsonnts pper mong the mthesD nd the mthes re orretX for exmpleD the stti GẼG gets mthed to the dynmi GẼG or G oeG for S A or G oeG or GẼG for S B F sn fritivesD with no regrd for ntiipted vowels oth dyn-to-stat ndD to lesser extentD stat-to-dyn SIFT mthes dynmi @sttiA fritives to stti @dynmiA vowelsF Stat-to-dyn works etterX for exmpleD its mthes for GKG re nsl vowelsD nd top mthes for Gf, s, SG do inlude not only vowelsD ut lso Gf, vGD Gs, zG nd GS, ZG respetivelyF roweverD when we tke into ount the voli ontextD the est mthes drstilly improve for Gf, vG nd Gs, zGX the est mth for the dynmi Gf(a)G is the stti Gf(u)G or Gf(i)G for S A D Gf(u)G for S B Y for Gf(o)GD it is Gw(a)G or Gf(o)G for S A nd Gf(u)G or Gf(o)G for S B Y for Gs(o)GD it is Gs(u)G or Gs(ø)G for S A nd Gs(u)G or Gf(u)G for S B F here is no improvement for GKG nd GS, ZGX the dynmi GKG gets mostly onfused with stti GkGD nd the stti GKG stright out does not orrespond to ny relile mthes in the dynmi dtsets @sillyD every dt split hnges the est ndidteD nd they do not repetAF yne possile explntion ould e tht the liquid sound GKG is produed through )eeting ontt etween the velum nd the tongue in Ews dt nd through lerly visileD fullEon onttD whih is more pproprite for stop or trillD in the stti dtF es for GS, ZGD their dynmi instnes re mthed to the stti GnGD GsGD GlGD GtG"ginD nd sounds with similr ple of rtiultionD ut di'erent mnnerF he SIFT mthes of the stti GSG re extremely noisyF henever SIFT does mthes it orretlyD it is to the dynmi GZG rther thn GSGF tust like in fritivesD in stops nd nsl stopsD with no regrd for ntiipted vowels dynto-stat SIFT mthes them to stti vowelsD ut when we tke into ount the voli ontextD it identi(es the pproximte ple of rtiultionX side from orret mthesD we my hve frequent onfusions like GnGEGtGEGfGEGsG nd GpGEGmGEGvGF st should e notedD howeverD tht no voli ontext of the stti GlG ever ppers mong the est mthes of the dynmi GlGF elsoD in the stat-to-dyn diretionD SIFT mthes the stti GnG not only to the dynmi GnGD ut lso to GtGD GvG nd GzGD thus fvoring dynmi mthes with lose ple of rtiultionD ut no ontt nd no nslityF he mthes of the stti GlG in its vrious voli ontexts most often re the instnes of GjGD though GnGD GvGD GzG pper tooF his mens tht SIFT is e0ient t identifying the ple of rtiultionD not only for the onsonntsD ut lso for the vowels @whether they re frontD entrl or kAF utting together the evidene of mismthesD the results of SIFT indite tht the sttiEws simultion of the liquids Gl, KG nd tretment of nslity my hve indeed een inorretX the stti nd dynmi smples of Gl, KG lmost never mthD nd the symmetry of the mthes in the nsls mens thtD in the se of nsl soundsD the velophryngel port ws not open enough in the stti ITP dtset to orretly re)et the dynmi oneD nd in the se of orl soundsD they were slightly nslizedD whih prevented the dynmi dtset9s orl sound smples from eing mthed to themF he nlysis of dyn-to-stat SIFT l mthes shows tht it mkes indisriminte mthes in the se of vowelsD mthing lmost ll of them to GoG nd GOG in the se of S A nd GuG nd GoG of S B F sn pproximnts nd liquidsD dynmi GjG re mthed to stti GlGD nd stti smples of GlG re mthed to dynmi ones of GjGF Stat-to-dyn SIFT l is very noisyD either produing new top mthes t lmost every dt splitD or mthing indisrimintely to Gu, oe, HGF fefore onluding tht sp seems to e most sensitive to rtiultionD let us investigte its methodology in greter detilF pigure SFIH shows wht kind of mthes n e mde orretly y sp nd how it re)ets the interpretility of these resultsF sn prtiulrD pigure SFIH nd pigure SFIH re exmples of very resonly identi(ed mthesF foth of them depit the sp trnsformtion from stti vowel"GaG in the se of pigure SFIHD GEG in the se of pigure SFIH"to dynmi oneD GaGF he lgorithm piks up on ll the key points of rtiultionF he question of interpretility omes up when ompring the resulting vlues of these omprisonsF he expeted resultD nturllyD would e tht the stti GaG resemles the dynmi GaG t lest slightly etter thn the stti GEG doesD despite the ft tht the gesture of GEG psses over shpe very similr to GaGF his is notD howeverD wht hppensF hue to the multiple mthes t the rtiultorily less relevnt hin nd mth of the onstrition of the vol trt despite its inorret lotionD the stti GEG gins higher mth ount nd is thus onsidered more similr to GaG thn the stti GaG is @SIFT (GaG st D GaG dyn ) = 0.1, SIFT (GaG st D GEG dyn ) = 0.1125AF his ehvior is the perfet exmple of the motivtion to nlyze the vlues over time nd on vergeD sine the hnge in time proves to e more illustrtive of the rel similrities"see pigure SFTF ometimes the mthes re indeed orretD ut we hve to dmit tht this suess does not do justie to wht is hppening in the frme from the rtiultory point of viewD s is shown in pigF SFIH where the lgorithm piks up on the shred ple of rtiultion etween Gn(i)G nd GzG rther impreiselyY lterntivelyD even if the mthes do identify ertin similrity s in pigure SFIHd etween Gp(a)G nd Gp(i)GD it my not e signi(nt enough to show up in the numeril results due to too few mthes in totlF st lso goes without sying tht the lgorithm my lso mke mistkesD the extreme ses of whih re shown in pigure SFIIX the mistkes my e mde in suh wy tht does not disqulify the (nl resultD s in pigureSFIID or suh tht nnot e treted in ny other wy thn noiseD s in pigure SFIIF es mentioned in the sp on(gurtion ove in ghpter SFPFPD s visully nlyzed set of frme pirs to (nd the right lne etween seletivity nd produtivity of sp mthes through vritions in its prmetersF he (nl set of prmeters ensured tht ses like in pigure SFII did not hppen oftenF husD it is resonle to onlude tht EMD nd SIFT l do not provide @muhA rtiultorily relevnt informtionD while SIFT de(nitely doesD espeilly on vergeD whih enles us to disrd noiseF SSIM n lso provide some indiret evidene in the stat-to-dyn diretionD one the rtiultors re lose enoughF sn the se of vowelsD SIFT is more sensitive to the frontEentrlEk distintionD while SSIM to openEmidEloseF he qulittive nlysis of mthes nd mismthes rings us to the following onlusionsX e do not dispose of the mens to vlidte or disredit the simultion of the qulity of ITQ vowel prodution in the stti ws dtsetF here is n issue of nslity representtion in the stti ws dtsetX the orl sounds re slightly too nslizedD nd the nsl sounds re not nslized enoughF he liquids Gl, KG produed for the stti ws dtset do not mth those seen in the relEtime dtF he phoneme lss tht is mthed the est is the fritives rring GS, ZGD whih seem to hve een prolemti to either produe in the stti ws setting or to mth y our methodsF tudging y mthes like GaGEGk(a)G or GoGEGp(o)GD wht my e hppening ould e (a)

the ontriution of onsonntEvowel trnsition periods in the relEtime dt tht is not represented y the stti frmesD or (b) the phoneti lel timingD otined through fored lignmentD ould e inorretD thus 'eting the nlysisF e urious e'et is tht it ws ommon tht voied onsonnts from relEtime dt mthed more often thn their unvoied ounterprtsD whih were tully reorded in the stti dtsetF st is not neessrily due to their vol trt shpes eing etter mthD ut ould e simply due to the proedure of vergingD sine voieless fritives typilly lst longer thn their voied ounterprtsD thus llowing for greter rtiultory vritionF 5.4 Evaluation he qulittive nlysis in the previous setion ove ws performed on the most frequent nd losest mthesF here is need to lso evlute the results quntittivelyD to over lrger expnse of dt rther thn the top most frequent mthes nd to mke sure their interprettion ws not 'eted y humn isF 5.4.1 Articulatory similarity measure vet us onsider funtion tht n ompre ny two imges depiting vol trt on(gurtions nd give vlue of how similr they reF vet us syD we ompre GlG nd numer of imgesX GlGD GuGD GkG nd GfGD nd ording to our nlysisD the imge tht is losest to GlG is GfGF his is inorretD ut these phonemes tully hve lot in ommonX oth re rtiulted t the front of the vol trt @lveolr nd liodentl onsonntsAD oth re orlD nd the lip position is very similr unless 'eted y ortiultionF st would e resonle to penlize this error less thn for ssuming tht the losest phoneme to GlG is GkGD whih hs di'erent ple of rtiultionF his motivted me to rete golden stndrd similrity mesure sed on the similrity of rtiultory feturesD ording to the priniples lid out ove in ghpter SFPFPF his mesureD RS(ph 1 (V 1 ), ph 2 (V 2 )) stnding for reference similarity @iqution SFIHA eE tween phonemes ph 1 nd ph 2 D optionlly ntiipting V 1 nd V 2 respetivelyD hs severl omE ponentsF he (rst omponent is CF @iqution SFSAD for common featuresD nd it is sed on the lists of phoneti lsses eh phoneme is ssoited toF por exmpleD GnG is nasal consonant ITR @whih refers to the lss of nsl stopsADalveolarD nasal @whih refers to ll phonemes with nslityAD consonantD nd GtG is plosiveD alveolarD oral consonantD oral nd consonantF he fetures they shre re alveolar nd consonant "two fetures in totlD"while the union of the lists of fetures is six fetures longF hus the proportion of the shred fetures is two over sixF

CF(ph 1 (V 1 ), ph 2 (V 2 )) = |F 1 ∩ F 2 | |F 1 ∪ F 2 | , F k = {articulatory features of ph k } @SFSA
he vlues of CF thus vry from 0 to 1D 0 stnding for two phonemes tht do not hve single feture in ommon nd 1 for those tht re identil from the rtiultory point of view @suh s GpG nd GpG or GkG nd GgGAF yne disdvntge of CF is tht it hs no informtion on how lose or fr to eh other two ples of rtiultion reD or how similr or distint two mnners of rtiultion reF his informtion is used in the next two omponentsD PlA nd MAD stnding for the places nd manners of articulation respetivelyF

PlA works on the ordering of ll possile ples of rtiultion within the vol trtX labialD alveolarD palatalD etF he distne etween element positions on this listD divided y the mximl distne possile @6 sine my implementtion hs 7 tegories of ples of rtiultionAD n e used to mesure the similrity of the ples of rtiultion"iqution SFTX

PlA(ph 1 (V 1 ), ph 2 (V 2 )) = |pl 1 -pl 2 | 6 , pl k = place of articulation of ph k @SFTA
imilrly to PlAD MA works on the ordering of ll possile mnners of rtiultionX plosiveD nasal consonantD fricativeD approximantD close vowelD close-mid vowelD etF"iqution SFUX

MA(ph 1 (V 1 ), ph 2 (V 2 )) = |m 1 -m 2 | 7
, m k = manner of articulation of ph k @SFUA he (nl two omponentsD CVsim @iqution SFVA nd VVsim @iqution SFWAD re vlues tht relte to the ntiipted vowels V 1 nd V 2 D tking into ount the ontriution of ortiultion into the similrity etween ph 1 nd ph 2 F ine V 1 nd V 2 re optionlD if one or oth of them re missingD the respetive omponents do not pplyF hey re lso omitted when deling with ses suh s Gp(a)G nd Gb(a)G"onsonnts tht vry only in voie nd ntiipte the sme vowel"in order not to penlize them for dissimilrity etween GpG nd GbG nd GaGF

CVsim(ph 1 (V 1 ), ph 2 (V 2 )) = RS(ph 1 , V 2 ) + RS(ph 2 , V 1 ) @SFVA VVsim(ph 1 (V 1 ), ph 2 (V 2 )) = RS(V 1 , V 2 ) @SFWA
hen RS n e de(ned s followsX pigure SFIHX ixmples of quite resonle mthes done y sp when ompring stti imges @left hlvesA to the dynmi ones @right hlvesAF girles indite res tht were feturesD iFeF tht were onsidered s potentil mthesF henever irle t the left is joined to irle t the rightD it mens the respetive res mthed ndD from the lgorithm9s point of viewD the on(gurtion of the rtiultor in tht region is similrF pigure SFIIX ixmples of inorret mthes done y sp when ompring stti imges @left hlvesA to the dynmi ones @right hlvesAF girles indite res tht were feturesD iFeF tht were onsidered s potentil mthesF henever irle t the left is joined to irle t the rightD it mens the respetive res mthed ndD from the lgorithm9s point of viewD the rtiultor9s on(gurtion there is similrF le SFTX ome referene similrity @RSA vlues for phoneme pirsF ITV es it ws disussed oveD for eh stti ws frme nd for eh Ews frme9s phoneti lel eh of the four mesures hd its est ndidtes"Ews phoneti lels or stti ws frmes respetively thtD on vergeD were the losestF RS enled me to rry out forml evlution of the mthes nd mismthes in these est ndidtesF purthermoreD s generlized PlA @iqution SFTA nd MA @iqution SFUA to e le to evlute the est ndidtes when deling not with phonemesD ut with rtiultory lsses"RPhClD stnding for referene phoneti lss distneD iqution SFIIX RPhClD(phcl 1 , phcl 2 ) = |index(phcl 1 ) -index(phcl 2 )| maximal index in this category @SFIIA hen pplied to the oral-nasal or rounded-unrounded distintionsD RPhClD is 0 when the phoneti lsses re the sme nd 1 otherwiseF hen pplied to the ples nd mnners of rtiultionD RPhClD n tke intermedite vlues etween 0 nd 1 s wellD to show tht fricative is s lose to n approximant s n approximant is to close vowelD nd tht is muh loser thn to n open vowelF 5.4.2 Articulatory error por eh lok of ggregted dt nd eh phoneme pir enountered in itD s produed the verges nd stndrd devitions s explined oveD thusD given phoneme in one orpus @stti or relEtimeAD otining rnking of phonemes of the omplementry orpus y their similrity to the initil phonemeF o eginD let us onsider only the est ndidte cand (1) ph for eh phoneme ph nd eh of the four similrity mesuresF he phoneme in question ould e ompred to the ndidteX RS (1) = RS(ph, cand (1) ph )

RS(ph 1 (V 1 ), ph 2 (V 2 )) = 1 f w f × f∈{CF,PlA,MA,CVsim,VVsim} w f × f(ph 1 (V 1 ), ph 2 (V 2 ) @SFIHA
sf the ndidte ws perfet mthD then RS (1) ws equl to 1F sf there ws nothing in ommon etween the two phonemesD RS (1) ws 0F his wyD for cand (1) ph D the mesure tht produed it produed n error 1 -RS (1) F hen evluting mesure @EMDD SIFT D SIFT l or SSIMA only on the (rst est ndidteD the error AED stnding for rtiultory errorD would umulte over the entire set of initil phonemes nd their est ndidtesX

AE 1 = ph 1 -RS(ph, cand (1) ph )
vet us onsider further est ndidtesF he mesure expettion for the (ph, cand (1) ph ) pir ws E[sim(ph, cand (1) ph )]Y forD syD the seond est ndidte it ws lower in the se of SIFT D SIFT l nd SSIM nd higher in the se of EMDX E[sim(ph, cand (2) ph )]F e n ount this di'erene in nd disount the error ontriuted y seondE or thirdEest ndidte proportionlly to the ITW di'erene etween their verges"iqution SFIPX

AE k =                ph k (1 -RS(ph, cand (k) ph )) × E[sim(ph,cand (k) ph )] E[sim(ph,cand (1) ph )] if sim = SIFT or SIFT l , ph k (1 -RS(ph, cand (k) ph )) × E[sim(ph,cand (k) ph )]+1 E[sim(ph,cand (1) ph )]+1 if sim = SSIM, ph k (1 -RS(ph, cand (k) ph )) × E[sim(ph,cand (1) ph )] E[sim(ph,cand (k) ph )]
if sim = EMD @SFIPA his wyD if the verge vlue for the seond est ndidte ws 2 3 times the verge vlue for the winnerD this RSErelted error will e ounted within the (nl AE k error vlue with the weight 2 3 F le SFU presents the est nd worst similrity mesuresD ording to the rtiultory error from iqution SFIPD ross the entire dtsetD ggregted y spekersD diretions of omprisons @stat-to-dyn or dyn-to-stat AD the numer of est mthes ounted into the formul @one or twoAD the est nd worst similrity mesures nd their verge error ross the entire orpus in tht spekerEdiretionEortiultion modlity settingF st shows thtD in ftD on verge the mthes on the frmes of S A were etter with SIFT D nd those of S B @nd onsequentlyD oth of them pulled togetherA with SSIM @ut for stat-to-dyn omprisons when ortiultion is tken into ountD when SIFT is etter for S B too"though tully their error vlues re lmost equl thereD 0.2406 for SIFT nd 0.2421 for SSIMAF he worst mthes re EMD nd SIFT l F IUH Towards bringing the two directions together yf the two pprohes s hve studiedD one driven y stti rtiultory trgets nd one synE thesizing the rtiultory prmeter sequene just like the oustisD oth hd their strong nd wek pointsD s disussed oveF st would e highly ene(il to rete rtiultory trgets with stti ptures nd trnsitions etween them with dynmi dtF he work presented ove n thus serve s point to uild upon when working towrds ringing the two pprohes together in hyrid rtiultory synthesizerF pirst o'D we showed how SSIM nd SIFT n help us (lter out ws ptures where the stti simultion of sound prodution did not sueedX these re the sound tegories @liquidsD siilntsA where oth SSIM nd SIFT repetedly fil to mth the dynmi frmes to the their stti ounterprtsF henD one ould ugment the dtset of loked rtiultions through omining the ensemE le of stti imges @tht would ring in the speker9s vol trt dimensionsD preserving the homogeneity of the dtseA with multiple dynmi frmes leled with the phoneti lel of interest @tht would ring in the rtiultory informtionA with tehniques suh s hs + IWF es for extrting the rtiultory strtegies for trnsitionsD SSIM nd SIFT do not o'er strightforwrd wy to do tht with Ews dtF e promising prospet would e to go k to individul windows in the se of SSIM nd feture mthes in the se of SIFT nd to try to identify those tht hve high sw sore or tht re lose mthesF uh spei( fous on one ple of rtiultion my e less informtive for the glol pitureD ut it ould ontriute more to our understnding of the rtiultors9 nvigtion to nd from the hypothetil rtiultory trgetF o mth our stti ws dtset to the dynmi Ews oneD we hve presented four imge similrity mesuresF ell four of them were pproprite ndidtes for investigtionX they were known to e used in the (eld of omputer visionD they were sensitive to the hnges in the dtD nd their distriutions were le to distinguish the spekers nd even speking stylesF wo of themD EMD nd SIFT l D were susequently disquli(ed from drwing onlusions reE grding rtiultion euse of their temporl ehvior @not informtive peks in the onsidered exmplesD high orreltion etween di'erent omprisons in the se of EMD nd most vlues eing too lose to 0 in omprison to ourring spikes in the se of SIFT l "pigures SFR nd SFTAD their distriutions @no disrimintion etween the similrity mesure distriutions for very di'erent soundsAD through the qulittive nlysis of the phonemes of one dtset most often found to e on verge the losest to given phoneme of the other dtset @no rtiuE ltory interprettion for the onfusionsAD nd through the quntittive nlysis with the AE 1 D AE 2 error funtions @iqution SFIPD le SFUAD where they were rnked s the worst similrity mesures in ll modlities of the experimentF he other twoD SIFT nd SSIMD were found to e ple of pturing some rtiultory informtionD whih ws vlidted (rst with the temporl nlysis on severl exmplesD thenD irumstntillyD through hving di'erent mesure distriutions for su0iently di'erent soundsD IUP then through qulittive nlysisD where despite using di'erent methodologyD they rehed rther onsistent onlusions regrding wht sounds n e nd nnot e mthed through delving into the omputtion proess of spD nd (nllyD through quntittive nlysisD where their performne ws onsistently the est judging y the AE 1 D AE 2 error funtionsF SIFT performed etter on mthing the frmes of S A "sme speker s in the stti dtsetD" nd SSIM fred etter with hnge of speker for S B F his ould possily relte to the shpe of the distriutions of SSIM omprison vluesD where while the mode of S B 9s omprisons ws lower thn the mode of S A 9s onesD the mss of imges with similrity vlue greter thn the mode ws higher for S B thn for S A F es the mesures were rther noisyD single omprison etween just two imges ws usully not informtive enoughF tterns pprent enough to let us drw onlusions ppered only through verging nd seleting the most frequently identi(ed est mthes in the se of qulittive nlysis nd ll the est mthes in the se of the quntittive oneF he qulity of the stti ws dtset seems to e ppropriteX in generlD it n serve oth s referene for the ple of rtiultion produed in the dynmi dt nd s mnifesttion of ortiultionF he prolemti sounds nd fetures tht s ws le to identify through the nlysis of mesure distriutions nd mismthes were the liquids Gl, KGD whose dynmi prodution ould not e mthed y their stti simultionD the lveolr fritives Gs, SGD ginD simulted unrelistilly in the stti settingD nd the feture of nslityX pprentlyD the orl sounds in the stti orpus were slightly too nslizedD nd in the nsl soundsD vie versD the velophryngel port did not open enoughF hese prolems tie well with the known issues of loked rtiultion with no phontion @the preision of rtiultion of the sounds with omplited temporl senrioAD ggrvted y the supine position in n ws mhineF purthermoreD the peks in the temporl ehvior nlysis nd the mismthes found in the qulittive nlysis re evidene of gret in)uene of trnsition periods in the Ews dt on the resultsF he shpes ppering t the time of trnsition to nd from sound my fil to e explined with the stti imge thtD ording to the phoneti lelD should e the losestF his mens thtD if we were to uild hyrid rtiultory speeh synthesizer oth on ws nd Ews dtD ws dt ould indeed e used on the ondition tht it would e lened out from the sounds whose simultion ws not suessfulF es for (nding rtiultory trgets in the Ews dt to use them in similr wy s the stti ws frmesD we hve to onlude tht the impt of ortiultory e'ets nd the rpidness of relEtime rtiultion would prevent us from singling out one prtiulr imge ofD for exmpleD Gb(i)G to use it s trgetF ine we were only le to (nd ptterns through verging nd the nlysis of most frequent vluesD the sme pproh would need to e used to rete rtiultory trgets from Ews frmesX we would need to omine numerous smples leled with Gb(i)G in one single pitureF tti pitures ould then serve s soure of informtion on the ple of rtiultionD to help hoose the est rtiulted frmes out of the entire phonemeF enother pproh ould e to ssume phoneti leling to e orret nd to indisrimintely tke the middle of eh phoneme s the trgetD s it ws done in vxIUF IUQ 5.6.2

Future work e diret ontinution of this study would e to repet it fousing on spei( rtiultorsD y tking windows s de(ned in ghpter RF por exmpleD window of the velum @s in pigure RFPA is expeted to let us zero in on the feture of nslity nd disregrd everything tht would e irrelevnt to itF hen uilding joint rtiultory synthesizer using oth stti nd dynmi rtiultory dtD in order to ene(t from the preision nd little vrition of the stti dt nd from the tul speeh prodution informtion in rel time s mnifested in the dynmi dtD it seems fesile to extend the olletion of stti frmes to over more phoneti ontextsD whih ould e done with imge omintion tehniques suh s hs + IWF purthermoreD the similrity mesures ould e integrted into fored lignment of Ews dt to improve the qulity of phoneti lelingF st would e espeilly fruitful to omine the informtion of SIFT nd SSIMD to e le to hndle oth similr nd di'erent spekers nd to ene(t from the strengths of oth mesuresF IUR 6 Conclusions 6.1

Global overview he lst three hpters presented the results for n rtiultory speeh synthesizer driven y ortiultionEwre stti vol trt on(gurtions @ghpter QAD n rtiultory speeh synE thesizer tht funtioned s regulr hxxEsed prmetri speeh synthesizer expnded with rtiultory prmeters @ghpter RA nd ridge etween the dt types underlying the two @ghpter SAF st ws shown thtD in oth methods used in rtiultory speeh synthesisD we n ene(t from some dvntges ut hve to fe ertin limittions inherent to the methodF uleEsed rtiultory speeh synthesis osted omprehensive ontrol over the entire set of the rtiultors"the jwD the tongueD the lipsD the velumD the epiglottis nd the lrynx" s well s pHD glottl openingD suglottl nd suprglottl pressureY with vrying qulity of the resultD it ould over the entirety of prenh phonology thnks to n extended set of vol trt on(gurtionsD oming from stti ws imgesD tht served s uilding loks for the systemF epplied s isD it used to hve prolem with ttining onstritions for stops nd nsl stopsD rehing too lose onstritions for fritives nd vowelsD nd mistreting the feture of nslityF his ws e0ientlyD if rutllyD solved y resetting those tues of re funtions tht ontrdited the phoneme in produtionD implemented t postEproessing stgeD right efore simulting the oustisF he movements nd the speeh generted y the ruleEsed system were orret enough to showse the generl vlidity of the pproh ut fr from nturl or intelligile enough to eome usle in ny relEse senrioF his ws due to two mjor ftorsX timing of the systemD thtD tooD ws mnged with rulesD nd the interply etween di'erent omponentsF he sid interply ws not s deisive in the se of vowelsD whih hve firly simple oustis of produtionD nd of stopsD whih hve very reognizle ousti ue of urstD ut quite ruil for fritivesD where the ple of rtiultionD the pressure nd the timing need to e espeilly well oordintedD nd for liquidsD tht were synthesized loser either to semiEvowels or to stops t the orresponding ples of rtiultionF he hxxEsed prmetri rtiultory speeh synthesizer tht ws presented then dE dressed the issue of the supr timing ontrol tht ws so onspiuous in the previous systemF st ws sed on stndrd deepElerning prmetri speeh synthesizer developed with the defult uild your own voie method in werlin uIT nd trined with feedEforwrd IUS network on the denoised udio reorded simultneously with Ews quisitionsF hen the originl ousti model of tht se speeh synthesizer ws ugmented with rtiultory pE rmeters utomtilly extrted from Ews imgesF hose prmeters did not represent s omprehensive piture of rtiultion s the (rst synthesizerD ut should give us the position of the lipsD indiret evidene of the position of the tongueD nslity nd key onstrition vlues etween the velum nd the tongue nd the tongue nd the phryngel wllF he qulity of speeh nd movements ttined y this rtiultory speeh synthesizer is onsiderly higher thn tht of the (rst oneF ertiultionEwiseD ompred to the (rst synthesizerD it dels well with vowels nd fritives ut struggles with those sounds tht require the rtiultors to ome in onttF es for rtiultory trjetoriesD they re hndled onsiderly more nturlly tht those in the (rst synthesizerF he ommon point etween the two systems ws the use of wsD leit of di'erent kindsX stti nd dynmiF st ws importnt to explore the reltion etween the two types of dt nd to try to identify some key vol trt on(gurtions mong those reorded in Ews similr to wht ws ptured in the stti settingF s onluded tht the stti ws dtset ws in generl vlidD lthough it struggled with the representtion of fritivesD where the erodynmis of the prodution is n importnt ftor to eing le to pronoune the sound orretlyD nd liquidsD the prodution of whih requires the knowledge of their temporl ehviorF he onsequene of tht isD possilyD n inferior qulity of synthesis of those two lsses of sounds y the ruleEsed synthesizer tht relied on stti ws dt ws not only due to misligned strtegy of ll its omponents ontrolD ut lso due to the shortomings of the originl dt s wellF 6.2 Future work foth rtiultory speeh synthesizers hd their strong nd wek pointsF eside from potentil studyEspei( diretions to explore tht were mentioned t the end of eh hpterD overllD it would e quite pertinent to unite the two pprohes in hyrid systemF his hyrid system ould e reltively simpleD suh s just using the durtion model from the hxxEsed rtiuE ltory speeh synthesizer to inform the timing strtegy of the ruleEsed synthesizerF st ould lso e muh more intertwinedF por exmpleD we ould mke use of drmtilly extended lirry of vol trt on(gurtions using dynmi dtD or pply n utomti delinetion lE gorithm on Ews dtD represent the ontours with the sme rtiultory model s in the ruleEsed synthesizer nd trin the hxxEsed speeh synthesizer with those rtiultory pE rmetersF enother promising prospet is to de(ne gestures or trgets s elements from joint sensoryEmotor spe rther thn only sensory or only the otherY this outlook is supported y psyholinguisti reserh hIWF gonsidering the longEterm pplitions envisioned for rtiultory speeh synthesisD it will e neessry to fous not only on the orretnessD omprehensivenessD nturlness nd intelligiility of systems in developmentD ut lso on their ility to serve their purposeF his would require them to e quite )exile nd highly retiveD for exmpleD to llow the user to djust the vol trt geometry nd promptly provide the updted ousti outputF es for the explortion of rtiultory speeh synthesis in pplition to speeh prodution studiesD with the qulity of synthesis suh s in the hxxEsed synthesizerD it is lredy within the relm of possiilityD even more so when tking into ount the stounding dvnes of deep IUT de hque setion de trnsition limitée pr ses n÷uds d9interpoltion orrespondnts Y gomplex X les trnsitions se font ve l9interpoltion préédente de rermite uiqueD mis le timing vrie selon les rtiulteursF geux qui sont ritiques tteignent leur position ile plus rpidement que les utresD tndis que les rtiulteurs dont l ontriution à l9intelligiilité sonore résultnte n9est ps ussi grnde se déplent plus lentement @pr exempleD l lngue peut être dns plusieurs positions pour le son GbGD mis les lèvres doivent entrer en onttAF he plusD les rtiulteurs omposés de tissus plus lourds @omme le dos de l lngueA se déplent plus lentement que eux légers et très moiles @omme les lèvresAF v9justement des onduits voux otenus selon une perspetive phonétiqueD pr exempleD pr l9imposition du ontt entre les lèvres lors de l prodution d9une olusive lile Y v simultion oustique permettnt d9otenir un signl oustiqueD ontrolée pr une séquene des prmetres oustiques qui doivent être en ord ve l9évolution du onduit volF ves résultts de ette synthèse ont été évlués de mnière visuelleD oustique et pereptuelleD et les prolèmes renontrés ont été identi(és et lssés selon leurs originesD qui pouvient être X les donnéesD leur modélistionD l9lgorithme ontrôlnt l forme du onduit volD l trdution de ette forme en fontions d9ireD ou enore l simultion oustiqueF ves résultts montrent que les formes des rtiulteurs hngent dns le temps en fontion des trjetoires produites du onduit volD et ellesEi sont phonétiquement orretesF illes évoluent en synhronisme ve d9utres prmètres de l prodution vole X pHD pression sousE glottle et suprglottleF insemleD le timing et les vleurs du système doivent être dns un équilire délit (n de ne ps produire d9rtefts dns le pipeline de synthèseF xous trouvons qu9ils sont su0smment ien ordés pour les voyelles et les olusives Y qunt ux fritivesD l9intertion entre le lieu d9rtiultionD le ontrôle de l pression et l9évolution temporelle est si omplexe qu9elle se résume essentiellement à l modélistion de hque fritive séprémentF t9en onlus queD dns le dre de l9ensemle des données du onduit volD un modèle sttistique pour le oder et un ensemle de règles pour le mnipulerD e trvil représente une IVP B.2. Synthèse articulatoire de la parole à partir des données IRM en temps réel explortion pprofondie de l9pproheF ves résultts montrent que de telles données et méthodes ne sont ps indptées à l onstrution d9un synthétiseur de prole rtiultoire Y ependntD étnt donné les limites de l9pproheD il serit plus prometteur de poursuivre e trvil en inorpornt des modèles provennt de ertines données dynmiques réellement enregistrées plutôt que de ontinuer à herher un meilleur ensemle de règles pr modélistion théoriqueD essis et erreursF

B.2

Synthèse articulatoire de la parole à partir des données IRM en temps réel v seonde pprohe été développée en s9ppuynt sur un synthétiseur de référene onstitué d9un réseu de neurones feedEforwrd entrîné à l9ide de l méthode stndrd du système werlin uIT sur des données udio omposées de prole en lngue frnçise enregistrée pr sw en temps réelF ges données ont été segmentées phonétiquement et linguistiquement ve evite r ffhIRD un outil que j9i omplété pr des orretions des erreurs réurrentesF ves étiquettes ont été lignées ve les données udio ve rite de ru iq + HP et werlin omme frontend uITF ges données udioD mlgré un déruitgeD étient fortement prsitées pr le son de l mhine à swD e qui usé des prolèmes de synhronistionF xous vons omplété le synthétiseur de référene en joutnt dix prmètres représentnt de l9informtion rtiultoire X v9ouverture des lèvres et leur protrusion e qui donne les informtions essentielles sur le omportement des lèvresD v distne entre l lngue et le vélumD e qui nous donne les informtions sur l9olusion entre es deux rtiulteurs et fournie une indie ltente de l position vertile de l lngue Y v distne entre le vélum et l proi phryngleD e qui ontrôle l9ès de l9ir dns l vité nsle etD pr onséquentD l nslistionD v distne entre l lngue et l proi phryngleD e qui represente une informtion sur l position horizontle de l lngueF ges prmètres ont été extrits utomtiquement à prtir des imges et lignés u signl et ux spéi(tions linguistiquesF ves séquenes rtiultoires et les séquenes de proleD générées onjointementD ont été évlE uées à l9ide de di'érentes mesures oustiques X l distortion melEepstrum moyenneD l9erreur de prédition de l9périodiitéD trois mesures pour pH X wi @root men squre errorAD gy @oé0ient de orréltionA nd G @frmeElevel voiedGunvoied errorAF IVQ t9i onlus que l9jout des prmètres rtiultoires n9 ps dégrdé le modèle oustique originlF n nlyse de l pertinene des prmètres rtiultoires pr rpport ux lels phonétiques églement été réliséeF v9erreur l plus réurrente été l9hèvement du ontt dns les s où le ontt est indispensle pour produire le sonF uisD j9i véri(é l similitude entre les séquenes originelles et elles qui étient généréesD ve l distne de déformtion temporelle dynmiqueF gette évlution permet de onlure que les prmètres rtiultoires générés s9pprohent de mnière eptle des prmètres originuxF du onduit volD provennt des imges sw sttiques qui onstituient l struture du systèmeF eppliqué tel quelD il vit un prolème pour tteindre des ontts pour les olusives et les olusives nslesD tteindre des pprohements trop étroits pour les fritifs et les voyellesD et mltriter l rtéristique de l nslitéF gei été résolu e0ementD ien que rutlementD en réinitilisnt es tues de fontions de zone qui n9étit ps en ord ve le phonème en produtionD mis en ÷uvre à l9étpe de postEtritementD juste vnt de simuler l9oustiqueF ves mouvements et les énonés générés pr le système fondé sur des règles étient su0smE ment orrets pour démontrer l vlidité générle de l9pproheD mis loin d9être ssez nturels ou intelligiles pour devenir utilisles dns un sénrio réelF gel s9explique pr deux fteurs mjeurs X le timing du systèmeD qui lui églement été géré selon des règlesD et l9intertion entre les di'érentes omposntesF gette intertion n9 ps été ussi déisive dns le s des voyellesD qui ont une oustique de prodution ssez simpleD et des olusivesD qui sont oustiquement très reonnisslesD mis tout à fit ruile pour les fritivesD où le lieu d9rtiultionD l pression et le timing doivent être prtiulièrement ien oordonnésD et pour les liquidesD qui sont synthétisés soit près des semi voyelles soit ux points d9rtiultion orrespondntsF ve synthétiseur de prole rtiultoire prmétrique sé sur hxx qui été présenté enE suite ordé l question du ontrôle de l synhronistion supr qui étit si visile dns le système préédentF sl étit sé sur un synthétiseur de prole prmétrique stndrd à pE prentissge profondD développé ve l méthode pr défut 4uild your own voie4 de werlin uIT et formé ve un réseu de feedEforwrd sur l9udio dénoisé enregistré simultnément ux quisitions sw en temps réelF insuiteD le modèle oustique originl de e synthétiseur de prole de se été omplété pr des prmètres rtiultoires extrits utomtiquement des imges sw en temps réelF ges prmètres ne représentient ps une imge de l9rtiultion ussi omplète que le premier synthétiseurD mis devient nous donner l position des lèvresD l preuve indirete de l position de l lngueD l nslité et les vleurs lés de onstrition entre le vélum et l lngue et l lngue et l lngue et l proi phryngeleF v qulité de l pE role et des mouvements tteints pr e synthétiseur de prole rtiultoire est onsidérlement supérieure à elle du premierF hu point de vue de l9rtiultionD pr rpport u premier synE thétiseurD il trite ien les voyelles et les fritivesD mis du ml ve les sons qui néessitent le ontt des rtiulteursF unt ux trjetoires rtiultoiresD elles sont tritées euoup plus nturellement que elles du premier synthétiseurF ve point ommun entre les deux systèmes étit l9utilistion de l9swD quoique de nture di'érente X sttique et dynmiqueF sl étit importnt d9explorer l reltion entre les deux types de données et d9essyer d9identi(er ertines on(gurtions lés des voies voles prmi elles enregistrées dns le sw en temps réel similire à e qui été sisi dns le dre sttiqueF t9i onlu que l9ensemle des données sttiques de l9sw étit en générl vlideD ien qu9il it eu des di0ultés ve l représenttion des fritifsD où l9érodynmique de l prodution est un fteur importnt pour pouvoir prononer orretement le sonD et des liquidesD dont l prodution exige l onnissne de leur omportement dns le tempsF v onséquene de el estD peutEêtreD une qulité inférieure de synthèse de es deux lsses de sons pr le synthétiseur sé sur des règles qui s9ppuyit sur des données sw sttiques n9étit ps seulement due à une strtégie ml lignée de tous ses omposnts de ontrôleD mis ussi ux lunes des données originles ussi ienF gette thèse se situe dns le domine de l synthèse rtiultoire de l prole et est orgnisé en trois grndes prties X les deux premières sont onsrées u développement de deux synthéE tiseurs rtiultoires de l prole Y l troisième trite des liens que l9on peut étlir entre les deux pprohes utiliséesF ve premier synthétiseur est issu d9une pprohe à se de règlesF gelleEi visit à otenir le ontrôle omplet sur les rtiulteurs @mâhoireD lngueD lèvresD vélumD lrynx et épiglotteAF ille s9ppuyit sur des données sttiques du pln sgittl médin otenues pr sw @smgerie pr ésonne wgnétiqueA orrespondnt à des rtiultions loquées de voyelles du frnçisD insi que des sylles de type onsonneEvoyelleD et étit omposée de plusieurs étpes X l9enodge de l9ensemle des données grâe à un modèle du onduit vol sé sur l9eg @nlyse en omE posntes priniplesA Y l9utilistion des on(gurtions rtiultoires otenues omme soures de positions à tteindre et destinées à piloter le synthétiseur à se de règles qui est l ontriuE tion priniple de ette première prtie Y l9justement des onduits voux otenus selon une perspetive phonétique Y l simultion oustique permettnt d9otenir un signl oustiqueF ves résultts de ette synthèse ont été évlués de mnière visuelleD oustique et pereptuelleD et les prolèmes renontrés ont été identi(és et lssés selon leurs originesD qui pouvient être X les donnéesD leur modélistionD l9lgorithme ontrôlnt l forme du onduit volD l trdution de ette forme en fontions d9ireD ou enore l simultion oustiqueF ges nlyses nous perE mettent de onlure queD prmi les test e'etuésD les strtégies rtiultoires des voyelles et des olusives sont les plus orretesD suivies pr elles des nsles et des fritivesF v seonde pprohe été développée en s9ppuynt sur un synthétiseur de référene onstiE tué d9un réseu de neurones feedEforwrd entrîné à l9ide de l méthode stndrd du système werlin uIT sur des données udio omposées de prole en lngue frnçise enregistrée pr sw en temps réelF ges données ont été segmentées phonétiquement et linguistiquementF ges données udioD mlgré un déruitgeD étient fortement prsitées pr le son de l mhine à swF xous vons omplété le synthétiseur de référene en joutnt huit prmètres représenE tnt de l9informtion rtiultoire X l9ouverture des lèvres et leur protrusionD l distne entre l lngue et le vélumD entre le vélum et l proi phryngleD et en(n entre l lngue et l proi phryngleF ges prmètres ont été extrits utomtiquement à prtir des imges et lignés u signl et ux spéi(tions linguistiquesF ves séquenes rtiultoires et les séquenes de proleD générées onjointementD ont été évluées à l9ide de di'érentes mesures X distne de déformtion temporelle dynmiqueD l distortion melEepstrum moyenneD l9erreur de prédition de l9périodiitéD et trois mesures pour pH X wi @root men squre errorAD gy @oé0ient de orréltionA nd G @frmeElevel voiedGunvoied errorAF n nlyse de l pertinene des prmètres rtiultoires pr rpport ux lels phonétiques églement été réliséeF ille perE met de onlure que les prmètres rtiultoires générés s9pprohent de mnière eptle des prmètres originuxD et que l9jout des prmètres rtiultoires n9 ps dégrdé le modèle oustique originlF ves deux pprohes présentées iEdessus ont en ommun l9utilistion de deux types de donE nées swF ge point ommun motivé l reherheD dns les données temps réelD des imges lésD PHQ 9estEàEdire les on(gurtions sttiques swD utilisées pour modéliser l ortiultionF e(n de omprer les imges sw sttiques ve les imges dynmiques en temps réelD nous vons utilisé plusieurs mesures X l similrité struturelleD l distne du 4terrssier4 et sp Y près voir véri(é l pertinene et l vlidité de es mesuresD j9i étudié qulittivement et quntittiveE mentD puis interprété leur omportementY j9i ensuite nlysé leur similritésF t9en i onlu que sp et l similrité struturelle pturient ien les informtions rtiultoires et que leur omportementD de mnière générleD vlidient les données d9sw sttiquesF ves phonèmes et trits phonétiques prolémtiques que j9i pu identi(er à trvers les nlyses des distriutions et inomptiilités de mesures étient les liquides Gl, KGD dont l prodution dynmique ne pouvit être imitée pr leur simultion sttiqueD les fritives lvéolires Gs, SGD elles ussi simulées de mnière non réliste dns le ontexte sttiqueD et les rtéristiques de l nslitéF sl semlerit que les sons du orpus sttique ient été légèrement trop nslisésD et que réiproquement les sons nslisés présentient une ouverture vélophryngée insu0snteF in(nD j9i disuté l9impt de ette étude pour de futurs synthétiseurs rtiultoires hyrides de l proleF Mots-clés: synthèse rtiultoireD rtiultionD onduit volD swD sw dynmique Abstract he thesis is set in the domin of rtiultory speeh synthesis nd onsists of three mjor prtsX the (rst two re dedited to the development of two rtiultory speeh synthesizers nd the third ddresses how we n relte them to eh otherF he (rst pproh results from ruleEsed pproh to rtiultory speeh synthesis tht imed to hve omprehensive ontrol over the rtiultors @the jwD the tongueD the lipsD the velumD the lrynx nd the epiglottisAF his pproh used dtset of stti midEsgittl mgE neti resonne imging @wsA ptures showing loked rtiultion of prenh vowels nd set of onsonntEvowel sylllesY tht dtset ws enoded with geEsed vol trt modelF hen the system omprised severl omponentsX using the reorded rtiultory on(gurtions to drive ruleEsed rtiultory speeh synthesizer s soure of trget positions to ttin @whih is the min ontriution of this (rst prtAY djusting the otined vol trt shpes from the phoneti perspetiveY running n ousti simultion unit to otin the soundF he results of this synthesis were evluted visullyD oustilly nd pereptullyD nd the proE lems enountered were roken down y their originX the dtsetD its modelingD the lgorithm for mnging the vol trt shpesD their trnsltion to the re funtionsD nd the ousti simultionF e onluded thtD mong our test exmplesD the rtiultory strtegies for vowels nd stops re most orretD followed y those of nsls nd fritivesF he seond explored pproh strted o' seline deep feedEforwrd neurl networkE sed speeh synthesizer trined with the stndrd reipe of werlin uIT on the udio reorded during relEtime ws @EwsA quisitionsX denoised @nd yet ontining onsidE erle mount of noise of the ws mhineA speeh in prenh nd foreEligned stte lels enoding phoneti nd linguisti informtionF his synthesizer ws ugmented with eight pE rmeters representing rtiultory informtion"the lips opening nd protrusionD the distne etween the tongue nd the velumD the velum nd the phryngel wll nd the tongue nd PHR the phryngel wll"tht were utomtilly extrted from the ptures nd ligned with the udio signl nd the linguisti spei(tionF he jointly synthesized speeh nd rtiultory sequenes were evluted ojetively with dynmi time wrping @hA distneD men melE epstrum distortion @wghAD fe @nd periodiity predition errorAD nd three mesures for pHX wi @root men squre errorAD gy @orreltion oe0ientA nd G @frmeElevel voiedGunvoied errorAF he onsisteny of rtiultory prmeters with the phoneti lel ws nlyzed s wellF s onluded tht the generted rtiultory prmeter sequenes mthed the originl ones eptly loselyD despite struggling more t ttining ontt etween the rE tiultorsD nd tht the ddition of rtiultory prmeters did not hinder the originl ousti modelF he two pprohes ove re linked through the use of two di'erent kinds of ws speeh dtF his motivted serh for suh ortiultionEwre trgets s those tht we hd in the stti se to e present or sent in the relEtime dtF o ompre stti nd relEtime ws pturesD the mesures of struturl similrityD irth mover9s distneD nd sp were utilizedY hving nlyzed these mesures for vlidity nd onsistenyD s qulittively nd quntittively studied their temporl ehviorD interpreted it nd nlyzed the identi(ed similritiesF s onE luded tht sp nd struturl similrity did pture some rtiultory informtion nd tht their ehviorD overllD vlidted the stti ws dtsetF he prolemti sounds nd fetures tht s ws le to identify through the nlysis of mesure distriutions nd mismthes were the liquids Gl, KGD whose dynmi prodution ould not e mthed y their stti simultionD the lveolr fritives Gs, SGD ginD simulted unrelistilly in the stti settingD nd the feture of nslityX pprentlyD the orl sounds in the stti orpus were slightly too nslizedD nd in the nsl soundsD vie versD the velophryngel port did not open enoughF pinllyD s ommented on the reperussions of the study for potentil hyrid rtiultory speeh synthesizersF Keywords: rtiultory speeh synthesisD rtiultionD vol trtD wsD Ews PHS PHT

  note trnsriptionsF honeti symols re enlosed in squre rkets D nd phonemes re enlosed in virgules IW pigure PFTX he division of the phryngel vity into the nsophrynxD orophrynxD nd lryngophrynx ypeIQ PH pigure PFUX widEsgittl setion of the vol trt with lels for ple of rtiultion sntWWF GGF ell sounds of nturl lnguges re divided into vowels nd onsonntsF Vowels re rtiulted with n open vol trt where the ir )ows virtully unimpededF ell orgns of speeh re tenseD inluding wlls of resonting vitiesF he ir strem is reltively wekF

  slie of the stti Qh imges of sujet S A for severl of the prenh vowelsF

  nd methods were hosenD the pipeline of the work ws s followsX repre the dtX uild the orpusD rry out the quisitionsD hek tht the dt re vlidY reproess the dtX perform phoneti nnottions nd otin linguisti spei(tionsD synhronize them with the individul frmesY SI ho imge segmenttion or extrt rtiultorily relevnt prmeters from the frmesY rin nd test joint oustiErtiultory hxxEsed prmetri model to otin n rtiultory speeh synthesizerY ivlute nd interpret the resultsF he following setions of this hpter re going to present the detilsF 4.2.1 Data preparation

  TH

  (a) Rapid motion of the velum and therefore blurring (in the middle): S B pronouncing /K/. (b) Swelling that is actually the velum's rolling up on itself: every third frame in S A 's production of /uKi/.

  TP

  4.2. Methods(a) The epiglottis pressing up to the tongue: S B pronouncing /a/. (b) The epiglottis being barely visible in the picture: S B pronouncing /ã/.

  (a) Raw ltered image (b) The upper and lower lip seeds pigure RFVX e smple imge to extrt rtiultorily relevnt informtion from the lipsF he leftmost white re orresponds to the end of the volEtrtErelted prt of the imge nd is thus irrelevntF he two elongted shpes in the enter re the lipsF here is the tongue tip pprohing the upper lip t the rightF he seed points re indited in redF he seed points n e used to suppress ll ontours deteted with the orderEfollowing lgorithm of + VS ut those tht orrespond to the lipsF pigure RFWD from left to rightD shows the vrious ses we my enounterX sf we (nd two smller ontoursD the lips re prtY if we (nd tll ontour strething over the entire imge heightD the lips re losedF e ompliting issue my e the tongue touhing either one of the lips or othD whih typilly hppens outside speeh segments s the speker liks their lipsD ut n nevertheless produe unrelisti vlues or ontoursF his wyD the ft whether the lips were open or losed ws the (rst piee of rtiultory informtion extrted from the imgesF sfD howeverD ny of the ontours ssumed to e the lips touhes the left order of the imgeD it is ssumed tht the window ws pled too muh to the right nd needs to e shifted to the left in order to inlude the outermost points of the lipsF he window is shifted to the leftD nd the previous steps re repetedF pigure RFWX gses to e treted when extrting the rtiultory informtion from the lipsD from left to rightX open lipsD )eetingly losedD losedD the tongue touhing the lower lip nd oth the lips nd the tongue eing in onttF TT 4.2. Methods

O

  eing pixel from the outer ontour of the joined lipsD I the inner one,

  uplpr, lwl p r = RL x -LL x , RL for the rightmostD LLfor the leftmost point of the upper or lower lip respF @RFSA TU essuming the window is stle enough nd rosses the lips t pproximtely the sme heightD this di'erene should e informtive enough out how fr the lips protrudeF (a) The leftmost contour is tall the lips are closed, ls_dist = 0 (b) The leftmost contour is short the lips are open, ls_dist to be computed as the distance between the lips; ls_cont is 0. (c) ls_dist, the distance between the lips, is the shortest distance between the contours of the upper and the lower lip. (d) The lips are closed.ls_cont is computed from the inner and outer contours.(e) ls_cont is the distance between the outer and the inner contour

  (a) The tongue in contact with the velum, the velum with the pharyngeal wall, but not the tongue with the pharyngeal wall. (b) The velum in contact with the tongue, the tongue with the pharyngeal wall, but not the velum with the pharyngeal wall. (c) The tongue in contact with the pharyngeal wall, the pharyngeal wall with the velum, but not the tongue with the velum.

  vtdist = min(eucl(V, T )), V for the velumDT the tongue if no ontt 0 otherwise @RFTA US vwdist = min(eucl(V, W)), V for the velumDW the phryngel wll if no ontt

  v_w_dist=4.47 (b) t_w_dist=3.00 pigure RFIVX vwdist nd twdist omputed in the se of the velum nd the tongue in onttX the miniml distne is omputed only in the re where the rtiultor is expeted to eF the following vluesX vtcont = 0 if no ontt etween the velum nd the tongue

  /Õ/: A contact at the expense of the lowered velum. (c) /K/: A turbulent rhotic contact, the velum is shown in an extensive contact with the tongue.

UV

  A eeting contact between the velum and the pharyngeal wall, about to disappear. (b) Firm contact between the velum and the pharyngeal wall, completely blocking o air from entering the nasal cavity.

  : consistent labels, absent closure and extra closure. The majority of the errors are extra closures for ls_dist and absent closures for ls_cont.(b) ls_dist, ls_cont: absent and extra closure occurrences. Absent closures (no lip contact when supposedly producing /b, p, m/) occurred much more often with respect to how often they could have in both parameters. pigure RFPIX ertiultory prmeter onsisteny in the originl orpusD overll"ontF on the next pgeF VQ (c)

  : absent and extra closure occurrences. Despite a lower total count of extra closures, with respect to the number of times they were checked they occurred much more frequently. pigure RFPIX @gontFA ertiultory prmeter onsisteny in the originl orpusD overll"ontF on the next pgeF VS (g) t_w_dist, t_w_cont: consistent labels, absent closure and extra closure. No closure is necessary to produce any sound, so the only error type that was checked was extra closure. (h) t_w_dist, t_w_cont: absent and extra closure occurrences (no absent closure occurrences since it is not necessary to produce any sound).

  : absent and extra closure occurrences.

  prmeter onsisteny in the originl orpusD roken down y spekers @ontF on the next pgeA VU (c) t_v_dist, t_v_cont: consistent labels, absent closure and extra closure.

SA

  is more likely to have extra closure errors than S B is; extra closures are so prevalent that the overall precision for S A considerably drops. (d) t_v_dist, t_v_cont: absent and extra closure occurrences.

SA

  is more likely to have extra closure errors than S B is.

  : consistent labels, absent closure and extra closure. Spontaneous speech fares better regarding absent closure. pigure RFPQX @gontFA ertiultory prmeter onsisteny in the originl orpusD roken down y spontneity"ontF on the next pgeF WQ (g) t_w_dist, t_w_cont: consistent labels and extra closure. No signicant dierence between spontaneous speech and not. (h) t_w_dist, t_w_cont: extra closure occurrences. No signicant dierence between spontaneous speech and not. pigure RFPQX @gontFA ertiultory prmeter onsisteny in the originl orpusD roken down y spontneityF WR 4.2. Methods (a) ls_dist, ls_cont: consistent labels, absent closure and extra closure.

  : absent and extra closure occurrences. Considering the number of times each error type was checked, for extra closures, there is a greater impact of who the speaker is, and for absent closures, whether it is spontaneous or non-spontaneous speech.pigure RFPRX ertiultory prmeter onsisteny in the originl orpusD roken down y spekers nd spontneity"ontF on the next pgeF WS (c) t_v_dist, t_v_cont: consistent labels, absent closure and extra closure. For both types of errors, the major dening factor is the identity of the speaker rather than the kind of speech. (d) t_v_dist, t_v_cont: absent and extra closure occurrences. For both types of errors, the major dening factor is the identity of the speaker rather than the kind of speech. pigure RFPRX @gontFA ertiultory prmeter onsisteny in the originl orpusD roken down y spekers nd spontneity"ontF on the next : consistent labels, absent closure and extra closure. In v_w_dist, for both types of errors and the resulting precision, the major dening factor is the identity of the speaker rather than the kind of speech. In v_w_cont, both kinds of errors are more dened by the kind of speech. (f ) v_w_dist, v_w_cont: absent and extra closure occurrences. In v_w_dist, for both types of errors, the major dening factor is the identity of the speaker rather than the kind of speech. In v_w_cont, both kinds of errors are more dened by the kind of speech. pigure RFPRX @gontFA ertiultory prmeter onsisteny in the originl orpusD roken down y spekers nd spontneity"ontF on the next pgeF WU (g) t_w_dist, t_w_cont: consistent labels, absent closure and extra closure. No substantial dierences across the speaker identities and the kind of speech produced. (h) t_w_dist, t_w_cont: extra closure occurrences. No substantial dierences across the speaker identities and the kind of speech produced.

  'erent ses for the nnottion of lip ontoursD green for the upper lip nd ornge for the low lipF es for the lip protrusionD oth the upper nd the lower lip were shown to hveD in generlD lrger protrusion vlues uplprotr nd lwlprotr on protruded vowels rther thn on the nonEprotruded onesX IHH 4.2. Methods /s/, ls d ist = 1.0 /m/, ls d ist = 1.0 /m/, ls d ist = 1.0 /m/, ls d ist = 3.16 /ã/, ls d ist = 4 /ã/, ls d ist = 5pigure RFPTX he volEtrt window of the sequene GsmãG @with only the ending of GsG nd the eginning of GãGA nd the respetive extrted distne etween the lipsX se when the ontt etween the lips is too )eeting to e reognizedF pF rF

IHT

  pigure RFPVX he synthesized sequenes of lsdist nd lscont for onjour4 GbÕZuKG with the voie of S A F he lip losure @in redA is onsistent with the prodution of the lil stop GbG nd the nrrowed lil opening for GuG nd with the sene of lil ontt throughout the rest of the utterneF S A X IHU pigure RFPWX he synthesized sequenes of lsdist nd lscont for onjour4 GbÕZuKG with the voie of S A F lues within the rnge ssoited with protrusion re mrked greenF sndeedD oth GÕG nd GuG require lip protrusionD nd it should e ntiiptedF hevelopmentX wiX 24.110 frmesGphonemeD gyX 0.738Y estX wiX 26.476 frmesGphonemeD gyX 0.676Y S B X hevelopmentX wiX 34.212 frmesGphonemeD gyX 0.585Y estX wiX 19.195 frmesGphonemeD gyX 0.580F es for the interpretility of the generted lelsD it generlly follows tht of the orpusF he mjor issue for ll of the rtiultors is ttining onttD thus rising the sent losure error ounts nd frequenyF pigures RFQR nd RFQS show the distriutionsF IHV 4.3. Evaluation (a) t_v_dist (b) t_v_cont pigure RFQHX he synthesized sequenes of tvdist nd tvcont for onjour4 GbÕZuKG with the voie of S A F es the velum lowers to produe the nsl GÕGD the tongueEvelum distne orretly deresesY the ehvior of the velum nd the tongue for the rhoti GKG follows tht of the originl dtD ndD furthermoreD the inrese of the tvcont vlues right efore the end of the utterne ould indite tht the prolem ws getting too open distne t GuG nd hving no time to lose it when GKG without drmti e'et on the derivtiveF IHW Chapter 4. Articulatory speech synthesis from real-time MRI data /u/, t_v_dist = 4other frme when produing GuKiG nd the ssoited tvdist vluesF he points produing the vlues re mrked in redF hespite the intertion etween the tongue nd the velum to produe the rhoti GKGD the extrted distnes etween the tongue nd the velum re positiveD t most reognizing frition ut not onttF e

  pigure RFQPX he synthesized sequenes of vwdist nd vwcont for onjour4 GbÕZuKG with the voie of S A @orl smples mrked in redAF he division etween orl nd nsl sounds on ll phonemes is orret with the exeption of vwdist vlues during the prodution of GbGD whih would e produed s GmG in this on(gurtionF his should e used y the sme e'et s in tvdist nd tvcontX the silent position for the velum is to e loweredD nd the velum ws generted to hve suh high vwdist tht the vlue did not hve the time to get s low s neessry for GbG without 'eting the derivtive @while it should e noted tht the derese from pau to GbG is quite steepAD nd then it ws lredy time to prepre for the nsl GÕG oming nextF III (a) t_w_dist (b) t_w_cont pigure RFQQX he synthesized sequenes of twdist nd twcont for onjour4 GbÕZuKG with the voie of S A F es it should eD there is no ontt etween the tongue nd the phryngel wll throughout the entire sequeneF IIP 4.3. Evaluation (a) ls_dist, ls_cont: consistent labels, absent closure and extra closure. The number of consistent labels is high. Most errors are due to the lips not being closed. (b) ls_dist, ls_cont: absent and extra closure occurrences. The frequency of absent lip closure is higher than that of the lips being closed. pigure RFQRX ertiultory prmeter onsisteny in the synthesized 261 sentenes"ontF on the next pgeF IIQ (c) t_v_dist, t_v_cont: consistent labels, absent closure and extra closure. Almost all labels are consistent. Most errors are due to the tongue and the velum not reaching contact. (d) t_v_dist, t_v_cont: absent and extra closure occurrences. The frequency of an absent contact between the tongue and the velum is higher than that of them touching when they should not. pigure RFQRX @gontFA ertiultory prmeter onsisteny in the synthesized 261 sentenes"ontF on the next pgeF IIR (e) v_w_dist, v_w_cont: consistent labels, absent closure and extra closure. The number of consistent labels is lower than in the case of other articulator pairs due to a higher absent closure error, which means that the sounds would be produced nasalized. (f ) v_w_dist, v_w_cont: consistent labels, absent closure and extra closure. The frequency of absent closures is higher than that of extra closures. pigure RFQRX @gontFA ertiultory prmeter onsisteny in the synthesized 261 sentenes"ontF on the next pgeF IIT 4.3. Evaluation IIU Chapter 4. Articulatory speech synthesis from real-time MRI data (g) t_w_dist, t_w_cont: consistent labels and extra closure. No extra closure occurrences. (h) t_w_dist, t_w_cont: extra closure occurrences. No extra closure occurrences. pigure RFQRX @gontFA ertiultory prmeter onsisteny in the synthesized 261 : consistent labels, absent closure and extra closure. The results of both speakers are quite comparable except for an increase in lip closure when producing a sound requiring the lips to be open in : absent and extra closure occurrences. The results are inconclusive. pigure RFQSX ertiultory prmeter onsisteny in the synthesized 261 sentenesD roken down y spekers"ontF on the next pgeF IIW (c) t_v_dist, t_v_cont: consistent labels, absent closure and extra closure.

  : absent and extra closure occurrences. The frequency of the tongue and the velum missing a contact is as high as : consistent labels, absent closure and extra closure. The number of consistent labels is higher for S gontFA ertiultory prmeter onsisteny in the synthesized 261 sentenesD roken down y spekers"ontF on the next pgeF IPP (g) t_w_dist, t_w_cont: no extra closure. (h) t_w_dist, t_w_cont: no extra closure occurrences. pigure RFQSX @gontFA ertiultory prmeter onsisteny in the synthesized 261 sentenesD roken down y spekersF IPQ vip protrusionD tooD di'ered etween protruded vowels nd not"S A nd S B togetherX uplprotrX 11.84 ± 1.91 protrudedD 11.49 ± 1.69 notY lwlprotrX 12.69 ± 1.79 protrudedD 12.45 ± 1.61 notF

  éblouit le veau et les pioupious qui sautaient à une encablure du Cher, DTW = 41.14, the timing and values are quite well-aligned.

Chapter 4 .

 4 Articulatory speech synthesis from real-time MRI data (b) t_v_dist, S A ,Il a pourri, DTW = 22.55. The peaks are more moderate.

A

  ,Il éblouit le veau et les pioupious qui sautaient à une encablure du Cher, DTW =

  ls_dist, ls_cont: consistent labels, absent closure and extra closurethe original counts. (b) ls_dist, ls_cont: consistent labels, absent closure and extra closurethe synthesis counts. pigure RFQUX ertiultory prmeter onsisteny ompred on the originl sentenes tht were tken out nd the generted onesD roken down y spekers"ontF on the next pgeF IQI (c) ls_dist, ls_cont: absent and extra closure occurrences the original counts. (d) ls_dist, ls_cont: absent and extra closure occurrencesthe synthesis counts.

  t_v_dist, t_v_cont: consistent labels, absent closure and extra closurethe synthesis counts.

  hxxEsed rtiultory speeh synthesizerD where rtiultion ws extrted fully utomtilly t omprehensive rte nd modeled jointly with the oustis of the ligned signlF ustntil work ws onduted to represent rtiultory informtion in onsistent wy suitle for the networkD iFeF s set of eight prmeters orresponding to the lips opening nd ontt surfeD the distne etween the velum nd the tongue nd the tegoril prmeter stting the presene or sene of ontt etween themD the distne etween the velum nd the phryngel wll nd its own tegoril prmeterD nd the distne etween the tongue nd the phryngel wllF hese prmeters should therefore e le to represent numerous rtiultory e'ets in speeh produtionX pull ehvior of the midEsgittl slie of the lipsX lips protrusion nd openingY xslityD represented through the distne nd ontt etween the velum nd the phE ryngel wllY rtil informtion of the velr regionX no notion of the velum nd tongue positionD ut set of vlues tht depend on themX girumstntil evidene of the tongue height through the distne etween the tongue nd the velumY girumstntil evidene of the tongue kness through the distne etween the tongue nd the phryngel wllF he results suggest tht the originl ousti model n hndle the dded informtionD mening tht the extrted prmeters were generlly oherent with the ousti dtX oE jetive evlution for the full art setup omes up to lmost identil no art MCDD BAPD F0 RMSED F0 CORR nd VUV vluesD nd pereptully the generted smples in the two modes re indistinguishleF he generted rtiultory prmeter sequenes mth the originl ones eptly loselyF hey struggle more t ttining ontt etween the rtiultorsD whih results in redued rte of prohiited losure errors nd n inresed one in sent onttsF husD within the de(ned ojetivesD otining fullE)edged rtiultory speeh synthesizer n

  used two ws orporX lssi stti wsD the sme s in ghpter QD nd EwsD s in ghpter RF he outline of the work ws s followsX ret the stti nd the dynmi frmes to mke them omprleY tilize struturl similrityD irth mover9s distneD nd sp mthes to ompre themY erify the vlidity nd onsisteny of the otined mesuresY tudy the temporl ehvior of eh mesure nd interpret itY enlyze the identi(ed similritiesF he following setions will over these steps in greter detilF5.2.1Treating MRI and RT-MRI captures hen mthing the imges of these two dtsetsD one hs to fe severl issuesX(1) he resolution nd qulity of the imges is not the smeX 256 × 256 pixels ginst 189 × 189F purthermoreD ws is very sensitive to movementD resulting in ertin mount of lurring in the dynmi imgesF

  lgorithm for imge omprison should e le toX sdentify shred mnner of rtiultion suh sX ontt for stop or nslD nrrowing in the vol trt for fritive or liquidD the sene of ostrution for vowelD n open or lose velophryngel port for nsl or n orl soundF eny suh shred feture should rise the resulting vlue of the similrity mesure y few pointsF gpture shred ple of rtiultion nd the ritil rtiultorX identify when oth the stti nd the dynmi frmes depitD for exmpleD n lveolr onsonntD nd give points for thtF his wyD when the phonemes re identil up to the feture of voie @for exmpleD stti GkG nd dynmi GkG or stti GpG nd dynmi GbGAD they should e reognized s even more similr thn those tht shre only the ple or only the mnnerF fe sensitive to ortiultory e'etsX we would hope tht euse of the protruded lips Gp(y)G should turn out to e loser to GyG thnD syD Gp(a)G isY orD to mke nother exmpleD the tongue dorsum in Gt(i)G should lredy e rised to ntiipte the lose vowel GiG nd thus Gt(i)G should e quite similr to GiGD whih should not e the se for Gt(E)G where the tongue is lredy prepring for n open GEG positionF

  (a) A shared place of articulation is a ground to increase the similarity value: static /t(u)/ (left) and a frame of dynamic /s(ã)/ (center) sharing the alveolar region, while dierent places should not: see the /s(ã)/ and static /k(E)/ with the velar closure (right). (b) A shared manner of articulation is a ground to very slightly increase the similarity value as an indication of a similar distance at the constriction: static /s(a)/ (left) and a frame of dynamic /S(a)/ (center), being both sibilants, are somewhat similar, while dierent manners are not: see the /S(a)/ and static /t(a)/ (right).

  (a) The temporal behavior of EMD over the course of the sequence. The lower the value, the more similar the images. No evidence of articulatory information. (b) The temporal behavior of SSIM over the course of the sequence. The higher the value, the more similar the images. The shapes most resembling the static samples of /f/ appear at the transitions from /f/ to vowels. (c) The temporal behavior of SIFT over the course of the sequence. The higher the value, the more similar the images. The values anticipate the beginning of /f/, the peaks occurring right before the beginning of the phoneme. (d) The temporal behavior of SIFT l over the course of the sequence. The higher the value, the more similar the images. The spikes generally occur at the end of /f/ and can last through the vowel and the subsequent beginning of silence. Minor peaks follow the behavior of SIFT (Figure 5.7c): they pre-empt the beginning of /f/.

s

  empirilly used w CF = 0.5, w PlA = 1, w MA = 1, w CVsim = 0.3 nd w VVsim = 0.1F en illustrtive exerpt of the vlues tht n e otined with RS is given in le SFTF ITS (a) An example of correct results produced by SIFT when comparing the static /a/ with the dynamic /a/: correctly matched the alveolar region, the palate, the upper and lower lips, two matches at the chin, the tongue, the opening between the tongue dorsum and the velum, the velum itself. The only imprecise match is the space between the uvula and the back of the tongue being mapped further down the pharyngeal wall of the speaker. The total number of matches is 8, which produces the value 8/80 = 0.1000. Compare this SIFT result to the one in Fig. 5.10b, where the same dynamic frame is compared to /E/ rather than /a/. (b) An example of correct results produced by SIFT when comparing the static /E/ with the same dynamic frame as in Fig. 5.10a, /a/: just like in that example, SIFT correctly matched the palate, the upper and lower lips, bottom of the tongue and the velum. One more match at the chin appeared. Despite the fact that /a/ is central and /E/ is front, the frontal narrowing between the tongue and the palate in the static frame was matched to the central one in the dynamic. The total number of matches is 9, which produces the value 9/80 = 0.1125, which is greater than 0.1000 from comparing the theoretically closer static frame of /a/ to this dynamic frame. (c) An example of correct results produced by SIFT when comparing the static /n(i)/ with S A 's dynamic /z/: The algorithm correctly matched the upper and lower lips, the chin and the palate, and it is correct not to match the alveolar contact of /n/ with the mere narrowing of the vocal tract for the alveolar fricative /z/. However, these matches only very approximately reect the shared place of articulation for the two sounds. (d) An example of correct results produced by SIFT when comparing the static /p(a)/ with S B 's dynamic /p(i)/: The algorithm correctly matched the place of articulation, the lips, but since this is the only match identied, it will not be signicant enough to consider the sounds similar.

  An example of mixed results produced by SIFT when comparing the static /m(Õ)/ with the dynamic pause (S A will eventually start pronouncing /ku/): two correct matches at the lips, one of the absence of alveolar contact, one of the sublingual cavity, and yet mismatching the chin with the epiglottis, the back of the tongue with the dorsum and the constriction between the tongue and the velum with a similar shape in the nasal cavity. This is an example of reasonably noisy matches. (b) A worst-case scenario for SIFT, where all the matches are very noisy: matching the static /a/ against the dynamic /l(e)/ and mapping the upper lip to the pharyngeal wall and to the back of the tongue, the teeth to the vertebra, the tongue dorsum to its root and the chin to the vertebra. The situation was probably aggravated by the incorrect window.

  ITU /a/ /ã/ /b(a)/ /b(ã)/ /d(a)/ /d(i)/ /ø/ /e/ /f(E)/ /m(a)/ /o/ /K(O)/ /s(e)/ /@/

  of the est mesure orst mesure irror of the worst mesure I est nd worst similrity mesures ording to the AE 1 D AE 2 error funtions @iqution SFIPAF IUI 5.5

  Tsukanova, AnastasiiaD sonnis hourosD enstsi himorinD nd ves vprieF gn stti vol trt positions represent rtiultory trgets in ontinuous speehc wthE ing stti ws ptures ginst relEtime ws for the prenh lngugeF4 International Congress of Phonetic SciencesF PHIWF hourosD sonnisD Anastasiia TsukanovaD uryn ssievD ierreEendré uissozD nd ves vprieF owrds method of dynmi vol trt shpes genertion y omining stti Qh nd dynmi Ph ws speeh dtF4 INTERSPEECH 2019F PHIWF hourosD sonnisD tques pellingerD tens prhmD uryn ssievD erun eF tosephD ves vprieD preddy ydilleD Anastasiia TsukanovaD hirk oitD nd ierreEendré uissozF e

		2
		Background and context
	2.1	Speech production
	wultimodl elEime ws ertiultory gorpus of prenh for peeh eserhF4 INTER-
	SPEECH 2019F PHIWF
	vprieD vesD fenjmin ilieD Anastasiia TsukanovaD nd ierreEendré uissozF genterE
	line rtiultory models of the velum nd epiglottis for rtiultory synthesis of speehF4
	2018 26th European Signal Processing Conference (EUSIPCO)F siiiD PHIVF
	Tsukanova, AnastasiiaD fenjmin ilieD nd ves vprieF ertiultory peeh ynthesis
	from tti gontextEewre ertiultory rgetsF4 sn International Seminar on Speech
	ProductionD ppF QUERUF pringerD ghmD PHIUF
	vprieD vesD fenjmin ilieD ierreEendré uissozD nd Anastasiia TsukanovaF ertiE
	ultory model of the epiglottisF4 sn The 11th International Seminar on Speech Produc-
	tionF PHIUF
	V

5

  Fewer and fewer speakers distinguish between / oe/ and /Ẽ/, especially in uent speech.

	IF /p/X 4pe4D FrF 4pe4D GkapGX orl voieless ilil stopY
	PF /b/X 4rbe4D FrF 4r4D GkKabG or GkKAbGX orl voied ilil stopY
	QF /f/X 4onf irmer4D FrF 4on(rm4D GkÕfiKmeGX orl voieless liodentl fritiveY
	RF /v/X 4ve4D FrF 4ellr4D Gka:vGX orl voied liodentl fritiveY
	SF /t/X 4tissu4D FrF 4fri4D GtisyGX entrl orl voieless lminl dentiElveolr 7 stopY
	TF /d/X 4deu4D FrF 4present4D GkadoGX entrl orl voied lminl dentiElveolr 8 stopY
	UF /s/X 4symole4D FrF 4symol4D GsẼbOlGX entrl orl voieless lminl lveolr dentlized 9
	siilnt fritiveY
	VF /z/X 4gz4D FrF 4gs4D Gga:zG or GgA:zGX entrl orl voied lminl lveolr dentlized 10
	siilnt fritiveY
	WF /S/X 4chômge4D FrF 4unemployment4D GSoma:ZGX entrl orl voieless pltoElveolr
	lilized 11 siilnt fritiveY
	IHF /Z/X 4âge4D FrF 4ge4D GA:ZGX entrl orl voied pltoElveolr lilized 12 siilnt
	fritiveY
	IIF /k/X 4occuper4D FrF 4oupy4D GOkypeGX entrl orl voieless velr stopY
	IPF /g/X 4glol4D FrF 4glol4D GglObalGX entrl orl voied velr stopY
	IQF /m/X 4munir4D FrF 4provide4D Gmyni:KGX ilil voied nslY
	IRF /n/X 4nsl4D FrF 4nsl4D GnazalGX lminl dentiElveolr 13 voied nslY
	ISF /l/X
	6 Though it should be noted that it does play a role in listening comprehension [FS97].
	PS

4llumer4D FrF 4light4D GalymeGX lterl orl pil lveolr 14 lterl voied liquid pproximntY ITF /K/X 4rien4D FrF 4nothing4D GKjẼGX entrl orl voied uvulr liquid fritive 15 Y

  Sometimes [l "] or [ l] occur. /l/ can come voiced or voiceless depending on its position in the word, but these dierences do not carry any meaning. 15 Other regional or contextual options: the voiceless uvular fricative [X] and the uvular trill [ R].

	IUF /ñ/ 16 X 4fourguignon4D FrF 4furgundin4D GbuKgiñÕGX pltl voied nslY
	IVF /w/X 4oui4D FrF 4yes4D GwiGX the entrl orl lioEvelr voied pproximnt"semivowel
	glideD orresponding to the lose vowel GuGY
	7 So, pronounced as [t "].
	8 So, pronounced as [d "].
	9 So, realised as [s "].
	10 So, pronounced as [z "].
	11 So, pronounced as [S w ].
	12 So, pronounced as [Z w ].
	13 So, pronounced as [n "].
	14 Carried out as [l "]. 16 Currently /ñ/ seems to be merging with /nj/ [GLM12].

  utternesD 52.65 minutes of speeh for speker S A nd 917 utternesD 59.53 minutes of speeh for speker S B F he proportions of rndom division of the dt for model triningD vlidtion nd testing were 907, 97 nd 17 respetivelyF ell input prmeters were normlized efore eing proessed y the networkF

	Linguistic specication
	o extrt linguisti fetures from the lelsD s identi(ed potentilly signi(nt phoneti pheE
	nomen tht were mrked in the r lelsX
	pei( phonemeY	M protr [par] ± σ protr [par]	M not protr [par]	±
	σ not protr [par] qroups of phonemes exhiiting llophoni vritionD suh s GeG nd GEG or GoG nd GOGY
	uplprotr IQFSS ± PFQW uplprotr IQFTW ± IFVT honeti lss" feture per eh position in the quinphoneX vowel nd its type @openD IQFHQ ± PFPP S A D nonEsp IPFWT ± IFTP S A D spont uplprotr IIFSH ± IFRV IIFIS ± IFRP losedD etFY frontD kD etFY rounded or unroundedAY onsonnt nd its type @voied or S B D nonEsp uplprotr IPFTR ± HFWW IPFHT ± IFHH unvoiedY stopD fritiveD nslD etFY rtiulted t the frontD enter or the k of the S B D spont S A D nonEsp lwlprotr IPFIV ± PFPP IIFTH ± PFIH vol trtAY
	S A D spont S B D nonEsp osition of the phoneme in the syllleD of the syllle in the wordD of the word in the lwlprotr IPFTP ± IFVP IIFRW ± IFRT lwlprotr IQFIU ± IFVV IPFUI ± IFUV S B D spont lwlprotr IQFTI ± IFQU IPFIR ± IFQV phrseX n ext numer s well s within rngeY
	S A D ll xumer of phonemes in the syllleD of syllles in the wordD of words in the phrseX n uplprotr IQFSU ± PFQP IQFHI ± PFHV
	S B D ll ext numer s well s within rngeY uplprotr IIFTP ± IFRU	IIFQS ± IFQW
	S A D ll	lwlprotr IPFPR ± PFIV	IIFSU ± IFWS
	S B D ll tress nd ent in the urrent syllle s well s in the neighorsY lwlprotr IQFPP ± IFVQ IPFSW ± IFUP
	S A+B D nonE sp histne to the stressed G ented syllles forwrds nd kwrdsY uplprotr IPFRI ± PFIW IIFWH ± PFHH
	S A+B D spont uplprotr IQFIS ± IFSU S A+B D nonEs lwlprotr IPFUQ ± PFIH rtEofEspeeh tgsX kwrdsD forwrds nd nowF	IPFRV ± IFRH IPFPU ± IFWW
	S A+B D spont lwlprotr IQFIP ± IFTV S A+B D ll uplprotr IPFRV ± PFHW Duration model	IIFVR ± IFRT IIFWT ± IFWS
	S A+B D ll he durtion model ws feedEforwrd 6Elyer networkD eh lyer with hyperoli tngent lwlprotr IPFUV ± PFHT IPFIU ± IFVV
	tivtion @tanhAF he hoie of the tivtion funtion ws done to llow the model prmeters le RFPX vip protrusion is shown to e di'erent on the vowels to require protrusion @lrger to updte regulrly nd void the model prmeters getting stukF uplprotrD lwlprotr vluesA nd those tht do not @smller uplprotrD lwlprotr he th size ws 64D the lerning rte ws 0.002 with n exponentil deyD 25 trining vluesAF epohsF
	he output of the model is the vlue dur for durtionF
	4.2.2	Implementation
	he speeh synthesizer ws uilt using werlin s frontend uIT sed o' their stndrd
	uild your own voie4 reipe nd using yvh vooderD for eh speker seprtelyF efter

exluding the phrses where there ws no phoneti leling @due to ugs in elite rA or rtiultory prmeters @whenD for exmpleD the rtiultory sequene ws to e estimted sed o' too few smples tht were leled s phoneme rther thn silene or puseA villeD s IHI otined 618

  le RFQX yjetive evlution of the no art nd full art setups of the oustiErtiultory modelsF MCDX men melEepstrum distortion @the distortion etween the generted sequene nd the trget oneAF BAPX bap @nd periodiityA predition errorF F0 RMSEX root men squre of pHF CORRX pH orreltion oe0ientF VUVX frmeElevel voiedGunvoied errorF he full art setup tkes slightly worse vlues tht ould e explined y the ft tht the network size of full art styed the sme while mnging more prmeters"mye it needed one more epohY howeverD S B 9s F0 RMSE nd F0 CORR were slightly improved in full artF sn generlD S A 9s results re etter thn S B 9sF

						4.3. Evaluation
	no art	S A	hev 4.871 df	0.116 df	16.176 rz 0.740	8.8597
	no art	S A	est 4.654 df	0.131 df	13.669 rz 0.840	6.2537
	full art S A	hev 4.878 df	0.119 df	16.131 rz 0.743	8.6937
	full art S A	est 4.709 df	0.135 df	15.785 rz 0.781	6.2787
	no art	S B	hev 5.352 df	0.133 df	18.645 rz 0.516	7.6317
	no art	S B	est 5.399 df	0.157 df	15.304 rz 0.549	10.7257
	full art S B	hev 5.390 df	0.132 df	18.530 rz 0.504	8.2007
	full art S B	est 5.498 df	0.161 df	14.777 rz 0.563	11.5677
	IIH

  the orresponding sentenes in the orpus @synth to corpAD ompred to the sme distnes etween the originl sentenes @intracorpAF hile synth to corp distnes re greter thn intracorpD vrine is very highD so they still fll within the eptle rngeF dist prmeters get synthesized loser to the orpus vlues for S A Y contD for S B F

				4.3. Evaluation
				dtw]
	lsdist	S A	49.67 ± 16.86	39.28 ± 18.53
	lsdist	S B	47.55 ± 13.88	32.83 ± 19.37
	lscont	S A	20.96 ± 13.10	15.51 ± 10.23
	lscont	S B	12.85 ± 9.75	11.08 ± 12.81
	tvdist	S A	47.10 ± 15.91	35.47 ± 18.28
	tvdist	S B	58.64 ± 19.47	46.09 ± 27.90
	tvcont	S A	7.92 ± 5.34	4.80 ± 5.68
	tvcont	S B	3.41 ± 4.64	2.18 ± 3.77
	vwdist	S A	32.63 ± 23.49	29.78 ± 17.09
	vwdist	S B	36.31 ± 19.64	31.69 ± 17.82
	vwcont S A	7.30 ± 2.16	4.94 ± 3.15
	vwcont S B	10.50 ± 2.61	6.90 ± 4.64
	twdist	S A	63.82 ± 15.93	26.48 ± 12.82
	twdist	S B	59.72 ± 10.93	37.73 ± 15.67
	twcont S A	1.02 ± 0.44	0.00 ± 0.00
	twcont S B	2.76 ± 3.07	3.24 ± 4.02
	le RFRX he men dynmiEtimeEwrping distne etween the generted rtiultory pE
	rmeter sequene nd IPR

  EMDD SSIMD SIFT nd SIFT l vlues ross the entire set of imge omprisonsF fetween ny two mesures not involving EMDD positive orreltion is inditive of n greement in judgmentF ith EMDD it is the reverse sine EMD is lower for similr imges while the others re higherF xone of the similrity mesures followed the norml distriutionX EMDX ISI hpiroEilk9s test @hAX sttisti 0.963 for S A nd 0.975 for S B D pEvlue 0.000 for othD h9egostino9s K 2 test @heAX 131, 032.833 for S A D 218, 447.082 for S B D pEvlue 0.000Y SSIMX hpiroEilk9s test @hAX sttisti 0.999 for S A nd 0.998 for S B D pEvlue 0.000 for othD h9egostino9s K 2 test @heAX 2, 272.411 for S A D 4492.322 for S B D pEvlue 0.000Y SIFT X hpiroEilk9s test @hAX sttisti 0.966 for S A nd 0.977 for S B D pEvlue 0.000 for othD h9egostino9s K 2 test @heAX 107, 402.547 for S A D 108, 058.162 for S B D pEvlue 0.000Y SIFT

		0016 ± 0.0003	0.0350 ± 0.0331	0.0665 ± 0.0317	0.0115 ± 0.0989
	S B	0.0018 ± 0.0002	0.0367 ± 0.0297	0.0613 ± 0.0299	0.0080 ± 0.0346
	le SFPX wens @EA nd stndrd devitions @SDA of the imge similrity mesuresD spekers
	S A nd S B F
		wesure	EMD	SSIM	SIFT	SIFT l
			S A S B S A	S B	S A	S B	S A	S B
		EMD		HFRP EHFHP HFHU HFHH HFHI EHFHI
		SSIM				HFHV HFHT HFHI HFHH
		SIFT					EHFHW EHFPS
	le SFQX gorreltions etween

l X hpiroEilk9s test @hAX sttisti 0.039 for S A nd 0.135 for S B D pEvlue 0.000 for othD h9egostino9s K 2 test @heAX 13, 912, 105.949 for S A D 11, 653, 781.164 for S B D pEvlue 0.000Y

  que sp et l similrité struturelle pturient ien les informtions rtiuE ltoires et que leur omportementD de mnière générleD vlidient les données d9sw sttiquesF ves phonèmes et trits phonétiques prolémtiques que j9i pu identi(er à trvers les nlyses des distriutions et inomptiilités de mesures étient les liquides Gl, KGD dont l prodution dynmique ne pouvit être imitée pr leur simultion sttiqueD les fritives lvéolires Gs, SGD elles ussi simulées de mnière non réliste dns le ontexte sttiqueD et les rtéristiques de l nslitéF sl semlerit que les sons du orpus sttique ient été légèrement trop nslisésD et que réiproquement les sons nslisés présentient une ouverture vélophryngée insu0snteF in(nD j9i disuté l9impt de ette étude pour de futurs synthétiseurs rtiultoires hyrides de l proleF

	B.3	Cibles statiques et la parole en temps réel pour la synthèse
		de la parole articulatoire
	ves deux pprohes présentées iEdessus ont en ommun l9utilistion de deux types de données
	swF ge point ommun motivé l reherheD dns les données temps réelD des imges lésD
	9estEàEdire les on(gurtions sttiques swD utilisées pour modéliser l ortiultionF e(n de
	omprer les imges sw sttiques ve les imges dynmiques en temps réelD nous vons utilisé
	plusieurs mesures X
		v similrité struturelleD
		v distne du 4terrssier4D
		sp Y
	eprès voir véri(é l pertinene et l vlidité de es mesuresD j9i étudié qulittivement et
	quntittivementD puis interprété leur omportementY j9i ensuite nlysé leur similritésF
	t9en i onlu B.4 Conclusion globale
	v synthèse vole rtiultoire sée sur des règles permettit un ontrôle omplet de l9ensemle

des rtiulteurs E l mâhoireD l lngueD les lèvresD le vélumD l9épiglotte et le lrynx E insi que du pHD de l9ouverture glottleD de l pression sousEglottle et suprglottle Y ve une qulité vrile du résulttD elle pouvit ouvrir l phonologie frnçise grâe à un jeu étendu de on(gurtions IVR B.4. Conclusion globale

  wqownF eerooustis of sF sn Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering SciencesD volume RTID pges IHHS!IHPVF he oyl oietyD PHHSF rwHV uiyoshi rond nd hinji wedF qlottlEopening nd ir)ow pttern during proE dution of voieless fritivesX new nonEinvsive instrumenttionF The Journal of the Acoustical Society of AmericaD IPQ@SAXQUQV!QUQVD PHHVF rwII sn rowrd nd iers wessumF wodeling the development of pronunition in infnt speeh quisitionF Motor ControlD IS@IAXVS!IIUD PHIIF rHH qregory rikok nd hvid oeppelF owrds funtionl neurontomy of speeh pereptionF Trends in cognitive sciencesD R@RAXIQI!IQVD PHHHF rHR qregory rikok nd hvid oeppelF horsl nd ventrl stremsX frmework for understnding spets of the funtionl ntomy of lngugeF CognitionD WP@IE PAXTU!WWD PHHRF rHU qregory rikok nd hvid oeppelF he ortil orgniztion of speeh proessingF Nature reviews neuroscienceD V@SAXQWQD PHHUF IWP rTS tF wF reinz nd uF xF tevensF yn the reltions etween lterl inerdiogrphsD re funtions nd ousti spetr of speehF sn Proceedings of the 5th International Congress on AcousticsD pge eRRFD IWTSF spUP uenzo sshizk nd tmes v plngnF ynthesis of voied sounds from twoEmss model of the vol ordsF Bell system technical journalD SI@TAXIPQQ!IPTVD IWUPF sntWW snterntionl honeti essoitionF Handbook of the International Phonetic Association: A guide to the use of the International Phonetic AlphabetF gmE ridge niversity ressD IWWWF tHI ul trdF Étude omprtive de l distriution )orle dns une portion des lpes et des jurF Bull Soc Vaudoise Sci NatD QUXSRU!SUWD IWHIF tpHH unEeh tun nd géile pougeronF e phonologil model of prenh intontionF sn IntonationD pges PHW!PRPF pringerD PHHHF trde + IQ tohn tnssonD endres rolmergD odrigo ilel de ereuD gem hegirmeniD tohn ro'mnD wikel urlssonD nd wts eomF edptive stilized (nite eleE ment frmework for simultion of vol fold turulent )uidEstruture intertionF sn Proceedings of Meetings on AcousticsD volume IWD pges HQS!HRIF eoustil oiety of emeriD PHIQF tweHT ghristine tdelotD wthieu wngeotD itienne etitjenD nd usnne lmonEeltF worphlou PFHF https://www.cnrtl.fr/lexiques/morphalou/LMF-Morphalou. phpD PHHTF ynlineY essed QHEtuneEPHIWF tonST hniel tonesF The pronunciation of EnglishD volume QTWF gmridge niversity ressD IWSTF ufgWW illim p utzD neh fhrdwjD nd furkhrd grstensF iletromgneti rtiulogrphy tretment for n dult with ro9s phsi nd prxi of speehF Journal of Speech, Language, and Hearing ResearchD RP@TAXIQSS!IQTTD IWWWF uifHV itor uupermnD wirjm irnestusD nd rrld fyenF prequeny distriutions of uniphonesD diphonesD nd triphones in spontneous speehF The Journal of the Acoustical Society of AmericaD IPR@TAXQVWU!QWHVD PHHVF uqIV edvit uoprkr nd rsnt uumr qhoshF e supervised irEtissue oundE ry segmenttion tehnique in relEtime mgneti resonne imging video using novel mesure of ontrst nd dynmi progrmmingF sn 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)D pges SHHR!SHHVF siiiD PHIVF uwqIH illim p utzD wlolm wxeilD nd hine w qrstF reting prxi of speeh @osA with emEsupplied visul ugmented feedkF AphasiologyD PR@TE VAXVPT!VQUD PHIHF uxWW frr uühnert nd prnis xolnF he origin of ortiultionF Coarticulation: Theory, data and techniquesD pges U!QHD IWWWF IWQ urHS wihel r urneF eeroousti prodution of lowEfrequeny unvoied speeh soundsF The Journal of the Acoustical Society of AmericaD IIV@IAXRIH!RPUD PHHSF uHP idith un nd mr wF he rin iruitry of syntti omprehensionF Trends in cognitive sciencesD T@VAXQSH!QSTD PHHPF uuwIQ tuhni unuutiD entti rsteD wrko ullioD nd reikki winnF ss rdi mgneti resonne imging using hxe dmgec European heart journalD QR@QHAXPQQU! PQQWD PHIQF vvWR tohn vverF Principles of phoneticsF gmridge niversity ressD IWWRF vfHP idwrd voper nd teven firdF xvuX the nturl lnguge toolkitF arXiv preprint cs/0205028D PHHPF vfII F vprie nd tF fussetF gonstrution nd evlution of n rtiultory model of the vol trtF sn 19th European Signal Processing Conference -EUSIPCO-2011D frelonD pinD eugust PHIIF vfII F vprie nd tF fussetF gonstrution nd evlution of n rtiultory model of the vol trtF sn 19th European Signal Processing Conference -EUSIPCO-2011D frelonD pinD eugust PHIIF vhIP eter vdefoged nd ndr perrri hisnerF Vowels and consonantsF tohn iley 8 onsD PHIPF viIS ves vprieD fenjmin ilieD nd enstsii suknovF Ph rtiultory velum modeling pplied to opy synthesis of sentenes ontining nsl phonemesF sn International Congress of Phonetic SciencesD PHISF viIV ves vprieD fenjmin ilieD enstsii suknovD nd ierreEendré uissozF genE terline rtiultory models of the velum nd epiglottis for rtiultory synthesis of speehF sn 2018 26th European Signal Processing Conference (EUSIPCO)D pges PIIH!PIIRF siiiD PHIVF vevTS ldimir s vevenshteinF finry odes ple of orreting deletionsD insertionsD nd reverslsF USSR's Reports of Science AcademiesD ITQ@RAXVRS!VRVD IWTSF vinWI igung vinF peeh prodution theory nd rtiultory speeh synthesisF The Journal of the Acoustical Society of AmericaD WH@RAXPPHQ!PPHQD IWWIF vtIR eter vdefoged nd ueith tohnsonF A course in phoneticsF gengge lerningD PHIRF vvUH ierre véon nd wonique véonF Introduction á la phonétique corrective á l'usage des professeurs de français á l'étrangerF rhetteD IWUHF vwWV eter vdefoged nd sn wddiesonF he sounds of the world9s lngugesF Lan-guageD UR@PAXQUR!QUTD IWWVF IWR vonVR pF vonhmpF ves sons du prnçis " enlyse oustique desriptiveF gours de phonétiqueD snstitut de honétiqueD niversité de xny ssD IWVRF vowWW hvid q voweF yjet reognition from lol sleEinvrint feturesF sn Computer vision, 1999. The proceedings of the seventh IEEE international conference onD volume PD pges IISH!IISUF seeeD IWWWF vowHR hvid q voweF histintive imge fetures from sleEinvrint keypointsF International journal of computer visionD TH@PAXWI!IIHD PHHRF vxIU edm g vmmertD homs p utieriD ghristine r hdleD nd hriknth xrynnF peed ury trdeo's in speeh produtionF ehnil reportD ws vsnoln vortory vexington nited ttesD PHIUF vwWW illem tw veveltD erdi oelofsD nd entje weyerF e theory of lexil ess in speeh produtionF Behavioral and brain sciencesD PP@IAXI!QVD IWWWF v + IQ edm g vmmertD ikrm mnrynnD wihel s rotorD hriknth xrynnD et lF ol trt rossEdistne estimtion from relEtime ws usE ing regionEofEinterest nlysisF sn InterspeechD pges WSW!WTPD PHIQF vpIP tohn i vloydD sn tvnessD nd idney pelsF ertisynthX fst intertive iomeE hnil modeling toolkit omining multiody nd (nite element simultionF sn Soft tissue biomechanical modeling for computer assisted surgeryD pges QSS! QWRF pringerD PHIPF vqHR omry vFD lmonEelt FD nd prnopoulo qF tndrds going onrete X from vwp to worphlouF sn Workshop on Electronic Dictionaries, ColingD qenevD witzerlndD PHHRF vxIV edm g vmmertD ghristine r hdleD hriknth xrynnD nd homs p utieriF peedEury trdeo's in humn speeh produtionF PloS oneD IQ@WAXeHPHPIVHD PHIVF vgIR F vprieD fF xelireD nd wF gdotF qeometri rtiultory model dpted to the prodution of onsonntsF sn 10th International Seminar on Speech Production (ISSP)D uölnD ellemgneD wy PHIRF vX http://hal.inria.fr/ hal-01002125F vv + IW ongwn vimD inghu huD jn qoud vinglD hni fyrdD hriknth xrynnD nd urishn hrinivs xykF Qd dynmi mri of the vol trt during nturl speehF Magnetic resonance in medicineD VI@QAXISII!ISPHD PHIWF weWH F wedF gompenstory rtiultion during speehX ividene from the nlysis nd synthesis of volEtrt shpes using n rtiultory modelF sn FtF rrdstle nd eF wrhlD editorsD Speech production and speech modellingD pges IQI!IRWF uluwer edemi ulisherD emsterdmD IWWHF IWS weWH F wedF gompenstory rtiultion during speehX ividene from the nlysis nd synthesis of volEtrt shpes using n rtiultory modelF sn F tF rrdsE tle nd eF wrshlD editorsD Speech Production and Speech ModellingF uluwer edemi ulishersD IWWHF wrWR etros wrgosF prtl signl nlysis using mthemtil morphologyF sn F rwkes nd fF uznD editorsD Advances in Electronics and Electron PhysicsD volume VVD hpter RD pges IWW!PRTF edemi ressD IWWRF wgWH iri woulines nd prnis ghrpentierF ithEsynhronous wveform proessing tehniques for textEtoEspeeh synthesis using diphonesF Speech communicationD W@SETAXRSQ!RTUD IWWHF wqVV ihrd wqownF en eroousti pproh to phontionF The Journal of the Acoustical Society of AmericaD VQ@PAXTWT!UHRD IWVVF wgyII tpl wustiD inent golotteD esterios outiosD nd lim yuniF sntroduing visul trget ost within n oustiEvisul unitEseletion speeh synthesizerF sn Auditory-Visual Speech Processing 2011D PHIIF werUQ F wermelsteinF ertiultory model for the study of speeh produtionF Journal of the Acoustical Society of AmericaD SQXIHUH!IHVPD IWUQF wsqqWW perdinndo e wussEsvldiD x qnthevD nd q qnthevF wotor primitivesD foreE(elds nd the equilirium point theoryF From Basic Motor Control to Functional Recovery. Academic Publishing House" Prof. M. Drinov", Soa, BulgariaD pges QWP!QWVD IWWWF wtfIP FF wqownD wFF tksonD nd wFeF fergerF enlyses of vol trt rossE distne to re mppingX n investigtion of set of vowel imgesF Journal of the Acoustical Society of AmericaD IQI@IAXRPR!RQRD PHIPF wup + IH w wxeilD p utzD h possettD hw qrstD xt zuminskyD q grterD nd u vimF i'ets of online ugmented kinemti nd pereptul feedk on tretE ment of speeh movements in prxi of speehF Folia Phoniatrica et Logopaed-icaD TP@QAXIPU!IQQD PHIHF wvIQ hinji wed nd ves vprieF owel nd prosodi ftor dependent vritions of volEtrt lengthF sn InterSpeech -14th Annual Conference of the International Speech Communication Association -2013D vyonD prneD eugust PHIQF vX http://hal.inria.fr/hal-00836829F wvIU wrius wuj nd hvid q voweF pst lirry for pproximte nerest neighorsF Dosegljivo: https://github. com/mariusmuja/annD PHIUF wr + IU tefni wrinD wrinne ouplierD hilip rooleD wnfred stätterD vsse fomienD son ghitornD nd elexei uohetovF owrds typology of onsoE nnt ortiultionX guging the spe etween universl nd lngugeEspei( ptterns of onsonnt timingF sn IPS Workshop on Abstraction, Diversity, and Speech DynamicsD PHIUF IWT w + IH rF xm F witrD wF iedeD iF ltzmnD vF qoldsteinD gF ipsyEilsonD nd wF rsegwEtohnsonF e proedure for estimting gesturl sores from nturl speehF sn 11th Annual Conference of the International Speech Communication Association -INTERSPEECH 2010D wkuhriD ghiD tpnD PHIHF wvSV ul woore nd rns on vedenF hynmi vritions of the virtory pttern in the norml lrynxF Folia Phoniatrica et LogopaedicaD IH@RAXPHS!PQVD IWSVF wpIW hesish y wohptrD itor ppiD nd idney pelsF en extended twoE dimensionl vol trt model for fst ousti simultion of singleExis symmetri threeEdimensionl tuesF arXiv preprint arXiv:1909.09585D PHIWF xfq + II hriknth xrynnD irik freshD rsnt uumr qhoshD vouis qoldsteinD ethnsios utsmnisD oon uimD edm vmmertD wihel rotorD ikrm mnrynnD nd inghu huF e multimodl relEtime ws rtiultory orE pus for speeh reserhF sn Twelfth Annual Conference of the International Speech Communication AssociationD PHIIF xwrt + IP rF xmD F witrD wF rsegwEtohnsonD gF ipsyEilsonD iF ltzmnD nd vF qoldsteinF e proedure for estimting gesturl sores from speeh oustisF Journal of the Acoustical Society of AmericaD IQP@TAXQHVH!QWVWD PHIPF xw + IP rosung xmD ikrmjit witrD wrk iedeD wrk rsegwEtohnsonD grol ispyE ilsonD illiot ltzmnD nd vouis qoldsteinF e proedure for estimting gesturl sores from speeh oustisF The Journal of the Acoustical Society of AmericaD IQP@TAXQWVH!QWVWD PHIPF x + IR hriknth xrynnD esterios outiosD ikrm mnrynnD edm vmmertD tngwon uimD ungok veeD urishn xykD oonEghul uimD inghu huD vouis qoldsteinD et lF elEtime mgneti resonne imging nd eletromgneti rE tiulogrphy dtse for speeh prodution reserh @tAF The Journal of the Acoustical Society of AmericaD IQT@QAXIQHU!IQIID PHIRF xu + IQ eron xieergllD huo hngD isther uunyD qötz ueydnD wihel toD wrE tin ekerD nd tens prhmF elEtime mri of speking t resolution of QQ msX ndersmpled rdil )sh with nonliner inverse reonstrutionF Magnetic Resonance in MedicineD TW@PAXRUU!RVSD PHIQF ÖhmTT FiF ÖhmnF gortiultion in g utternesX petrogrphi mesurementsF Journal of the Acoustical Society of AmericaD QW@IAXISI!ITVD IWTTF ÖhmTU ven iq ÖhmnF xumeril model of ortiultionF The Journal of the Acoustical Society of AmericaD RI@PAXQIH!QPHD IWTUF ypeIQ ypentx gollegeF yrgns nd strutures of the respirtory systemF http://cnx. org/contents/t2sgkCQ-@8/Organs-and-Structures-of-the-RD PHIQF yfIP elexey yzerovD immnuel inentD nd prédéri fimotF e generl )exile frmeE work for the hndling of prior informtion in udio soure seprtionF IEEE IWU Transactions on Audio, Speech, and Language ProcessingD PH@RAXIIIV!IIQQD PHIPF fIS imon reuÿ nd eter firkholzF yptil sensor lirtion for eletroEoptil stomtogrphyF sn Sixteenth Annual Conference of the International Speech Communication AssociationD PHISF fuxIH wihel s rotorD hniel foneD ethnsios utsmnisD nd hriknth xrynnF pid semiEutomti segmenttion of relEtime mgneti resonne imges for prmetri vol trt nlysisF sn Eleventh Annual Conference of the International Speech Communication AssociationD PHIHF fvIQ tonthn v restonD xikole frikD nd xiole vndiF ltrsound iofeedk tretment for persisting hildhood prxi of speehF American Journal of Speech-Language PathologyD PHIQF erIU sl errierF ht gols for rtiultory speeh synthesisc sn The 11th International Seminar on Speech ProductionD PHIUF vwIU tonthn v restonD wegn g veeeD nd idwin wsF wotorEsed tretment with nd without ultrsound feedk for residul speehEsound errorsF International Journal of Language & Communication DisordersD SP@IAXVH!WRD PHIUF wg + IR tonthn v restonD trii wgeD ehmed iverEgmposD tessi v hittleD irik vndryD nd idwin wsF ltrsound visul feedk tretment nd prtie vriility for residul speeh sound errorsF Journal of Speech, Language, and Hearing ResearchD SU@TAXPIHP!PIISD PHIRF IR wnfred stätter nd wrinne ouplierF he rtiultory modeling of germn oronl onsonnts using tdF sn Proceedings of the 10th International Seminar on Speech ProductionD pges QHV!QIID PHIRF IS wnfred stätter nd wrinne ouplierF ynsetEvowel timing s funtion of ortiultion resistneX ividene from rtiultory dtF sn ICPhSD PHISF g + WT vier elorsonD g esoviD i gstelliD e rirshergD et ijnndsD nd rwe fillietF hesription of the )ow through inEvitro models of the glottis during phontionF pplition to voied sounds synthesisF Acta Acustica united with AcusticaD VP@PAXQSV!QTID IWWTF ffhIR ophie oekhutD ndrine frognuxD ihrd feufortD nd hierry hutoitF eviteErX n outil ev pour l génértion de synthèse hmm en frnçisF sn Démonstration aux Journées d'étude de la parole (JEP)D PHIRF wgIW inwn niD qri0n wilspD nd xthn i groneF he potentil for speeh rin!omputer interfe using hroni eletroortiogrphyF NeurotherapeuticsD IT@IAXIRR!ITSD PHIWF IWV r + HU xiols utyD vier elorsonD ennemie n rirtumD snes vopezEertegD nd evrhm rirshergF en in vitro setup to test the relevne nd the ury of lowEorder vol folds modelsF The Journal of the Acoustical Society of AmericaD IPI@IAXRUW!RWHD PHHUF gIQ eyn eesyD ylvi uedD tyrm u dupD nd tohn golemnF eutomti segmenttion of vol trt w imgesF sn 2013 IEEE 10th International Symposium on Biomedical ImagingD pges IQPV!IQQIF siiiD PHIQF qHH ossi unerD grlo omsiD nd veonids t quisF he erth mover9s distne s metri for imge retrievlF International journal of computer visionD RH@PAXWW! IPID PHHHF + IV ikrm mnrynnD m ilsenD wihel rotorD tohnnes ögerD vouis qoldsteinD urishn xykD nd hriknth xrynnF enlysis of speeh proE dution relEtime mriF Computer Speech & LanguageD PHIVF xIT ikrm mnrynnD wrten n egroekD nd hriknth xrynnF hiE retly dtEderived rtiultory gestureElike representtions retin disrimintory informtion out phone tegoriesF Computer speech & languageD QTXQQH!QRTD PHITF yIQ gmille v ynF vnguge use in the united sttesX PHIIF Economics and Statistics AdministrationD PHIQF + VS toshi uzuki et lF opologil struturl nlysis of digitized inry imges y order followingF Computer vision, graphics, and image processingD QH@IAXQP! RTD IWVSF fHV entoine errurier nd ierre fdinF e threeEdimensionl rtiultory model of the velum nd nsophryngel wll sed on mri nd t dtF The Journal of the Acoustical Society of AmericaD IPQ@RAXPQQS!PQSSD PHHVF fIT imon tone nd eter firkholzF engle orretion in optopltogrphi tongue distne mesurementsF IEEE Sensors JournalD IU@PAXRSW!RTVD PHITF hPI illim v hwrtzF yllition in prenh nd suggestions for enting the letter iF The Modern Language JournalD S@UAXQUR!QUUD IWPIF qfVV i ltzmnD v qoldsteinD g frowmnD nd uinF wodeling speeh prodution using dynmi gesturl struturesF The Journal of the Acoustical Society of AmericaD VR@IAXIRT!IRTD IWVVF uVU illiot ltzmn nd te uelsoF killed tionsX tskEdynmi pprohF Psychological reviewD WR@IAXVRD IWVUF vwhHP eF oquetD F veuitD F wetensD nd hF hemolinF widEsgittl ut to re funtion trnformtionsX hiret mesurements of midEsgittl distne nd re with wsF Speech CommunicationD QT@QERAXITW!IVHD wrh PHHPF IWW wVW illiot v ltzmn nd uevin q wunhllF e dynmil pproh to gesturl ptE terning in speeh produtionF Ecological psychologyD I@RAXQQQ!QVPD IWVWF wfIV imon toneD wihel wrxenD nd eter firkholzF gonstrution nd evlution of prmetri oneEdimensionl vol trt modelF IEEE/ACM Transactions on Audio, Speech, and Language ProcessingD PT@VAXIQVI!IQWPD PHIVF wgHQ tephnie wF trsselD hvid willerD uevin lkerD nd ghristopher gieriF hred resoures for roust speehEtoEtext tehnologyF sn 8th European Conference on Speech Communication and Technology, EUROSPEECH 2003 -INTER-SPEECH 2003, Geneva, Switzerland, September 1-4, 2003D PHHQF vX http://www.isca-speech.org/archive/eurospeech_2003/e03_1609.htmlF xeIQ hnny h teinergD riroshi xgtD nd hvid elineF Psycholinguistics: Language, mind and worldF outledgeD PHIQF h + HI onld g hererD houd hinwriD uenneth t he ittD gho hngD fogdn uuinshiD nd edollh e efjehF sntrglottl pressure pro(les for symmetri nd olique glottis with divergene ngle of IH degreesF The Journal of the Acoustical Society of AmericaD IHW@RAXITIT!ITQHD PHHIF + IU nner orensenD isis sson kordilisD esterios outiosD oonEghul uimD inghu huD tngwon uimD edm g vmmertD ikrm mnrynnD vouis qoldsteinD hni fyrdD et lF htse of volumetri nd relEtime vol trt ws for speeh sieneF sn INTERSPEECHD pges TRS!TRWD PHIUF toIQ frd r toryF hrseElevel speeh simultion with n irwy modultion model of speeh produtionF Computer speech & languageD PU@RAXWVW!IHIHD PHIQF xIU isis sson kordilisD esterios outiosD tohnnes ögerD nd hriknth xrynnF istimtion of vol trt re funtion from volumetri mgneti resonne imgE ingF sn Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference onD pges WPR!WPVF siiiD PHIUF rIW yn tokesD tonthn r eneziD nd qregory rikokF he motor system9s modest ontriution to speeh pereptionF Psychonomic bulletin & reviewD PHIWF p + IV hihu uD tinguo eiD ing pngD tinrong ngD nd uiyoshi rondF ongue segmenttion with geometrilly onstrined snke modelF sn InterspeechD pges QIIU!QIPID PHIVF zeIH ihrd zeliskiF Computer Vision: Algorithms and ApplicationsF pringerE erlgD ferlinD reidelergD Ist editionD PHIHF ivIU enstsii suknovD fenjmin ilieD nd ves vprieF ertiultory speeh synE thesis from stti ontextEwre rtiultory trgetsF sn International Seminar on Speech ProductionD pges QU!RUF pringerD PHIUF PHH qr + IW rironori kemotoD sus qotoD uy rgihrD yk rmnkD tsuy uitmurD ukiko xotD nd uikuo wekwF peeh orgn ontour extrtion using elEime ws nd mhine lerning methodF Proc. Interspeech 2019D pges WHR!WHVD PHIWF rw + HT rF kemotoD uF rondD F wskiD F himdD nd sF pujimotoF wesurement of temporl hnges in vol trt re funtion from Qd ineEmri dtF Journal of the Acoustical Society of AmericaD IIW@PAXIHQU!IHRWD PHHTF hoVT t homsF e (nite element model of )uid )ow in the vol trtF Computer Speech & LanguageD I@PAXIQI!ISID IWVTF itUQ sngo itzeF he humn vol ordsX mthemtil modelF PhoneticaD PV@QE RAXIPW!IUHD IWUQF wWV grlo omsi nd oerto wnduhiF filterl (ltering for gry nd olor imgesF sn IccvD volume ID pge PD IWWVF xIS esterios outios nd hriknth xrynnF ptor nlysis of volEtrt outlines derived from relEtime mgneti resonne imging dtF sn ICPhSD PHISF xIT esterios outios nd hriknth xrynnF edvnes in relEtime mgneti resonne imging of the vol trt for speeh siene nd tehnology reserhF APSIPA Transactions on Signal and Information ProcessingD SD PHITF x + IQ ueiihi okudD oshihiko xnkkuD kehi odD reishun enD tunihi mE gishiD nd ueiihiro yurF peeh synthesis sed on hidden mrkov modelsF Proceedings of the IEEED IHI@SAXIPQR!IPSPD PHIQF vX http://dx.doi.org/ 10.1109/JPROC.2013.2251852D doi:10.1109/JPROC.2013.2251852F rVU fernrd rnelF The sounds of French: An introductionF gmridge university pressD IWVUF + IT esterios outiosD nner orensenD urishn omndeplliD hel elexnderD nd hriknth xrynnF ertiultory synthesis sed on relEtime mgneti resoE nne imging dtF sn INTERSPEECHD pges IRWP!IRWTD PHITF WH rerert wF eger nd hushn wF egerF ividene for nonliner sound produE tion mehnisms in the vol trtF sn Speech production and speech modellingD pges PRI!PTIF pringerD IWWHF + IH wrtin ekerD huo hngD hirk oitD elexnder urusD ulusEhietmr werE oldtD nd tens prhmF elEtime ws t resolution of PH msF NMR in BiomedicineD PQ@VAXWVT!WWRD PHIHF + WU tohn g ells et lF mpX omputer redle phoneti lphetF Handbook of standards and resources for spoken language systemsD RD IWWUF fHR hou ngD eln g fovikD rmid heikhD nd iero imonelliF smge qulity ssessmentX from error visiility to struturl similrityF IEEE transactions on image processingD IQ@RAXTHH!TIPD PHHRF PHI igIW wtthew lenskiD idurdo iuropD hvid gplnD nd gynthi u hompsonF xeurl networks for sentene omprehension nd produtionX en leEsed metE nlysis of neuroimging studiesF Human brain mappingD PHIWF rIT qrnt w lker nd qregory rikokF fridging omputtionl pprohes to speeh produtionX he semnti!lexil!uditory!motor model @slmAF Psychonomic bulletin & reviewD PQ@PAXQQW!QSPD PHITF rIV qex evui nd qiqy rsguyuF peeh prodution integrting psyE holinguistiD neurosieneD nd motor ontrol perspetivesF The Oxford Handbook of PsycholinguisticsD PHIVF rIW itor etzelD homs rélieD nd prie ilvF ower lned timeEvrying lumped prmeter model of vol trtX modelling nd simultionF PHIWF fHQ hou ngD iero imonelliD nd eln g fovikF wultisle struturl similrity for imge qulity ssessmentF sn The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003D volume PD pges IQWV!IRHPF siiiD PHHQF fIU fenjmin eitzD sngmr teinerD nd eter firkholzF qestureEsed rtiultory text to speeh synthesisF Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2017D pges QPR!QQID PHIUF uIT hizheng uD yliver ttsD nd imon uingF werlinX en open soure neurl network speeh synthesis systemF Proc. SSW, Sunnyvale, USAD PHITF iq + HP teve oungD qunnr ivermnnD wrk qlesD homs rinD hn uershwD unE ying viuD qreth wooreD tulin ydellD hve yllsonD hn oveyD et lF he ru ookF Cambridge university engineering departmentD QXIUSD PHHPF xIW suks oshingD uzunori xozkiD nd higeo dF e simpli(ed vol trt model for rtiultion of sX he e'et of tongue tip elevtion on sF PLOS ONED IR@IHAXeHPPQQVPD PHIWF emIH illrd emlinF Speech and Hearing Science, Anatomy and Physiology. erson idution @AD pourth idition editionD PHIHF enHT reig enF en exmple of ontextEdependent lel formt for rwwEsed speeh synthesis in inglishF The HTS CMUARCTIC demoD IQQD PHHTF PHP Résumé

	Bibliography
	fqWP fxHV phHH qrSH	gF F frowmn nd vF qoldsteinF ertiultory phonologyX n overviewF ttus irik fresh nd hriknth xrynnF egion segmenttion in the frequeny doE Annual Conference of the International Speech Communication Association-hn poygel nd qry hellF wodels of impired lexil ess in speeh produE wurie qrmmontF Traité de phonétiqueF viririe helgrveD IWSHF
		report on speeh reserhD rskins vortoryD IWWPF min pplied to upper irwy relEtime mgneti resonne imgesF IEEE trans-2019D PHIWF tionF Journal of Memory and LanguageD RQ@PAXIVP!PITD PHHHF
	fqWP qHT tenEwihel qérrdD einer ilhelmsEririoD sl errierD nd ohn ynF gtherine frowmn nd vouis qoldsteinF ertiultory phonologyX en overviewF actions on medical imagingD PV@QAXQPQ!QQVD PHHVF ifII edulkdir iryildirim nd wrieEydile fergerF e guided pproh for utomti phHR géile pougeron nd iliseth helisEoussrieF viisons et enhînementsX e Qh dynmil iomehnil tongue model to study speeh motor ontrolF arXiv
	PhoneticaD RW@QERAXISS!IVHD IWWPF frHU to fenestyD w wohn ondhiD nd iteng rungF Springer handbook of segmenttion nd modeling of the vol trt in mri imgesF sn 2011 19th Euro-fisen á fezen prlnt F Actes des Journées d'Etudes sur la ParoleD pges preprint physics/0606148D PHHTF
	Bibliography speech processingF pringerD PHHUF pean Signal Processing ConferenceD pges TI!TSF siiiD PHIIF PPI!PPRD PHHRF fqrHI ellen frunD endre quilleminD vr roseyD nd wry rgF he neurl orgE niztion of disourseX en hPISoEpet study of nrrtive prodution in english nd merin sign lngugeF BrainD IPR@IHAXPHPV!PHRRD PHHIF fIV ul foersm nd hvid eeninkF extqrid (le formtsF http://www.fon.hum. uva.nl/praat/manual/TextGrid_file_formats.htmlD PHIVF ynlineY essed ivIT fenjmin ilie nd ves vprieF ixtension of the singleEmtrix formultion of the vol trtX gonsidertion of ilterl hnnels nd onnetion of selfEosillting putHT rrUT illim t rrdstleF Physiology of speech production: an introduction for suzsnn pgylD hougls uieeD nd prederi tenkinsF French: A linguistic introductionF gmridge niversity ressD PHHTF speech scientistsF edemi ressD IWUTF
	egpIU frHU hvid t frenner nd iri t rllF gomputed tomogrphy"n inresing soure prneso evnziniD iero gosiD olndo püstösD nd endre ndiF hen fntsy meets sieneX en ttempt to rerete the voie of ötzi the iemnF StudiAISVD of rdition exposureF New England Journal of MedicineD QSU@PPAXPPUU!PPVRD PHHUF QHEtuneEPHIWF fHV models of the vol folds with glottl hinkF Speech CommunicationD VPXVS!WTD PHITF pvTV rrWW t rrdstleF iletromyogrphyF Coarticulation: theory, data and techniques. tmes v plngn nd vorind v vndgrfF elfEosillting soure for volEtrt synthesizersF Audio and Electroacoustics, IEEE Transactions onD IT@IAXSU!TRD University Press, CambridgeD pges PUH!PVQD IWWWF eln g frooksD ionn hoD nd hrsyvoulos x ppsF truturl similrity qulity metris in oding ontextX ixploring the spe of relisti distortionsF ivIT fenjmin ilie nd ves vprieF e glottl hink model for the synthesis of voied IWTVF rfWT endrew t runt nd eln flkF nit seletion in ontentive speeh synE PHIUF eiIU sn esdidi nd ingin irzinF ol trt irwy tissue oundry trking for rtws using shpe nd pperne priorsF sn InterspeechD pges TQT!TRHD PHIUF frq + IT plorent foqueletD homs rueerD vurent qirinD ghristophe vriuxD nd flise vertF elEtime ontrol of n rtiultoryEsed speeh synthesizer for rin omputer interfesF PLoS computational biologyD IP@IIAXeIHHSIIWD PHITF IEEE Transactions on image processingD IU@VAXIPTI!IPUQD PHHVF glVW fritivesF sn Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE plIQ thesis system using lrge speeh dtseF sn Acoustics, Speech, and Signal tmes v plngnF Speech analysis, synthesis and perceptionD volume QF International Conference onD pges SPRH!SPRRF siiiD PHITF pringer iene 8 fusiness wediD PHIQF Processing, 1996. ICASSP-96. Conference Proceedings., 1996 IEEE Interna-glliopeF hesription oustiqueF sn La parole et son traitement automatiqueD hpter QF wssonD risD IWVWF ivyIT tional Conference onD volume ID pges QUQ!QUTF siiiD IWWTF fenjmin ilieD ves vprieD ierreEendré uissozD nd preddy ydilleF righ spE pwtIS hominique pohrD ydile wellD nd henis touvetF he l9importne de erw + IS eter endersonD xegr w rrndiD ott woisikD sn tvnessD nd idney pelsF e omprehensive Qh iomehnillyEdriven vol trt model inluding inverse frIR r wellister fyunD iline rithokD nd wihelle wrtzF etro)ex versus unhed in tretment for rhoti misrtiultionX ividene from ultrsound glVW tiotemporl inews (lms using ompressed sensing for quiring rtiultory l9homogénéistion des onventions de trnsription pour l9lignement utomtique rfIW sn rowrd nd eter firkholzF wodelling vowel quisition using the irkholz glliopeF La parole et son traitement automatiqueF wssonD risD IWVWF dtF sn Eusipco, BudapestD pges IQSQ!IQSUD eugust PHITF de orpus orux de prole spontnéeF sn 8es Journées Internationales de Lin-synthesizerF Studientexte zur Sprachkommunikation: Elektronische Sprachsig-dynmis for speeh reserhF sn Sixteenth Annual Conference of the Interna-tional Speech Communication AssociationD PHISF endVP tephen endersonF he nlysis of prenh shwX orD how to get something for nothingF LanguageD pges SQR!SUQD IWVPF ef + HW wihel eronD esterios outiosD wrieEydile fergerD irwn uerrienD frigitte roelEhutourtD nd ves vprieF egistrtion of multimodl dt for estimtE ing the prmeters of n rtiultory modelF sn IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)D iwn ipeiD PHHWF iofeedk interventionF Journal of Speech, Language, and Hearing ResearchD grTV xom ghomsky nd worris rlleF The sound pattern of English. isgD IWTVF ingHH guistique de Corpus (JLC2015)D PHISF nalverarbeitung 2019D pges QHR!QIID PHIWF ylov ingwllF ere stti ws mesurements representtive of dynmi speehc SU@TAXPIIT!PIQHD PHIRF firHU eter firkholzF gontrol of n rtiultory speeh synthesizer sed on dynmi pproximtion of sptil rtiultory trgetsF sn INTERSPEECHD pges PVTS! PVTVD PHHUF firIQ eter firkholzF wodeling onsonntEvowel ortiultion for rtiultory speeh synthesisF PloS oneD V@RAXeTHTHQD PHIQF hgqyIW r hhmniD inent golotteD lérin qirrdD nd lim yuniF gonditionl vritionl utoEenoder for textEdriven expressive udiovisul speeh synthesisF results from omprtive study using wsD iq nd iweF sn Sixth Interna-pouSW ierre pouhéF Traité de prononciation françaiseF ulinksiekD risD IWSWF riIP qregory rikokF gomputtionl neurontomy of speeh produtionF Nature tional Conference on Spoken Language ProcessingD PHHHF pouHI reviews neuroscienceD IQ@PAXIQSD PHIPF géile pougeronF ertiultory properties of initil segments in severl prosodi Proc. Interspeech 2019D pges PSWV!PTHPD PHIWF hpp + IW sonnis hourosD tques pellingerD tens prhmD uryn ssievD erun eF tosephD ves vprieD preddy ydilleD enstsii suknovD hirk oitD nd ierreEendré uissozF e multimodl relEtime ws rtiultory orpus of prenh for speeh i + II fyron h irthD en h etersonD wtís ñrtuD qeorge odikD nd onstituents in prenhF Journal of phoneticsD PW@PAXIHW!IQSD PHHIF riIR qregory rikokF owrds n integrted psyholinguistiD neurolinguistiD sensoriE wihel lesnikF e theoretil model of the pressure (eld rising from symE pWU géile pougeron nd hon terideF hoes deletion of prenh shw led to neuE motor frmework for speeh produtionF Language, Cognition and NeuroscienceD metri intrglottl )ows pplied to twoEmss model of the vol foldsF The Journal of the Acoustical Society of AmericaD IQH@IAXQVW!RHQD PHIIF trliztion of lexil distintionsc sn EUROSPEECHD IWWUF PW@IAXSP!SWD PHIRF
	vX http://hal.inria.fr/inria-00350298/en/F turl similrityF Optical reviewD IT@TAXTIQ!TPID PHHWF tolnt fhnF i0ient diphone dtse retion for mrolD multilingul speeh synthesiserF sn XII International PhD Workshop (OWD 2010). Con-de/index.php?page=vocaltractlab-aboutD PHIQF ynlineY essed IHEytoerE PHIWF F firkholz nd hF tkelF e threeEdimensionl model of the vol trt for speeh leorgne nd lelongF BrainD IQH@SAXIRQP!IRRID PHHUF peil iurorometerF PHIPF iuropens nd their lngugesF European Com-Engineering in Medicine and Biology SocietyD pges SQTH!SQTQF siiiD PHIIF nisF ul ro9s histori sesX high resolution mr imging of the rins of PHIWF time speeh synthesisF sn 2011 Annual International Conference of the IEEE hsgHU xin p hronkersD ydile lisntD wrie herese sEizenD nd immnuel e gE evHW fIH ftHQ iurIU qfII prnk r quenther nd tonthn frumergF frinEmhine interfes for relE ynline versionX http://www.ethnologue.comD PHIWF ynlineY essed PPEtuneE elirez xsiri evnkiF ixt glol histogrm spei(tion optimized for struE firIQ eter firkholzF olrtv PFHX e tool for rtiultory speeh synthesisF ehnishe niversität hresdenF ynline versionX http://www.vocaltractlab. reserhF sn InterSpeech-20th Annual Conference of the International Speech Communication Association-2019D PHIWF ieIW vnguges of the worldF wentyEseond editionF hllsD exsX sv snterntionlF informtion in prenh vowelsF Language and SpeechD QI@IAXSU!USD IWVVF hvid wF ierhrdD qry pF imonsD nd ghrles hF pennig @edsFAF ithnologueX qfVV erry v qottfried nd trie peeter feddorF ereption of temporl nd spetrl rwHS w rowe nd
	ference Archives PTETiSD volume PVD pges QHQ!QHVD PHIHF ierre fdinD qerrd fillyD vionel everetD woni fiuD ghristoph egerthD nd ghristophe vriuxF hreeEdimensionl liner rtiultory modeling of synthesisF sn 15th International Congress of Phonetic Sciences -ICPhS'2003, ff + HP Barcelona, SpainD pges PSWU!PTHHD eug PHHQF ftuHT F firkholzD hF tkèlD nd fF tF urögerF gonstrution nd ontrol of threeE dimensionl vol trt modelF sn Proc. Intl. Conf. Acoust., Spch., and Sig. h + WT missionD PHIUF qf + HW prnk r quentherD tonthn frumergD i toseph rightD elfonso xietoE hierry hutoitD inent gelD xiols ierretD prnçois ftilleD nd ylivier n der rekenF he mrol projetX owrds set of high qulity speeh synE thesizers free of use for non ommeril purposesF sn Proceeding of Fourth In-ternational Conference on Spoken Language Processing. ICSLP'96D volume QD peHR gstnonD tson e ourvilleD wikhil nkoD oert vwD teven e ieertD tess v engel h priederii nd ui elterF vterliztion of uditory lnguge funtionsX frtelsD hinl endresenD et lF e wireless rinEmhine interfe for relEtime dynmi dul pthwy modelF Brain and languageD VW@PAXPTU!PUTD PHHRF speeh synthesisF PloS oneD R@IPAXeVPIVD PHHWF tongueD lips nd feD sed on ws nd video imgesF Journal of PhoneticsD Proc. (ICASSP 2006)D pges VUQ!VUTD PHHTF pges IQWQ!IQWTF siiiD IWWTF pnTH qF pntF Acoustic Theory of Speech ProductionF he rgueX wouton 8 goFD qqqqII wurie qrevisseD endré qoosseD wurie qrevisseD nd wurie qrevisseF Le bon QH@QAXSQQ!SSQD PHHPF fenVH ton vouis fentleyF wultidimensionl divideEndEonquerF Communications of ftuHT eter firkholzD hietmr tkèlD nd fernd t urögerF gonstrution nd ontrol of hIW wy hvis nd weliss e edfordF he emergene of disrete pereptulEmotor IWTHF usage: grammaire langue françaiseF he foekFD PHIIF
	the ACMD PQ@RAXPIR!PPWD IWVHF threeEdimensionl vol trt modelF sn Acoustics, Speech and Signal Process-units in prodution model tht ssumes holisti phonologil representtionsF qunnr pntF Acoustic theory of speech production: with calculations based qvwIP pnUI ndll qessD ghntl vyheD nd rudel weisenurgF Phonological variation
	fp + hs + IW ing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference FRONTIERS IN PSYCHOLOGYD IHD PHIWF on X-ray studies of Russian articulationsD volume PF lter de qruyterD IWUIF in French: Illustrations from three continentsD volume IIF tohn fenjmins ghrlotte fellinghusenD homs pngmeierD fernhrd hröderD tohnn uellerD usnne hrehselD eter firkholzD vudger ertz vn ilstD nd endres iedelF onD volume ID pges s!sF siiiD PHHTF sonnis hourosD enstsii suknovD uryn ssievD ierreEendré uissozD nd pnUI qunnr pntF he pEptterns of ompound tue resontors nd hornsF Acous-ulishingD PHIPF
	fvhy qxWW	yn the role of dis)uent speeh for unertinty in rtiultory speeh synthesisF sn héo fisuttoEvervtD r hhmniD nd lim yuniF wodeling lil ortiulE ves vprieF owrds method of dynmi vol trt shpes genertion y tic theory of speech production: with calculations based on X-ray studies of pion qion nd uterin xiolidisF ltogrphyF Coarticulation: Theory,
		The 9th Workshop on Disuency in Spontaneous SpeechD pge QWF tion with idiretionl gted reurrent networks nd trnsfer lerningF omining stti Qh nd dynmi Ph ws speeh dtF sn InterSpeech-20th Russian articulationsD IWUIF data and techniquesD pges PPW!PRRD IWWWF
	IVT IVV IWH	IVU	IVS IVW IWI

The gure can get higher if we factor in smaller cavities and areas that are not clearly separated from some of the cavities' proper, such as two small cavities in form of a deep depression, lateral to the aditus laryngis, that are situated at the bottom of the pharynx and called pyriform sinuses, or the sublingual cavity under the tongue. However, they do not seem to play a crucial role in speech production.IU

Acknowledgments

Acknowledgments R