Single cell genetic manipulation is expected to significantly advance the field of systems neuroscience. However, existing gene delivery techniques do not permit electrophysiological characterization of cells that would establish an experimental link between physiology and genetics for understanding neuronal function. Here we demonstrate in the mouse brain in vivo that (i) neurons remain intact after 'blind' whole-cell recording, (ii) that DNA vectors can be delivered through the patch-pipette during such recordings and (iii) that these vectors drive protein expression in recorded cells for at least seven days. We illustrate the utility of this approach by recording visually-evoked synaptic responses of primary visual cortical cells while delivering DNA plasmids that permit retrograde, mono-synaptic tracing of that neuron's presynaptic inputs. By providing a biophysical profile of the cell prior to its specific genetic perturbation, this combinatorial method has captured the first synaptic and anatomical receptive field of a neuron.

A central goal for neuroscience is to understand how neurons in the brain and spinal cord process information. Ultimately this requires obtaining information from combined electrophysiological, neuroanatomical, optical and genetic methods that together can address the necessary broad range of questions from the level of genes and molecules, up to single cells and circuits [START_REF] Kleinfeld | From art to engineering? The rise of in vivo mammalian electrophysiology via genetically targeted labeling and nonlinear imaging[END_REF][START_REF] Helmchen | Deep tissue two-photon microscopy[END_REF][START_REF] Luo | Genetic dissection of neural circuits[END_REF][START_REF] Arenkiel | Molecular genetics and imaging technologies for circuit-based neuroanatomy[END_REF][START_REF] Geschwind | Neuroscience in the era of functional genomics and systems biology[END_REF][START_REF] O'connor | Reverse engineering the mouse brain[END_REF][START_REF] Scanziani | Electrophysiology in the age of light[END_REF][START_REF] Gradinaru | Molecular and cellular approaches for diversifying and extending optogenetics[END_REF][START_REF] Lichtman | A technicolour approach to the connectome[END_REF] . Electrical recording methods, such as the whole-cell patch-clamp technique, can provide a relatively complete biophysical characterization of an individual neuron. Its recent optimization for in vivo applications has afforded increased stability and longevity such that the intrinsic, synaptic and spiking properties of cells in many superficial [START_REF] Margrie | Action potential propagation in mitral cell lateral dendrites is decremental and controls recurrent and lateral inhibition in the mammalian olfactory bulb[END_REF][START_REF] Wehr | Balanced inhibition underlies tuning and sharpens spike timing in auditory cortex[END_REF][START_REF] Waters | Supralinear Ca2+ influx into dendritic tufts of layer 2/3 neocortical pyramidal neurons in vitro and in vivo[END_REF][START_REF] Loewenstein | Bistability of cerebellar Purkinje cells modulated by sensory stimulation[END_REF] , and deep structures [START_REF] Margrie | In vivo whole-cell recordings from neurons in the anaesthetized and awake mammalian brain[END_REF][START_REF] Brecht | Whisker movements evoked by stimulation of single pyramidal cells in rat motor cortex[END_REF][START_REF] Arenz | The contribution of single synapses to sensory representation in vivo[END_REF][START_REF] Harvey | Intracellular dynamics of hippocampal place cells during virtual navigation[END_REF][START_REF] Epsztein | Impact of spikelets on hippocampal CA1 pyramidal cell activity during spatial exploration[END_REF] have been obtained. This method has been especially useful for determining the synaptic receptive fields of single cells in anesthetized [START_REF] Margrie | In vivo whole-cell recordings from neurons in the anaesthetized and awake mammalian brain[END_REF] , awake head-fixed [START_REF] Margrie | In vivo whole-cell recordings from neurons in the anaesthetized and awake mammalian brain[END_REF][START_REF] Harvey | Intracellular dynamics of hippocampal place cells during virtual navigation[END_REF][START_REF] Hromadka | Sparse representation of sounds in the unanesthetized auditory cortex[END_REF] and freely moving [START_REF] Lee | Head-anchored whole-cell recordings in freely moving rats[END_REF] preparations so that synaptic activity may be linked directly to sensory processing [START_REF] Arenz | The contribution of single synapses to sensory representation in vivo[END_REF] and behavior [START_REF] Harvey | Intracellular dynamics of hippocampal place cells during virtual navigation[END_REF][START_REF] Epsztein | Impact of spikelets on hippocampal CA1 pyramidal cell activity during spatial exploration[END_REF] .

Concurrently, recent developments in optogenetics [START_REF] Gradinaru | Molecular and cellular approaches for diversifying and extending optogenetics[END_REF][START_REF] Zhang | Circuit-breakers: optical technologies for probing neural signals and systems[END_REF][START_REF] Zhang | Channelrhodopsin-2 and optical control of excitable cells[END_REF] and virus-based circuit tracing [START_REF] Luo | Genetic dissection of neural circuits[END_REF][START_REF] Arenkiel | Molecular genetics and imaging technologies for circuit-based neuroanatomy[END_REF][START_REF] Wickersham | Monosynaptic restriction of transsynaptic tracing from single, genetically targeted neurons[END_REF] are providing effective tools for manipulating and tracing neuronal circuits both in vitro and in vivo. For example, genetically controlling expression of channel-or halorhodopsin make it possible to selectively excite or inhibit the activity of defined sets of neurons in vivo [START_REF] Gradinaru | Molecular and cellular approaches for diversifying and extending optogenetics[END_REF][START_REF] Huber | Sparse optical microstimulation in barrel cortex drives learned behaviour in freely moving mice[END_REF] . Viral strategies for expression are also becoming increasingly useful for spatially, temporally and genetically controlling network function [START_REF] Luo | Genetic dissection of neural circuits[END_REF][START_REF] Komai | Postsynaptic excitability is necessary for strengthening of cortical sensory responses during experience-dependent development[END_REF] and even tracing the anatomical connectivity of a single cell [START_REF] Marshel | Targeting single neuronal networks for gene expression and cell labeling in vivo[END_REF] .

At present however, there is no method available that delivers both single cell electrophysiology and genetic manipulation. Application of such a tool could be used to provide answers to questions that are fundamental to our understanding of the role of single neurons within the context of a local network in the intact animal [START_REF] Douglas | Mapping the matrix: the ways of neocortex[END_REF] .

The relatively large tip size of patch pipettes not only permits low-resistance intracellular recordings but readily allows dialysis of exogenous materials into the recorded cell [START_REF] Waters | Supralinear Ca2+ influx into dendritic tufts of layer 2/3 neocortical pyramidal neurons in vitro and in vivo[END_REF][START_REF] Margrie | In vivo whole-cell recordings from neurons in the anaesthetized and awake mammalian brain[END_REF][START_REF] Eilers | A Practical Guide: Dye Loading with Patch Pipettes[END_REF] , which raises the possibility of delivering macromolecules such as plasmids. Here we examine the feasibility of combining in vivo whole-cell recording and gene delivery by answering four specific questions: Do cells survive for significant periods following whole-cell recording in vivo? What is the likelihood and longevity of whole-cell recordings with internal solutions containing plasmid DNA? Can plasmid DNA dialyze into the cell and drive protein expression? Is protein expression restricted to only the patched cell?

After demonstrating the feasibility of this approach we then highlight its potential by recording the synaptic responses of neurons in mouse primary visual cortex to presentation of drifting gratings, while delivering vectors to permit monosynaptic retrograde labeling using a modified rabies virus [START_REF] Wickersham | Monosynaptic restriction of transsynaptic tracing from single, genetically targeted neurons[END_REF] . Several days after virus injection we observed unprecedented numbers of labeled neurons permitting registration of a cell's synaptic receptive field and the presynaptic cells that define it.

RESULTS (i) Recovery rates of in vivo whole-cell recorded neurons

It is generally assumed that whole-cell recording substantially compromises cell viability. Therefore, when morphological confirmation is required following in vivo whole-cell recording, animals are usually immediately sacrificed and the tissue fixed for histological analysis [START_REF] Brecht | Organization of rat vibrissa motor cortex and adjacent areas according to cytoarchitectonics, microstimulation, and intracellular stimulation of identified cells[END_REF] . We first directly tested this assumption by performing in vivo whole-cell recordings from cells in layers 2-5 of mouse neocortex using a biocytin-containing intracellular solution and assessing the incidence and morphology of labeled neurons several days after recording. The method for obtaining whole-cell recordings was essentially as described previously [START_REF] Margrie | In vivo whole-cell recordings from neurons in the anaesthetized and awake mammalian brain[END_REF] . Briefly, a glass electrode was filled with a potassium-based intracellular solution and advanced quickly to the target depth under moderate positive pressure (Fig. 1a). The positive pressure was then reduced and the electrode was advanced while the tip-resistance was monitored. The pressure was removed upon 'hitting' a cell, judged by changes in the tip-resistance (Fig. 1b), and negative pressure was applied to help seal formation (Fig. 1c). Whole-cell access was achieved by brief bursts of negative pressure to rupture the cell membrane (Fig. 1d). In the whole-cell configuration and in current clamp mode, the current-voltage relationship (Fig. 1e) and spontaneous and evoked activity was recorded. At the end of the recording the patch pipette was slowly withdrawn while constantly monitoring the access resistance in voltage-clamp mode (Fig. 1f,g). Pipette retraction usually resulted in membrane resealing (Fig. 1g) and formation of an outside-out patch (Fig. 1h) that was considered an important, though not essential, predictor of cell viability. Following whole-cell recording, mice were recovered and returned to their home cages (see methods).

To assess the survival rate of recorded neurons, mice were sacrificed 24-51 hours after recording, the brain removed, fixed and slices were prepared (see methods).

The vast majority of cells recorded were pyramidal cells and their recovered morphologies typically included the apical dendrite that extended up to layer 1 (Fig. 1i). The morphological criterion for identifying determining cells was that the cell soma and the proximal regions of at least four dendrites were visible. The number of recovered cells per brain that met this criterion never exceeded the number of cells recorded.

Our criteria for considering recorded cells as being potentially recoverable included that the initial series resistance was below 80 MΩ (Fig. 1j). In most cases (>90%), these recordings were actively terminated as described above (Fig. 1). In 10 cases where retraction criterion was not met or series resistance was above 80 MΩ, another recording in the same animal was successfully obtained at a different cortical depth. In these cases we recovered 3/10 cells that did no satisfy both criteria (Fig. 1j, grey circles). Cells that do not meet our criteria have not been included in the recording duration and recovery interval analysis. Parameters such as recording duration (up to 13 minutes) and recovery interval (up to 51 hours) did not correlate with the likelihood of observing biocytin filled cells (Fig. 1k,l). According to these criteria, we recovered 78% (47/60) of recorded cells following biocytin visualization with Atto-565 or Alexa-488 conjugated streptavidin (Fig. 1i). This indicates that neurons can reliably survive whole-cell recording and maintain their structural integrity.

(ii) Whole-cell recordings with internal solutions containing plasmid DNA

To determine the feasibility of DNA loading during whole-cell recording we next performed recordings with intracellular solutions containing up to 350 ng/µl of plasmid DNA encoding fluorescent proteins (see methods) and biocytin.

Qualitatively, we found that the likelihood of obtaining seals and low-resistance recordings with internal solutions containing DNA was similar to that using standard internals.

Cells recorded with plasmid DNA in the pipette had firing profiles typical of cortical neuronal subclasses [START_REF] Brecht | Whisker movements evoked by stimulation of single pyramidal cells in rat motor cortex[END_REF][START_REF] Brecht | Organization of rat vibrissa motor cortex and adjacent areas according to cytoarchitectonics, microstimulation, and intracellular stimulation of identified cells[END_REF][START_REF] Margrie | Targeted whole-cell recordings in the mammalian brain in vivo[END_REF] (Fig. 2a). Based on stable resting membrane potentials, ongoing synaptic activity (Fig. 2b) and overshooting action potentials (Fig. 2b), cells remained healthy for the duration of the recording (typically 5-13 minutes). All recordings were actively terminated after the intrinsic biophysical properties (Fig. 2a) and synaptic receptive fields of cells were obtained (Fig. 2c).

After slowly retracting the pipette, mice were allowed to recover for at least 24 hours (range 24-51 hours) before sacrificing. Usually only one cell was recorded per animal, though in some mice several cells (typically two) were recorded in different layers, allowing electrophysiology and GFP fluorescence to be unambiguously matched. All recordings for these experiments were actively and successfully terminated.

(iii) Protein expression

We found that we often recovered intensely fluorescent cells, which were always in the expected locations and had elaborate dendritic morphologies consistent with predictions from electrophysiological data (Fig. 2a,b). Of the 39 cells recorded with internal solution containing DNA, 22 were found to express GFP. In all cases where GFP labeled cells were observed (within 51 hours after recording), cells were also positive for biocytin (Fig. 2d1,d2). Nine cells were however found to be biocytin positive and did not express GFP. Following immuno-histochemical amplification, GFP-positive cells always showed label in dendritic spines and the axon (Fig. 2e).

Unlike the biocytin label, we only observed expression of the fluorescent protein in single cells and never along the electrode tract (cf; Fig 2d1 ,d2). As with recordings with biocytin, we observed a similar effect of series resistance on recovery probability (Fig. 2f) while recording duration and recovery interval appeared to have no such impact (Fig. 2g,h). Thus, following whole-cell recording with plasmidcontaining intracellular solution, 56% of cells expressed fluorescent protein specifically in, and only in the recorded cell.

There exist obvious advantages in delivering multiple genes into a specific neuron that has been electrophysiologically characterized. We therefore next enquired whether multiple vectors could be reliably co-expressed by including two or three types of plasmids encoding proteins fluorescent at different wavelengths (e.g. td-Tomato, Cerulean, Venus) in our intracellular solution. For experiments using multiple plasmids, 3/7 cells were recovered, and fluorescence was observed at all appropriate wavelengths in all recovered cells (Fig. 3a,b).

Together, these results show that neurons can survive in vivo whole-cell recordings.

DNA plasmids in the intracellular solution do not impair electrophysiological characterization of cells but readily dialyze into the neuron resulting in expression of exogenous genes. Since plasmid DNA expelled into the extracellular space is not expressed, transfection is restricted to the recorded cell. This method therefore appears ideally suited to span neurophysiological and genetic approaches for understanding the function of specific cells and circuits in vivo.

Application: bridging single cell function and connectivity

A method describing monosynaptic control of modified rabies virus spread into presynaptic neurons has recently been reported [START_REF] Wickersham | Monosynaptic restriction of transsynaptic tracing from single, genetically targeted neurons[END_REF] . It is widely accepted that the native, non-modified rabies virus infects synaptically connected neurons exclusively in the retrograde direction [START_REF] Ugolini | Specificity of rabies virus as a transneuronal tracer of motor networks: transfer from hypoglossal motoneurons to connected secondorder and higher order central nervous system cell groups[END_REF][START_REF] Ugolini | Advances in viral transneuronal tracing[END_REF] , a process that relies on the rabies virus glycoprotein (RV-G) [START_REF] Mebatsion | Budding of rabies virus particles in the absence of the spike glycoprotein[END_REF][START_REF] Mebatsion | Extensive attenuation of rabies virus by simultaneously modifying the dynein light chain binding site in the P protein and replacing Arg333 in the G protein[END_REF] . When the RV-G gene is deleted and replaced by a fluorescent proteinencoding gene, this modified rabies virus is rendered incompetent for trans-synaptic infection, but can replicate inside the initially infected cell. Infection by RV-Gdeficient rabies virus encapsulated with the avian sarcoma and leucosis virus envelope protein (SAD-∆G-mCherry-EnvA) is restricted to only those cells that express an avian receptor protein (TVA) that is not expressed in mammals. Thus, by transfecting a single cell with vectors expressing TVA and RV-G, only this cell will be (i) susceptible to initial infection (via TVA) and (ii) provides the incompetent virus with RV-G required for trans-synaptic infection. The virus can therefore spread from the primarily infected cell to its presynaptic inputs and express fluorescent protein in these cells. However, since RV-G is not expressed in the presynaptic cells, the virus can spread no further. This approach therefore labels only first-order neurons presynaptically connected to the primary cell [START_REF] Wickersham | Monosynaptic restriction of transsynaptic tracing from single, genetically targeted neurons[END_REF][START_REF] Marshel | Targeting single neuronal networks for gene expression and cell labeling in vivo[END_REF][START_REF] Ugolini | Advances in viral transneuronal tracing[END_REF] .

To demonstrate the potential of genetic manipulation via whole-cell recording in vivo to bridge our understanding of cell function and connectivity we first recorded the intrinsic properties (Fig. 4a) and visually-evoked synaptic responses (Fig. 4b) of a layer 5 pyramidal cells in mouse visual cortex (n = 8) to drifting gratings of different orientations (see methods). Synaptic responses were more broadly tuned than spiking output but showed clearly similar orientation preferences [START_REF] Ferster | Origin of orientation-selective EPSPs in simple cells of cat visual cortex[END_REF][START_REF] Marino | Invariant computations in local cortical networks with balanced excitation and inhibition[END_REF] (Fig. 4b,c).

Furthermore, and in an orientation-specific manner, the evoked synaptic activity fluctuated substantially during such responses (Fig. 4c). During functional characterization we simultaneously delivered a vector that directed expression of both the TVA receptor and RV-G, and a second vector directing expression of GFP (Fig. 4d,e). Two days after recording we injected ~200nl of SAD-∆G-mCherry-EnvA into layer 5.

Between four and thirteen days after injection animals were transcardially perfused, sacrificed (n = 8) and 100 or 200 µm thick brain slices were prepared (see methods).

Fluorescent signals were then imaged using confocal microscopy [START_REF] Conchello | Optical sectioning microscopy[END_REF] . In 3 of the 8 animals we observed hundreds of mCherry-expressing neurons (n = 152, 221, 666; Fig. 4d-g; Supplementary Fig. 1) within and outside primary visual cortex. In 2 of the remaining 5 mice we observed a single GFP-expressing cell but no mCherry signal, most likely indicating successful transfection in cells that were not subsequently infected with the pseudotyped rabies virus. In the remaining 3 mice we failed to detect cells expressing either GFP or mCherry. In control animals (n = 2) that only received virus injection (500 nl) we observed a total of 6 mCherry-positive glia and 1 neuron (Supplementary Fig. 1). These data show that by combining in vivo whole-cell transfection with trans-synaptic tracing methods it is possible to record the synaptic receptive fields of neurons while delivering the genetic machinery necessary to successfully identify the presynaptic cells that transmit such signals.

DISCUSSION

Here we have shown that the whole-cell patch clamp method can be used to transfect single cells during neuronal recording in vivo. Perhaps the most surprising aspect of the data presented here is the high survival rate after in vivo whole-cell recording. This finding is in line with a previous study where, in acute cortical slices neurons that are patched, and then re-patched up to 12 hours later, remain healthy though their connectivity may be altered [START_REF] Le Be | Spontaneous and evoked synaptic rewiring in the neonatal neocortex[END_REF] . Here, we have investigated cell mortality rates in vivo up to 51 hours post recording using morphological criteria. Consistent with data obtained from slices, the large number of healthy neurons observed after in vivo recording ensures that the likelihood of genetic manipulation is relatively high (~50 %). Following DNA delivery via whole-cell recording in vivo, cells remain viable since they continue to express reporter proteins for up to a week. During this time recorded cells are also able to withstand viral infection and its subsequent demands placed on their metabolic processes. This is the first demonstration that DNA may be delivered into neurons via an intracellular recording method. This approach should therefore be considered useful for any experimental system including neuronal and non-neuronal cultures where cells can be maintained for extended periods of time. In combination with electrically recording from the cell, the list of potential applications of this method ranges from genetic perturbation of intracellular signaling cascades, up-or downregulation of ion channels, membrane receptors, light-activatable ion channels or genetically encoded indicators, expressed across functionally-related presynaptic circuits.

Series resistance appears to be the most important factor influencing the recovery rate of both biocytin-filled and DNA loaded cells. Again, this is in line with previous reports indicating that the efficiency of loading substances such as dyes and biocytin is reduced as series resistance increases [START_REF] Eilers | A Practical Guide: Dye Loading with Patch Pipettes[END_REF] . As with dye loading, the relatively low access resistance of intracellular recordings obtained with the patch pipettes used in vivo are suitable for DNA delivery. In cases where we recovered cells recorded with intracellular solutions containing multiple reporter plasmids, we always observed all of the expected fluorescent proteins (up to three). Though we have not established an upper limit, this suggests that our transfection approach results in the successful delivery of many plasmids. Given that plasmids may contain more that one proteinencoding sequence, a substantial number of genes may be simultaneously delivered via this method.

Here we used modified rabies virus-mediated monosynaptic retrograde tracing to demonstrate one application of combined whole-cell recording and multi-gene delivery. During recording of intrinsic properties and visually-evoked synaptic responses we delivered genes to direct infection and spread of SAD-∆G-mCherry-EnvA in the recorded cell. The resulting number of mCherry-labeled cells presynaptic to layer 5 neurons shown here substantially exceeds that for Layer 2/3 cells observed in a recent electroporation study [START_REF] Marshel | Targeting single neuronal networks for gene expression and cell labeling in vivo[END_REF] . This disparity may reflect differences between the connectivity of layer 2/3 and layer 5 pyramidal neurons. However, the difference in the extent of presynaptic labeling may also reflect differences in the efficiency of plasmid delivery and protein expression between these two methods. Genetic manipulations via electroporation has been complemented with two-photon imaging to allow targeted gene delivery to single neurons in the mouse brain, 26 with similar success rates to that reported here. Though such a targeted electroporation method is promising, its application is somewhat limited. Specifically, although action potentials in the target cell could, in principle, be detected [START_REF] De Kock | Spiking in primary somatosensory cortex during natural whisking in awake head-restrained rats is cell-type specific[END_REF] prior to electroporation, cell attached recordings are highly restricted since they preclude measurements of synaptic and intrinsic properties. Secondly, successful single-cell in vivo electroporation has required visualization [START_REF] Marshel | Targeting single neuronal networks for gene expression and cell labeling in vivo[END_REF][START_REF] Nevian | Calcium indicator loading of neurons using single-cell electroporation[END_REF][START_REF] Kitamura | Targeted patch-clamp recordings and single-cell electroporation of unlabeled neurons in vivo[END_REF] , limiting this method to superficial cells in the intact brain. Although the whole-cell transfection method described here can certainly be combined with optical monitoring to target specific cell types [START_REF] Margrie | Targeted whole-cell recordings in the mammalian brain in vivo[END_REF][START_REF] Kitamura | Targeted patch-clamp recordings and single-cell electroporation of unlabeled neurons in vivo[END_REF] , we have used blind whole-cell recordings for single-cell transfection in optically inaccessible, deeper structures [START_REF] Margrie | In vivo whole-cell recordings from neurons in the anaesthetized and awake mammalian brain[END_REF][START_REF] Brecht | Whisker movements evoked by stimulation of single pyramidal cells in rat motor cortex[END_REF][START_REF] Arenz | The contribution of single synapses to sensory representation in vivo[END_REF][START_REF] Harvey | Intracellular dynamics of hippocampal place cells during virtual navigation[END_REF][START_REF] Epsztein | Impact of spikelets on hippocampal CA1 pyramidal cell activity during spatial exploration[END_REF] . It therefore appears that any cell type that can be patched can be genetically modified in the process.

Wild-type rabies virus has long been used as an extremely reliable retrograde transneuronal tracer [START_REF] Ugolini | Advances in viral transneuronal tracing[END_REF] . By combining whole-cell recording and DNA delivery, with retrograde tracing using modified rabies virus, we have determined both the physiological properties and connectivity profile of a single neuron. By further combining this with either bolus loading of calcium dyes 42 or using virus-mediated genetically driven expression of calcium indicators [START_REF] Tian | Imaging neural activity in worms, flies and mice with improved GCaMP calcium indicators[END_REF][START_REF] Mank | A genetically encoded calcium indicator for chronic in vivo two-photon imaging[END_REF][START_REF] Lutcke | Optical recording of neuronal activity with a geneticallyencoded calcium indicator in anesthetized and freely moving mice[END_REF] it will be possible to implement in vivo two-photon based population imaging 2, 42, 46-48 (at least of cells in upper cortical layers) to investigate temporal correlations of firing of known presynaptic cells and the unique synaptic responses recorded from the postsynaptic neuron. It will be interesting, for example, to investigate whether commonalities in connectivity reflect any bias in function.

Only by probing both the function and connectivity of specific cells and circuits can we obtain a complete understanding of the functional architecture of the brain [START_REF] Douglas | Mapping the matrix: the ways of neocortex[END_REF] .

The method of DNA delivery via whole-cell recording described here provides a versatile new tool for assaying and manipulating both single cells and probing the functional connectivity of specific local and long-range networks.

METHODS

Intracellular solutions

Intracellular solutions for whole cell recordings were made up in concentrated stock (1.2-3 times the final concentration) to allow for plasmid addition. The final concentrations were (in mM, all from Sigma-Aldrich or VWR International, UK.) 130 K-methanesulphonate, 10 Hepes, 7 KCl, 0.05 EGTA, 2 Na 2 ATP, 2 MgATP, 0.5 Na 2 GTP; pH was adjusted to 7.28 using KOH. The final osmolarity after adding either 0.5% biocytin or suspended plasmids was adjusted to the range of 280-290 mOsm.

Intracellular solutions were filtered through a 0.2 µm pore size syringe filter (Whatman) or a 0.22 µm pore size centrifuge filter (Costar Spin-X). Spectrophotometry (NanoDrop 1000, Thermo Scientific) showed that DNA was not lost during filtration.

Plasmid production

The coding sequences of DsRed2, tdTomato, Venus, Cerulean or ChR2-Cerulean were subcloned into pCAGGs. For GFP we used pCA-b-GFPm5 silencer 3 (generous gift from Hendrik Wildner and Francios Guillemot). For rabies virus mediated transneuronal tracing, coding sequences of rabies SAD-ΔG and TVA800 [START_REF] Wickersham | Monosynaptic restriction of transsynaptic tracing from single, genetically targeted neurons[END_REF]

Discontinuation of whole-cell recording

After a 2-16 minute recording, the patch pipette was slowly withdrawn (~3 µm/sec) while the seal test pulse was monitored in voltage clamp mode to determine access resistance. Often this resulted in pulling an outside-out patch of membrane, which we consider to be an important though not an essential criterion for cell survival.

The electrical access to the cell could also be monitored by shifts in holding current due to ongoing synaptic input and in some cases the occurrence of unclamped action-currents. Applying a quick negative pressure pulse to abruptly break the seal with no indication of action currents indicated successful outside-out patch formation.

Synaptic receptive field mapping

Visual stimuli were generated using Matlab (Mathworks) and Psychophysics Toolbox 50 . Stimuli were presented on a 17 inch LCD monitor positioned 21 cm from the contralateral eye spanning 65° (vertical) and 81° (horizontal) of visual space.

Stimuli consisted of 100 % contrast square wave gratings (spatial frequency 0.04 cycles/°) drifting in one of eight directions at 2.6 cycles/second. For each trial, one direction was chosen randomly and presented in the following sequence: stationary (1 second) -moving (1 second) -stationary (1 second).

Virus injections

For injections, long-shanked, volume-calibrated pipettes (Blaubrand) were pulled to achieve a tip size of approximately 10 µm diameter and tip-filled using negative pressure [START_REF] Cetin | Stereotaxic gene delivery in the rodent brain[END_REF] . Animals were anesthetized as described above, sutures were removed and the Kwik-Sil covering the craniotomy was withdrawn. Injection pipettes were lowered as accurately as possible using the blood vessel pattern and under visual guidance along the same trajectory as recording pipettes to within 100-200 µm of the patched cell. Approximately 150-200 nl of virus solution was injected under 50 mbar of positive pressure for 2-3 minutes. The pipette was withdrawn, Kwik-Sil reapplied and the animal sutured and recovered.

Tissue processing and imaging

Brains were analyzed 1-9 days after recording. Deeply anaesthetised mice were usually transcardially perfused with cold PBS (0.1 M) followed by 4% paraformaldehyde (PFA) in PBS (0.1 M), though sometimes directly decapitated, and the brain was fixed in 4% PFA at 4 °C overnight. Coronal brain sections (100-200 µm) were cut on a vibrating microtome (Leica VT1200S or Microm HM 650V) and rinsed in PBS. Fluorescence images were acquired on a Leica SP1, SP5 or a Zeiss 510 laserscanning confocal microscope. For experiments containing biocytin in the intracellular solution, slices were first treated with 1% Triton-X100 in PBS for 12 hours then incubated in a PBS solution containing either Atto-565-or Alexa-488conjugated streptavidin (Sigma-Aldrich 1:100-1:200), 5% normal goat serum and 0.1% Triton-X for 12 hours at 4˚C. For immunostaining for GFP and mCherry, rabbit anti-GFP (Invitrogen 1:1000) and rat anti-RFP primary antibodies (Chromotek 1:1000; 24-48 hours at 4 °C) were visualized with donkey Alexa-488 anti-rabbit (Invitrogen 1:500) and goat Alexa-568 anti-rat (Invitrogen 1:500, 24 hours at 4 °C) secondary antibodies. After thorough washing in PBS, slices were mounted with an antifading medium (Mowiol 4-88, Calbiochem + 2.5% DABCO 33-LV, Sigma-Aldrich) and imaged as described above.

Figure 1 :

 1 Figure 1: Recording methodology and biocytin recovery rates a, Seal test current trace used to monitor pipette resistance. b, Contact with cell membrane while stepping the electrode indicated by rhythmic changes in pipette resistance associate with heartbeat-related movement. c, Traces showing seal formation and d, whole-cell access. e, IV relationship (current injections 50 pA steps for 600 ms) recorded from the cell shown in i. f, Current traces in response to voltage steps immediately prior and g, during pipette retraction whereby the outside-out patch configuration is achieved h. Inset trace: detail from h showing single channel currents. i, Example morphology of a biocytin-labeled layer 2/3 pyramidal cell (animal sacrificed 29 hours after recording). j, Success rates of biocytin-filled cell recovery plotted against series resistance of recording (determined within 30 seconds after break-in). Circles represent, on a cell-by-cell basis, recovery success or failure. Grey-filled circles indicate those recordings that did not meet retraction and/or series resistance criteria. Large open red circles represent the mean success for each bin (range: 20-40, 41-50, 53-70, 71-200 MΩ). All error bars are SD. Numbers indicate the n for the respective bin. Dashed lines are linear fits to the individual data points. k, Success rates of cell recovery plotted against duration of recording (bin ranges: 1-4, 5-6, 7-9 and 10-13 mins) and l, the interval between recording and sacrificing the animal (bin ranges: 23-24, 25-29, 31-45, 47-51 hours).

Figure 2 :

 2 Figure 2: Physiological characterization and genetic manipulation a, The firing profile and IV relationship recorded from the layer 5 neuron shown in panels d1 and d2. Voltage traces shown are in response to 50 pA current steps of 600 ms duration. b, Membrane voltage traces showing spontaneous synaptic activity and the kinetics of a resultant action potential. c1, Synaptic responses (average of 10 sweeps) to drifting gratings moving in the direction indicated by the arrows (left). Arrow (top) indicates onset of grating drift. c2, Polar plot showing integral of the membrane voltage traces (for 1400ms from stimulus onset). d, Layer 5 pyramidal cell in primary visual cortex 26 hours after recording showing GFP expression (d1) and biocytin-labeling (d2). Note the weak, non-specific biocytin signal along the electrode track due to the plume of biocytin expelled while approaching the recorded cell. e, GFP signal in dendritic spines and axonal arborizations in the same cell. f, Success rates of GFP-labeled cell recovery plotted against series resistance of recording (determined within 30 seconds after break-in). Circles represent, on a cellby-cell basis, recovery success or failure. Grey-filled circles are cases that did not meet retraction and/or series resistance criteria. Open green circles represent the mean success for each bin (range 20-40, 41-50, 53-70, 71-200 MΩ. All error bars are SD. Numbers indicate the n for the respective bin. Dashed lines are linear fits to the individual data points. g, Success rates of cell recovery plotted against duration of recording (bin ranges: 1-4, 5-6, 7-8, 9-13 mins) and h, the interval between recording and sacrificing the animal (bin ranges: 23-25, 26-28, 29-33, 40-47, 48-51 hours).

Figure 3 :

 3 Figure 3: Multiple gene delivery a, Native fluorescence images of a layer 5 neuron patched in motor cortex 3 days after patching with pCAGGS-tdTomato (top right) and pCAGGS-Cerulean (50 ng/µl each bottom right), scale bar: 1 mm and 20 µm. b, Gallery of native fluorescence images for a layer 5 cell in primary somatosensory cortex 5 days after recording with pCAGGS-DsRed2, pCAGGS-Venus and pCAGGS-ChR2-Cerulean. Scale bar: 20 µm.

Figure 4 :

 4 Figure 4: Synaptic receptive mapping and connectivity of a visual cortical neuron a, The firing profile and IV relationship recorded from the GFP-expressing cell shown in panel d. Voltage traces shown are in response to 50 pA current steps of 600 ms duration. b, Synaptic responses (average of 10 sweeps) to drifting gratings moving in the direction indicated by the arrows (left). Arrow (top) indicates onset of drift. Current injection (-200 pA) was delivered to isolate evoked synaptic responses from AP activity. c, Polar plot showing integral of the voltage traces (red) recorded during the duration of the drifting grating. The integral of synaptic responses (1400 ms from stim onset) are normalized to largest response. Below are 5 example traces recorded for the least and most preferred orientation. For the spiking receptive field, holding current was removed and the stimulus sequence (b) was repeated five times. d, Coronal slice (100 µm thick) showing anti-GFP (green) and anti-RFP (red) fluorescence. The recorded layer 5 neuron (green-yellow) in primary visual cortex (V1) and mCherry-labeled cells (red) are clearly identifiable. In this slice approximately 200 presynaptic cells were observed. Scale bar = 500 µm. Abbreviations: V2ML: secondary visual cortex mediolateral, V2L: secondary visual cortex lateral, CA1-3: hippocampus fields CA1-3, DLG: dorsolateral geniculate nucleus. e, High magnification image showing GFP and mCherry labeled cells in V1. Scale bar = 100 µm. f. In this slice 8 mCerry-labeled cells were observed in the DLG. Scale bar = 50 µm. g. Five maximum intensity projections for consecutive 100 µm thick slices. The middle panel contains the host cell (green; scale bar: 200 µm). This mouse was sacrificed 7 and 9 days after virus injection and recording respectively.

  were linked with an internal ribosomal entry site (IRES2. DB Biosciences) and sub-cloned into pCA-b-GFPm5 silencer 3 by replacing the GFP coding sequence. Correct insertion and coding sequences were verified by subsequent DNA sequencing. Resulting plasmids were transformed and individual clones were grown in E.coli Stbl3 (Invitrogen) cells to avoid recombination. DNA for in vivo recordings was prepared using an endotoxin-free maxiprep kit (Qiagen) and suspended in ultrapure H 2 O or TE buffer (Qiagen). Plasmid aliquots (10 µl) were diluted in H 2 O and added to the intracellular solution for a final concentration of ~ 50-350 ng/µl. Production of pseudotyped rabies virus BHK cells were plated at a density of ~1.5 x 10 7 . The following day, cells were transfected with 15 μg plasmid pCAGG/SADΔG (generous gift from Dr. Karl-Klaus Conzelmann) by CaP transfection. 24 h later rabies virus SADΔG-mCherry was added at a MOI of 3. 48 h later the SADΔG-mCherry containing supernatant was equally distributed onto four 15cm plates containing pCAGGs/SAD-G (15 μg/plate) transfected BHK cells (~1.5 x 10 7 cells/plate). Two days later the virus containing supernatant was applied onto four 15cm plates containing BHK-EnvARGCD cells 23 (~1.5 x 10 7 cells/plate) at a MOI of 1.5 for pseudotyping. 12 hours later cells were trypsinized and replated onto eight 15cm dishes. Pseudotyped rabies viruscontaining supernatant was harvested 2 days later. The supernatant was spun at 2000 rpm at 4°C for 10 min. and subsequently filtered through a 0.45 μm filter. The filtered virus suspension was centrifuged for 90 min at 25000 rpm (SW28, 4 °C) in a Beckmann 80K ultracentrifuge. After centrifugation the supernatant was discarded and the pellet was aspirated in PBS. Pseudotyped rabies virus-containing solution was aliquoted in 10 μl aliquots and frozen at -70 °C. Virus titers were determined by serial dilution and over night infection of primary cortical neurons that had been infected with rAAV1/2 expressing TVA800-IRES-GFP 23 under control of the human synapsin promoter/ enhancer 49 . Three days later the number of red fluorescent, SADΔG-mCherry containing neurons were counted. Biological titers on TVA-electrodes were pulled from borosilicate glass (GC150F, Harvard Apparatus Ltd, UK) on a vertical puller (PC-10, Narishige, Japan) to a tip resistance of 6-7 MΩ. Recordings were carried out with an AxoClamp 2B or Mulitclamp 700B amplifier (Axon Instruments, USA). Data were filtered at 3 KHz and digitized at 10-20 kHz using an ITC-18 A/D-D/A interface (InstruTECH, Heka Elektronik, Germany) and the Neuromatic package (www.neuromatic.thinkrandom.com) under Igor Pro 5 (www.wavemetrics.com).

  Figure 2 (Rancz etal)

Acknowledgements

KMF thanks Steven Siegelbaum and Richard Axel for their generous support and encouragement, and Abigail Mulligan for technical assistance. We also thank Peter H. Seeburg for support and Diogo Castro, Ben Matynoga and Daniela Drechsel for advice. We thank Karl-Klaus Conzelmann for gifting rabies virus SADΔG-mCherry and Péter Boross, Ed Callaway, Hendrik Wildner and Francois Guillemot for helpful discussions and reagents. EAR is a Sir Henry Wellcome Postdoctoral Fellow. This work was supported by The Robert Leet and Clara Guthrie Patterson Trust and a NIDCD K99 grant (KMF), The Max-Planck Gesellschaft (ATS, MS), The MRC and The Alexander Von Humboldt Foundation (TWM).

expressing neurons for SADΔG-mCherry for in vivo injections were ~2.5 x 10 7 /ml.

Surgical procedures

Young adult C57/BL6 mice (4-6 weeks old) were anaesthetized with a mixture of ketamine and xylazine (100 mg/kg and 10 mg/kg, respectively) injected intraperitoneally, supplemented as necessary whenever the animal showed toepinch reflexes. Mice were head-fixed using non-puncture ear-bars and a nose-clamp (SG-4N, Narishige, Japan), and their body-temperature maintained at 36-37 °C using a rectal probe and a heating blanket (FHC, Bowdoinham, ME, USA). An incision was made in the scalp and a small craniotomy was drilled above the relevant area of cortex using a dental drill (Osada Electric Co, Japan) and the dura removed. The surface of the brain was kept moist by applying sterile phosphate buffered saline solution (PBS, Sigma Aldrich) at regular intervals. Following recordings, the craniotomy was sealed using a silicone sealant (Kwik-Sil, World Precision Instruments) and the scalp sutured. The wound was infiltrated with lidocaine and a topical antibiotic (Cicatrin, GlaxoSmithKline, UK) was applied. The time from the initial anaesthesia to completing this procedure was kept as short as possible (< 2 hours). During recovery, mice were kept warm in a heating box (Harvard Apparatus, Holliston, MA, USA) for 3-4 hours under constant observation. All procedures were approved by the local ethics panel and UK Home Office under the Animals (Scientific Procedures) Act 1986.

Obtaining whole-cell recording

In vivo whole-cell recordings were carried out as described previously [START_REF] Margrie | In vivo whole-cell recordings from neurons in the anaesthetized and awake mammalian brain[END_REF] . Briefly, glass


Viet-Anh Tran
email: viet-anh.tran@gipsa-lab.inpg.fr

Gérard Bailly
email: gerard.bailly@gipsa-lab.inpg.fr

Hélène Loevenbruck
email: helene.loevenbruck@gipsa-lab.inpg.fr

Tomoki Toda
email: tomoki@is.naist.jp

Improvement to a NAM-captured whisper-to-speech system

Keywords: non-audible murmur, whispered speech, audiovisual voice conversion, silent speech interface

Exploiting a tissue-conductive sensor -a stethoscopic microphone -the system developed at NAIST which converts Non-Audible Murmur (NAM) to audible speech by GMM-based statistical mapping is a very promising technique. The quality of the converted speech is however still insufficient for computer-mediated communication, notably because of the poor estimation of F 0 from unvoiced speech and because of impoverished phonetic contrasts. This paper presents our investigations to improve the intelligibility and naturalness of the synthesized speech and first objective and subjective evaluations of the resulting system. The first improvement concerns voicing and F 0 estimation. Instead of using a single GMM for both, we estimate a continuous F 0 using a GMM, trained on target voiced segments only. The continuous F 0 estimation is filtered by a voicing decision computed by a neural network. The objective and subjective improvement is significant. The second improvement concerns the input time window and its dimensionality reduction: we show that the precision of F 0 estimation is also significantly improved by extending the input time window from 90 to 450ms and by using a Linear Discriminant Analysis (LDA) instead of the original Principal Component Analysis (PCA). Estimation of spectral envelope is also slightly improved with LDA but is degraded with larger time windows. A third improvement consists in adding visual parameters both as input and output parameters. The positive contribution

ACCEPTED MANUSCRIPT

of this information is confirmed by a subjective test. Finally, HMM-based conversion is compared with GMM-based conversion.

Introduction

In recent years, advances in wireless communication technology have led to the widespread use of cellular phones for speech communication. Because of noisy environmental conditions and competing surrounding conversations, users tend to speak loudly. As a consequence, private policies and public legislation tend to restrain the use of cellular phones in public places. Silent speech which can only be heard by a limited set of listeners close to the speaker is an attractive solution to this problem if it can efficiently be used for quiet and private communication. Silent speech capture interfaces have already been developed including electromyography ( (Hueber, Chollet et al. 2007). By introducing linguistic levels both in the recognition and synthesis, such systems can potentially compensate for the impoverished input by including linguistic knowledge into the recognition process. The quality of the output speech is either excellent or extremely degraded depending on recognition performance. (b) Direct signal-to-signal mapping using aligned corpora [START_REF] Toda | NAM-to-Speech Conversion with Gaussian Mixture Models[END_REF]. By using the NAM microphone to capture nonaudible murmur, Toda et al. proposed a NAM-to-speech conversion system based on a GMM model in order to convert "non-audible" speech to audible speech. It was shown that this system provides intelligible speech with constant quality but the naturalness of the converted speech is still unsatisfactory. This is due to the poor F 0 estimation from unvoiced speech. Note that the F 0 estimation problem is a difficulty shared by all the systems described above, either using whispered speech, or non-audible murmur. Although whispered speech typically does not involve any vocal fold vibration, laryngeal activity may however exist during whispered speech, and this could represent useful information for the F 0 estimation process. [START_REF] Coleman | Larynx movements and intonation in whispered speech, Summary of research supported by British Academy[END_REF] have shown, using dynamic Magnetic Resonance Imaging, that larynx movements related to intonation changes (rising and falling pitch) can be observed in whispered speech. According to the authors this offers an explanation for perceived pitch in whispered speech: laryngeal movements modify the shape of the oral cavity, thus altering the vocal tract acoustics so that pitch changes can be inferred. Subvocal speech, such as the murmur that can sometimes be observed in hallucinating schizophrenic patients, has also been shown to be associated with laryngeal activity. [START_REF] Inouye | The electromyographic study of verbal hallucinations[END_REF] reported increased EMG activity in speech-related muscles including laryngeal muscles (cricothyroid, sternohyoid, orbicularis oris, and depressor anguli oris) in 47.6% of the hallucinations of nine schizophrenic patients. These works suggest that whispered speech might carry information on laryngeal activity. This activity could be useful in the recovery of F 0 because it could modify the shape of the oral cavity and hence be audible. It has been shown by [START_REF] Higashikawa | Perceived pitch of whispered vowels -relationship with formant frequencies: A preliminary study[END_REF] that when speakers try to whisper high pitch vowels vs low pitch vowels, F1 and F2 are raised and listeners are able to identify the pitch correctly. This formant frequency raising could be due to Section 2 gives a brief description of the NAM microphone. Section 3 describes the original NAMto-speech system proposed by Toda et al. Section 4 then describes our improvement for the performance of this system, especially for the naturalness of the converted speech while section 5 focuses on the contribution of visual features. Session 6 is dedicated to the testing of a HMM-based conversion system. Finally, a discussion of our results is provided in section 7.

NAM microphone

The tissue-conductive microphone proposed by [START_REF] Nakajima | Non-audible murmur recognition Input Interface using stethoscopic microphone attached to the skin[END_REF] comprises an electret condenser microphone (ECM) covered with a soft polymer material, such as soft silicone or urethane elastomer, which provides better impedance matching with the soft tissue of the neck. This microphone can capture acoustic vibrations in the vocal tract from a sensor placed on the skin, below the ear (Figure 1). This position allows a high quality recording of various types of bodytransmitted speech utterances, such as normal speech, whisper and non-audible murmur, even in the environmental noises. Body tissue and lip radiation act as a low-pass filter and the high frequency components are attenuated. However, the non-audible murmur spectral components still provide sufficient information to distinguish and recognize sound accurately. The soft silicone NAM microphone used in our laboratory can record sound with frequency components up to 3 kHz while being little sensitive to external noise. Figure 2 shows an example of a whispered utterance in French, captured by this microphone.

Original NAM-to-Speech conversion system

To characterize whisper, namely unvoiced speech without vocal fold vibration, spectral envelope and power information are used, while spectral envelope, power, F 0 and aperiodic components are used for speech synthesis using STRAIGHT, a versatile speech vocoder developed by [START_REF] Kawahara | Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequencybased F0 extraction: Possible role of a repetitive structure in sounds[END_REF] .

Before training the models for spectrum and excitation estimation, pairs of whisper and speech uttered by a speaker must be aligned, in order to determine corresponding frames. Phonetic transcription information were used to get a better alignment compared to automatic Dynamic Time Warping (DTW).

ACCEPTED MANUSCRIPT

Spectral estimation

We use the same schema for spectral estimation as the one proposed by Toda et al. (

, X Y Τ Τ Τ     , 2 2 , X Y Τ Τ Τ     , …, , T T X Y Τ Τ Τ   
 , the joint probability density of the input and output parameter vectors is modelled by a GMM as follows:

( , )

) ([ , ] ; , ) M X Y X Y t t m t t m m m P X Y w X Y λ µ Τ Τ Τ = = Ν Σ ∑         = ) ( ) ( ) , ( Y m X m Y X m µ µ µ ,         Σ Σ Σ Σ = Σ ) ( ) ( ) ( ) ( ) , ( YY m YX m XY m XX m Y X m ( , ) 1 ( ,
where N( , ) denotes Gaussian distribution with a mean vector and a covariance matrix . The mixture component index is m. The total number of mixture components is M. A parameter set of the GMM is λ , which consists of weights w m , mean vectors

Σ

for individual mixture components.

The probability density of the GV of the output static feature vectors over an utterance is also modelled by a Gaussian distribution,

) , ); ( ( ) ) ( ( ) ( ) ( ) ( v v v y v N y v P Σ = µ λ
where the GV ( ) (1),..., ( )

y v y v v D Τ   =   is calculated by ∑ ∑ = =       - = T t T t d y T d y T d v 1 2 1 ) ( 1 ) ( 1 ) ( τ τ A parameter set ) (v λ consists of a mean vector ) (v µ and a diagonal covariance matrix ) (v
Σ . This global variance information is used to reduce the over-smoothing, which is inevitable in the conventional ML-based parameter estimation [START_REF] Toda | Speech parameter generation algorithm considering global variance for HMM-based speech synthesis[END_REF]).

In the conversion procedure, let 1 ,..., ,...,

t T X X X X Τ Τ Τ Τ   =   and 1 ,..., ,..., t T Y Y Y Y Τ Τ Τ Τ   = 
 be the time sequence of the input parameter vectors and that of the output parameter vectors, respectively. The converted static feature sequence 1 2

,..., ,..., T y y y y

Τ Τ Τ Τ   = 
 is determined by maximizing a product of the conditional probability density of Y given X and the GV probability density as follows:

) ) ( ( ) , ( max arg ˆ) ( v w y y v P X Y P y λ λ = subject to y W Y y =
where y W is a window matrix to extend the static feature vector sequence to the parameter vector sequence consisting of static and dynamic features.

Excitation estimation

The speech excitation is decomposed into two components: a periodic or quasi-periodic component which takes into account the quasi-periodic segments of speech produced by the regular vibrations of the vocal folds and an aperiodic component for the noise (frication, aspiration, bursts). In the STRAIGHT system (Kawahara, Masuda-Katsuse et al. 1999), the mixed excitation is defined as the frequency-dependent weighted sum of these two components. The weight is determined based on an aperiodic component in each frequency band which is calculated from the smoothed power spectrum by a subtraction of the lower spectral envelope from the upper spectral envelope. The upper envelope is calculated by connecting spectral peaks and the lower envelope is calculated by connecting spectral valleys.

Two GMM models for F 0 estimation and aperiodic estimation are constructed in the same way as the spectral estimation except that the global variance (GV) is not used because GV does not cause large differences to the converted speech in the aperiodic conversion. Static and dynamic features Y t of F 0 and aperiodic components are used while keeping the same feature vector of whisper X t as that used for the spectral estimation.

Improvement of intelligibility and naturalness of the converted speech

This section is described in more details in [START_REF] Tran | Predicting F0 and voicing from NAM-captured whispered speech[END_REF].

Feature extraction

The training corpus consists in 200 utterance pairs of whisper and speech uttered by a native male speaker of French and captured by a NAM microphone and a head-set microphone. Respective speech durations are 4.9 minutes for whisper (9.7 minutes with silences) and 4.8 minutes for speech (7.2 minutes with silences). The 0 th through 24 th mel-cepstral coefficients are used as spectral features at each frame. The spectral segment features of whisper are constructed by concatenating feature vectors at each current whispered frame ± 8 frames (i.e. representing 90 ms of speech) to take into account the context. Then the vector dimension is reduced to 50 using a PCA technique.

Log-scaled F 0 extracted with fixed-point analysis and 5 average dB values of the aperiodic components on five frequency bands (0 to 1, 1 to 2, 2 to 4, 4 to 6, 6 to 8 kHz) are used to characterize the excitation feature.

The test corpus consists of 70 utterance pairs not included in the training data.

Voicing decision

In the original system, F 0 and voicing are jointly estimated by a unique GMM model: unvoiced frames are assigned with F 0 = 0 for training and voicing decision is then determined using a simple threshold. The main drawback of this approach is to waste Gaussian components to represent zero values of F 0 for unvoiced segments and its danger is to predict unstable F 0 values for unvoiced segments incorrectly labelled as voiced. In order to improve this joint representation, we only use voiced segments to train the GMM model for F 0 . For synthesis, a feed-forward neural network is used to predict the segments from the whispered speech. This network has 50 input cells (for the 50 dimension feature vector), 17 hidden cells, and 1 continuous output. The continuous output is converted to a binary output by applying a threshold. When the output is larger than .5, the frame is labelled as voiced. Otherwise, it is labelled as unvoiced. Only voiced frames are used in the GMM.

Continuous F 0 values are then predicted, from the voiced values. Figure 3 presents the synopsis of the system.

Figure 3. Synopsis of the whisper-to-speech conversion method

Table 1 shows the detection performance of this network in comparison with the error in the original system. We have a significant improvement of the voiced/unvoiced detection.

F 0 estimation

As stated above, only voiced segments in whispered speech were used to train the GMM model for the F 0 estimation. We also compared our system with the original one with different number of mixtures on both the training and the test data. Full covariance matrices were used for both GMMs.

The difference was calculated as the normalized difference between synthetic F 0 and natural F 0 in the voiced segments that were well detected by the two systems. This difference is given by the following formula: Diff = (synthetic_F 0 -natural_F 0 )/ natural_F 0 . Figure 4 shows that the proposed framework outperforms the original system. In addition, when the number of Gaussian mixtures increases, the errors of both systems on the training data decrease, but the errors on test data are not sensitive to the number of mixtures.

Perceptual Evaluation

Sixteen French listeners who had never listened to NAM participated in our perceptual tests on intelligibility and naturalness of the converted speech from the two systems. We used 20 test utterances not included in the training set.

Each listener heard an utterance pronounced in modal speech and the converted utterances obtained from the whispered speech with both systems. For intelligibility testing, all parameters were obtained by voice conversion and synthesis was performed using STRAIGHT (Kawahara, Masuda-Katsuse et al. 1999). For naturalness, the spectral parameters were original and the stimuli were obtained by substituting only predicted voicing and F 0 values to the original warped target frames.

This procedure was chosen because the quality of the converted spectrum could also influence the perception in the naturalness test. We wanted to make sure that we were testing F0 and voicing alone (not other potentially influential parameters).

For each utterance, listeners were asked which one was closer to the original one, in terms of intelligibility and in terms of naturalness. An ABX test (or matching-to-sample test) was used. It is a discrimination procedure involving presentation of two test items and a target item. The listeners are asked to tell which test item (A or B) is closest to the target (X). Figure 6.a displays the mean intelligibility scores for all the listeners for the converted sentences using the original system and our new system. An ANOVA showed that the intelligibility score is higher for the sentences obtained with our new system (F = 23.41, p <.001). Figure 6.b shows the mean naturalness. Again the proposed system is strongly preferred to the original one, as shown by an ANOVA (F = 74.89, p <.001).

. F0 estimation

For this evaluation, the whispered frames are classified into 13 classes: unvoiced frames are labelled with '0' label and voiced frames fall into 12 other labels, depending on which interval the F 0 value in this frame belongs to (bark scale between 70Hz and 300 Hz). The class of a whispered frame is deduced from the class of the corresponding speech frame using the warp path. The number of Gaussian mixtures for F 0 estimation varied from 8 to 64 [START_REF] Heracleous | A tissue-conductive acoustic sensor applied in speech recognition for privacy[END_REF][START_REF] Jorgensen | Web browser control using EMG-based subvocal speech recognition[END_REF][START_REF] Tran | Predicting F0 and voicing from NAM-captured whispered speech[END_REF]64). The size of the context window was also varied from the phoneme size (90 ms) to the syllable size (450 ms) (by picking one frame every 1-5 frames).

Table 2 shows that using LDA and a large window size improves the precision of pitch estimation with respect to PCA with a small window. When using LDA with a window of 5, the F 0 error decreases by 16% compared to our previous system with PCA and a small window size (10.90% → 9.15%). However, using LDA instead of PCA with the same context window size does not significantly improve the precision (9.17% → 9.15%).

Spectral estimation

We also evaluated the influence of LDA and long-term spectral variation to the spectral estimation.

The phonetic segmentation was used here to perform the LDA: each whispered frame was classified into one of 34 classes, depending on which phonetic segment it belonged to.

Table 3 provides the cepstral distortion between the converted and the target speech (the higher the distortion, the worse the performance). It shows that LDA is slightly better than PCA. But contrary to F 0 estimation, the spectral distortion increases when the size of the time window increases. The most plausible interpretation is that a phoneme-sized window optimally contains necessary contextual cues for spectral conversion.

Audiovisual corpus

An audiovisual conversion system was built using audiovisual data pronounced by a native Japanese speaker. The system captures, at a sampling rate of 50 Hz, the 3D positions of 142 coloured beads glued on the speaker' s face (see Figure 8) in synchrony with the acoustic signal sampled at 16000 Hz.

Visual parameters extraction and shape modelling

A shape model is built using a so-called guided Principal Component Analysis (PCA) where a priori knowledge is introduced during the linear decomposition. We compute and iteratively subtract predictors using carefully chosen data subsets [START_REF] Badin | Threedimensional linear articulatory modeling of tongue, lips and face based on MRI and video images[END_REF]. For speech movements and for each particular speaker and language, this methodology extracts 5 components that are directly related to the rotation of the jaw, to lip rounding, to upper and lower lip vertical movements and to movements of the throat that are associated with underlying movements of the larynx and hyoid bone. The resulting articulatory model also includes components for head movements and facial expressions but only components related to speech articulation are considered here.

Contribution of visual parameters for the conversion system

For this evaluation, the database consists of 150 sentences for training and 40 sentences for testing, pronounced by a native Japanese speaker. The audiovisual feature vector combines whispered spectral and visual feature vectors in an identical way to the AAM (Active Appearance Models) introduced by Cootes [START_REF] Cootes | Active Appearance Models[END_REF] where two distinct dimensionality reductions by PCA are performed in shape and appearance and further combined into a third PCA to get a joint shape and appearance model. Acoustic and visual features are fused here, using an identical process. The 0 th through 19 th mel-cepstral coefficients are used as spectral features at each frame.

The input feature vector for computing speech spectrum is constructed by concatenating feature vectors at current ±8 frames and further reduced to a 40-dimention vector by a PCA. Similarly to the processing of the acoustic signal, each visual frame is interpolated at 200 Hz -so as to be synchronous with the audio processing -and characterized by a feature vector obtained by concatenating and projecting ±8 frames centred around the current frame on the first n principal components. The dimension of the visual vector n is set to 10, 20 or 40. Prior to joint PCA, the visual vector is weighted. The weight w was also changed from 0.25 to 2. The conversion system uses the first 40 principal components of this joint audiovisual vector. In the evaluation, the number of Gaussian is fixed at 16 for the spectral estimation, 16 for the F 0 estimation and 16 for the aperiodic components estimation.

Voicing decision

In the same way as for the evaluation of voicing decision described in section 4.2, the audiovisual vectors of whisper in the training corpus of the conversion system are used to train the network.

This network has 40 input neurons, 17 hidden neurons and 1 output neuron. Table 4 shows that visual information improves the accurateness of voicing decision. With a visual weight empirally set at 0.25, the voicing error is decreased by 5.4 % (8.05 % → 7.62%) compared to audio-only and 21.9 % compared with the baseline system (9.76 % → 7.62%).

Table 5. Influence of visual information on the estimation of spectral and excitation features.

Visual weight Distorsions Visual dimension

Audioonly 0.25 0.5 0.75 1 1. [START_REF] Sodoyer | Developing an audio-visual speech source separation algorithm[END_REF] Figure 12 displays the evolution of the identification scores along the four conditions ('A from A', 'AV from A', 'A from AV', 'AV from AV') for the five articulatory groups. The average of the correct identification rates for each consonant in an articulatory group was calculated to provide an overall correct identification rate for each group. This articulatory grouping shows that visual information significantly helps the participants to identify bilabials. The identification ratio for the bilabial consonants rises from 11.67 % in the 'A from A' condition to 39.83 % in the 'AV from AV' condition (F = 2.62, p < 0.079). The identification ratios for all the other groups are quite stable and are not significantly increased by the addition of visual information.

Table 6 provides the confusion matrices with places of articulation chosen among 5 categories (bilabial, alveolar, palatal, fricative, or other), rather than among 27 individual consonants. The focus here is on the broad category the consonant belongs to, not on the score of individual consonants. Interestingly, the bilabial place of articulation, which is poorly identified in the 'A from A' and 'AV from A' conditions (less than 20 %), becomes quite well identified in the 'AV from AV' condition (75 %). When the bilabial place of articulation is wrongly identified, it is most often mistaken for what we named "non-alveolar fricatives", a group which contains labiodentals or rounded consonants. This suggests that the labial place of articulation remains quite perceptible.

Table 6 also shows that the alveolar place of articulation is quite well recovered from the audio alone (approximately 60 %). Although its recovery does benefit from the addition of visual information, the 'AV from AV' score does not reach as high a score as the one for the bilabial place of articulation (70.83 % vs. 75 %). We must recall here that the visual synthesis used in this study does not provide information on tongue movements, which could in fact be visible for alveolar consonants. This probably explains why the addition of visual information does not provide such a drastic improvement on the scores. The palatal place of articulation reaches a good score (above 50 %) but does not benefit much from visual information, as could be expected. The other two groups do not benefit from visual information and are not well identified (less than 50 %).

Experiments and results

The same Japanese data (cf. section 5.1) are used: 150 utterances for training and 40 utterances for test. The 0 th through 19 th mel-cepstral coefficients and their deltas are used as spectral features for both aligned speech and whisper. With a recognition rate of 68.17 % for the input whisper, the spectral distortion between converted speech and target speech is 6.46 dB, compared with 5.99 dB for the GMM-based system. Note that our recognition rate is quite similar to the 60 % obtained by Hueber et al. (Hueber, Chollet et al. 2007) using context-independent phones and multimodal input.

The degradation, compared with the GMM-based system, could be explained by the low recognition rate which could be improved by using Gaussian mixtures and more contextual information to expand HMM modelling.

Conclusion and perspectives

This paper proposes several solutions to improve the intelligibility and the naturalness of the speech generated by a whisper-to-speech conversion system. The original system proposed by Toda et al. is based on a GMM model predicting the three parametric streams of the STRAIGHT speech vocoder (F0, harmonic and noise spectrum) from spectral characterization of the non-audible-murmur input.

The first improvement concerns characteristics of the voiced source. We have shown that the estimation of voicing and F 0 by separate predictors improves both predictions. We have also shown that F0 prediction is improved by the use of a large input context window (>400 ms) compared to the original smaller window (90ms). Predictions of all parametric streams are further improved by a data reduction technique that makes use of the phonetic structure of the speech stream.

The second part of the paper compared objective and subjective benefits offered by multimodal data. Improvements obtained by adding visual information in the input stream as well as including articulatory parameters in the output facial animation are very significant.

We will explore several other research directions in the near future: we first plan to use two NAM microphones and source separation techniques to have better signal-to-noise ratios in the input.

Source separation can benefit from the joint audiovisual model implicitly captured by the voice conversion system. Such a model can in fact provide very efficient a priori information for source deconvolution techniques [START_REF] Sodoyer | Developing an audio-visual speech source separation algorithm[END_REF]). Finally the voice conversion system can also benefit from other input sources such as ultrasound imaging, surface EMG or EEG (electroglottography) that can provide additional information on the underlying articulatory speech movements.

A short-term perspective is to develop a portable real-time voice conversion system that will allow us to conduct usage studies and test several experimental variables that are still unexplored, notably the impact of auditory feedback -using the amplified NAM signal or predicted speech -on silent speech production as well as that of conversion delay on production and conversation.

  Bett and Jorgensen 2005; Jorgensen and Binsted 2005; Jou, Schultz et al. 2006; Walliczek, Kraft et al. 2006), non-audible murmur microphone (Heracleous, Nakajima et al. 2005; Toda and Shikano 2005), ultrasound and optical imagery (Hueber, Aversano et al. 2007; Hueber, Chollet et al. 2007; Hueber, Chollet et al. 2007; Hueber, Chollet et al. 2008; Hueber, Chollet et al. 2008) together with several techniques to convert silent speech signals to audible speech. Two possible ways to map silent speech to modal speech have been proposed: (a) Combining speech recognition and synthesis techniques as proposed by Hueber et al. in the Ouisper project

  laryngeal movement. Other sources of pitch information may exist. Zeroual et al. (2005) and Crevier-Buchman et al. (2008) have shown with ultra-high-speed cinematography that whisper is associated with an anterior-posterior epilaryngeal compression and an abduction of the ventricular bands. These supraglottic changes may also be sources of information on F 0 during whisper. In this article, we propose two major improvements to the original signal-based GMM mapping from whisper to speech proposed by Toda et al. As advocated by Nakagiri et al. (2006), whisper is used here instead of NAM because of the difficulty of getting accurate phonetic NAM segmentation. The first improvement concerns the input feature space. The second improvement consists in incorporating visual information into both the input and output feature spaces. The visual parameters are obtained by the face cloning methodology developed at Gipsa-Lab (Revéret, Bailly et al. 2000; Badin, Bailly et al. 2002; Bailly, Bérar et al. 2003).

Figure 1 :Figure 2 :

 12 Figure 1 : Position of the NAM microphone

Figure 4 .

 4 Figure 4. Parameter evaluation on training (solid line) and test corpus (dashed line).

Figure 5

 5 Figure 5 shows an example of a natural (target) F 0 curve and the synthetic F 0 curves generated by

Figure 5 .

 5 Figure 5. Natural and synthetic F 0 curve for the utterance: "Armstrong tombe et s' envole".

Figure 6 .

 6 Figure 6. ABX results. (a) intelligibility ratings (b) naturalness ratings

Figure 7

 7 Figure 7 shows an example of a natural (target) F 0 curve and the synthetic F 0 curves generated by

Figure 7 .

 7 Figure 7. Comparing natural and synthetic F 0 curves

  To convey a message, humans produce sounds by controlling the configuration of oral cavities. The speech articulators determine the resonance characteristics of the vocal tract during speech production. Movements of visible articulators such jaw and lips are known to significantly contribute to the intelligibility of speech during face-to-face communication[START_REF] Summerfield | Use of visual information for phonetic perception[END_REF][START_REF] Summerfield | Lips, teeth, and the benefits of lipreading[END_REF]). In the field of person-machine communication, the visual information can be helpful both as input and output modalities[START_REF] Bailly | Audiovisual speech synthesis[END_REF][START_REF] Potamianos | Joint audiovisual speech processing for recognition and enhancement[END_REF].

Figure 8 .

 8 Figure 8. Characteristic points used for capturing the movements. The mesh used to capture and train the texture model is superimposed to the original video-captured images.

Figure 12 .

 12 Figure 12. Recognition ratios for different groups of consonants, classified according to their places of articulation.

  Toda and Shikano 2005). As described in Toda et al. (2009), let us use an input static feature

	x	t	=	[	x	t	(1),..., ( ) t x x D	] Τ	and an output static feature vector	y	t	=	[	y	t	(1),..., ( ) t x y D	] Τ	at frame t,
	respectively. In order to compensate for the lost characteristics of some phonemes due to body
	transmission, we use a segment feature	t X W x x t L Τ - = 	,... , t x x Τ	t L Τ +	 	Τ	+	x b	extracted over several frames
	(t ± L) as an input speech parameter vector where W x and b x are determined by PCA in this paper.
	As an output speech parameter vector, we use	t Y	=	 	y	t Τ	,	y ∆	t Τ	  consisting of both static and dynamic
	feature vectors.
	Using parallel training data set consisting of time-aligned input and output parameter
	vectors 1 1

Table 1 : Voicing error using neural network or GMM.

 1

	Type of error	Feed-fwd NN (%)	GMM (%)
	Voiced error	2.4	3.3
	Unvoiced error	4.4	5.9
	Total	~ 6.8	~ 9.2

Table 2 . F 0 difference (%) between converted and target speech.

 2

		window size		Number of Gaussian mixtures
	method	(frame interval)
			8	16	32	64
		1	10.96	10.90	10.92	10.90
		2	10.77	10.41	10.29	10.44
	PCA	3	10.33	9.98	10.08	10.28
		4	9.90	9.58	9.47	9.82
		5	9.44	9.17	9.32	9.31
		1	10.85	10.58	10.56	10.64
		2	10.36	10.23	10.11	10.36
	LDA	3	9.98	9.94	9.93	10.29
		4	9.45	9.43	9.62	9.67
		5	9.15	9.22	9.25	9.37

Table 3 . Cepstral distortion (dB) between converted and target speech.

 3

	Number of Gaussian mixtures

Table 4 . Influence of visual information on voicing decision.

 4

			Feed-fwd NN (%)		GMM (%)
	Type of error	AU		AUVI
		w = 1 w = 0.75 w = 0.5 w = 0.25
	Voiced error	3.71 3.58	3.74	3.34	2.75	4.29
	Unvoiced error 4.34 4.19	4.71	4.37	4.87	5.47
	Total	8.05 7.77	8.45	7.71	7.62	9.76

  if accurate phoneme detection was too difficult. Therefore we also grouped the consonants into different place of articulation categories to examine which consonantal groups benefited most from the addition of visual information. The 27 consonants were thus grouped into bilabials (/p/, /pj/, /b/,

	/bj/, /m/, /mj/), alveolars (/d/, /t/, /s/, /ts/, /z/, /j/, /n/, /nj/), palatals (/k/, /kj/, /g/, /gj/), non-alveolar
	fricatives (/f/, / /, /t /, / /) and others (/h/, /hj/, /r/, /rj/, /w/). The first aim was to check whether
	consonants belonging to the bilabial category, which are intrinsically more salient visually, were
	better identified as bilabial (be it /p/, /pj/, /b/, /bj/, /m/ or /mj/), when visual information was taken
	into account in the speech conversion procedure and when audiovisual stimuli were presented. The
	second aim was to examine what kind of perceptual confusions were observed.
				1.5 1.75	2	Video-
						only
	Cepstral	10		7.01 5.97 5.80 5.78 5.88 5.87 5.86 5.84
	distortion	20	5.99	7.01 5.97 5.79 5.77 5.86 5.83 5.95 5.89	9.28
	(dB)	40		7.02 5.96 5.79 5.77 5.86 5.82 5.94 5.88
	F 0	10		14.55 12.44 11.42 10.99 11.71 12.75 14.24 15.77
	estimation	20	11.56	14.79 12.38 11.40 10.78 11.66 12.62 14.31 15.43	12.95
	(%).	40		14.57 12.39 11.04 10.76 11.66 12.62 14.33 15.46
	AP	10		41.46 38.26 37.73 37.57 37.34 37.95 37.57 37.95
	distortion	20	38.72	41.42 38.25 37.79 37.55 37.33 38 37.90 37.95	51.45
	(dB)	40		41.42 38.21 37.78 37.53 37.33 37.99 37.53 37.93

Table 6 . Confusion matrices for the 5 places of articulation in the 4 conditions

 6 In order to compare the performance of the GMM-based voice conversion technique with the approach combining NAM recognition and speech synthesis(Hueber, Chollet et al. 2007; Hueber, Chollet et al. 2007), an HMM-based whisper-to-speech conversion system was developed. It combines HMM recognition and synthesis: instead of a corpus-based synthesis, we in fact use HMM synthesis (Tokuda, Yoshimura et al. 2000). The voice conversion is performed in three steps: 1. Using aligned training utterances, the joint probability densities of source and target parameters are modelled by context-dependent phone-sized HMM. Because of limited training data, we only used the right context, where subsequent phonemes are classified

			bilabial alveolar palatal fricative other
	A from A	bilabial	18.29	10.86	22.86	34.86	13.14
		alveolar	2.38	59.52	11.90	8.73	17.46
		palatal	3.86	15.44	54.83	12.74	13.13
		fricative	0.95	25.71	21.90	47.62	3.81
		other	5.00	20.71	25.71	9.29	39.29
	AV from A	bilabial	15.00	10.50	28.00	36.50 10.00
		alveolar	3.47	65.97	9.03	9.72 11.81
		palatal	2.36	16.89	65.20	9.12	6.42
		fricative	0.83	21.67	27.50	49.17	0.83
		other	7.50	19.38	27.50	10.63 35.00
	A from AV	bilabial	32.57	5.14	20.57	30.29 11.43
		alveolar	0.79	63.49	15.87	6.35 13.49
		palatal	6.56	13.51	57.14	12.74 10.04
		fricative	3.81	21.90	24.76	47.62	1.90
		other	5.00	20.00	21.43	6.43 47.14
	AV from AV bilabial	75.00	1.00	9.50	10.50	4.00
		alveolar	0.69	70.83	11.81	2.78 13.89
		palatal	1.01	17.91	62.84	10.14	8.11
		fricative	0.83	23.33	26.67	47.50	1.67
		other	4.38	16.88	25.00	6.88 46.88

coarsely into 3 groups for vowels ({/a/},{/i/,/e/},{/u/,/o/} without distinguishing between long and short vowels), 2 groups for consonants and 2 groups for utterance-final andinternal silences. Joint observations of each HMM state are modelled with one Gaussian with a diagonal covariance matrix.

2. HMM recognition is performed using the source stream with the HTK toolkit

[START_REF] Young | The HTK Book[END_REF]

). The linguistic model is limited to phone bi-grams learnt on the training corpus. 3. HMM synthesis of the recognized context-dependent phone sequence and target stream is performed using the HTS software (Tokuda, Yoshimura et al. 2000; Zen, Nose et al. 2007).

Acknowledgements

The authors are grateful to C. Vilain, C. Savariaux, A. Arnal, K. Nakamura for data acquisition, to Prof. Hideki Kawahara of Wakayama University in Japan for the permission to use the STRAIGHT analysis-synthesis system. We also acknowledge the subjects of the perceptual tests who kindly accepted to identify several hundreds of audio and audiovisual stimuli.

Spectral and excitation estimations

The visual information also enhances the performance of the conversion system. As shown in Table 5, the best results are obtained with weighting equally acoustic and visual parameters (w = 1) and with the dimension of the visual vector equal to 40. The spectral distortion between the converted speech and the target speech is decreased by 3.7% (5.99 dB → 5.77 dB) while the difference between the converted speech and that of target speech is decreased by 6.9 % (11.56 % → 10.76 %) for F 0 estimation and 3.6 % for aperiodic components estimation. Figure 9 shows an example of F 0 curves converted from audio and audiovisual input whisper. With visual information as an additional input, we have a better converted F 0 .

Facial animation

The audiovisual rendering of estimated visual parameters is performed by texturing the mesh piloted by the shape model introduced in section 5.2. An appearance model that computes a texture at each time frame is built using a technique similar to AAM [START_REF] Cootes | Active Appearance Models[END_REF]) in three steps: (1) shape-free (SF) images are obtained by morphing key images to a reference mesh; [START_REF] Bailly | Speaking with smile or disgust: data and models[END_REF] contrary to AAM, these pooled SF images are then directly linked to articulatory parameters via a multilinear regression; (3) the resulting appearance model is then used to compute a SF synthetic image given the set of articulatory parameter of each frame and used to texture the corresponding

ACCEPTED MANUSCRIPT

shape. The main difference with AAM is the direct multilinear regression used instead of a joint PCA and the number of configurations used: while AAM typically uses a few dozen images on which a generic mesh is adapted by hand or semi-automatically, we use here more than a thousand configurations on which the mesh is positioned automatically thanks to marked fleshpoints [START_REF] Bailly | Speaking with smile or disgust: data and models[END_REF]). The videorealistic audiovisual rendering of computed facial movements is illustrated in Figure 10.

Subjective evaluation

Eight Japanese listeners participated in our perceptual tests on audiovisual converted speech. The stimuli consisted of Japanese VCV (with V chosen amongst five vowels and C amongst twentyseven consonants) sequences with four conditions:

1. Speech generated from whispered audio (named 'condition A from A') 2. Speech generated from whispered audio and video ('condition A from AV') 3. Speech and facial animation generated from whispered audio ('condition AV from A') 4. Speech and facial animation generated from whispered audio and video ('condition AV from AV')

The five vowels were /a/, /i/, /e/, /o/, /u/. The 27 consonants were the following: /p/, /pj/, /b/, /bj/, /m/, /mj/, /d/, /t/, /s/, /ts/, /z/, /j/, /n/, /nj/, /k/, /kj/, /g/, /gj/, /f/, / /, /t /, / /, /h/, /hj/, /r/, /rj/, /w/.

Only 115 combinations of vowels and consonants were tested.

ACCEPTED MANUSCRIPT

Each participant heard and viewed a list of randomized synthetic audio and audiovisual VCV. For each VCV, she/he was asked to select what consonant she/he heard among a list of 27 possible Japanese consonants. Figure 11 provides the mean recognition scores for all the participants. Visual parameters significantly improve consonant recognition: the identification ratios for 'AV from A' (28.37 %), 'A from AV' (30.56 %) and 'AV from AV' (36.21 %) are all significantly higher than that of the 'A from A' condition (23.84 %) (F = 1.23, p < 0.303). The figure also shows that providing visual information to the speakers is more beneficial when it is synthesized from audiovisual data ('AV from AV') than when it is derived from audio data alone ('AV from A').

Furthermore the addition of visual information in the input data ('A from AV') increases identification scores compared with audio data alone ('A from A'), but is not significantly different from providing audiovisual information derived from the audio alone ('AV from A'). Although adding visual information greatly increases the identification scores on all the tested VCVs, the scores are still quite low (less than 40%). It should be noted however that nonsense VCV recognition is a difficult task, especially in a language with a lot of consonants. It could therefore be argued that identification scores on words or sentences could be much higher, with the help of lexical, syntactic and contextual information. With this in mind, it could be interesting to check whether the addition of visual information provided in fact some cues to place of articulation, even