Alexis Thual

Corentin Dancette

Julien Karadayi

Juan Benjumea

Emmanuel Dupoux

A K-NEAREST NEIGHBOURS APPROACH TO UNSUPERVISED SPOKEN TERM DISCOVERY

Keywords: spoken term discovery, word discovery, unsupervised, word segmentation

Unsupervised spoken term discovery is the task of finding recurrent acoustic patterns in speech without any annotations. Current approaches consists of two steps: (1) discovering similar patterns in speech, and (2) partitioning those pairs of acoustic tokens using graph clustering methods. We propose a new approach for the first step. Previous systems used various approximation algorithms to make the search tractable on large amounts of data. Our approach is based on an optimized k-nearest neighbours (KNN) search coupled with a fixed word embedding algorithm. The results show that the KNN algorithm is robust across languages, consistently outperforms the DTW-based baseline, and is competitive with current state-of-the-art spoken term discovery systems.

INTRODUCTION

Current automatic speech recognition pipelines require considerable textual resources to train acoustic and language models. Several approaches have been pursued to address the challenge of building useful technology for languages where few textual resources are available. The "zero resource" setting [START_REF] Glass | Towards unsupervised speech processing[END_REF] tackles the most extreme case, where no textual resource is available at all, and aims at extracting useful linguistic units from raw speech in an unsupervised way. Spoken term discovery focuses on discovering word-sized units; it can be used for document classification [START_REF] Dredze | Nlp on spoken documents without asr[END_REF], segmentation [START_REF] Zheng | Acoustic texttiling for story segmentation of spoken documents[END_REF] or retrieval [START_REF] White | Using zero-resource spoken term discovery for ranked retrieval[END_REF][START_REF] Lee | Spoken content retrievalbeyond cascading speech recognition with text retrieval[END_REF], and idiolect recovery [START_REF] Jansen | Unsupervised idiolect discovery for speaker recognition[END_REF]. It can also help other zero resource objectives, such as learning phonetic unit or representations [START_REF] Jansen | Towards unsupervised training of speaker independent acoustic models[END_REF][START_REF] Kamper | Unsupervised neural network based feature extraction using weak top-down constraints[END_REF][START_REF] Synnaeve | Phonetics embedding learning with side information[END_REF].

Searching for all matching motifs in a speech corpus scales quadratically and can be impractical for large corpora. Here, we separate spoken term discovery into two subproblems: representing candidate motifs as a fixed-sized vector, and similarity-based search. For the latter, we use a high performance k-nearest neighbours library developed by Facebook [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF], which can scale to billion of search terms. We test our systems on datasets from the Zero Ressource Challenges [START_REF] Versteegh | The zero resource speech challenge 2015[END_REF][START_REF] Dunbar | The zero resource speech challenge 2017[END_REF], and compare it to the state-of-the-art.

RELATED WORK

Spoken term discovery systems can be sorted in two classes. The first class attempts to discover recurring motifs in the speech signal by using a DTW-based distance metric over motifs, and clustering these motifs to form a lexicon [START_REF] Park | Unsupervised Pattern Discovery in Speech[END_REF][START_REF] Muscariello | Audio keyword extraction by unsupervised word discovery[END_REF][START_REF] Muscariello | Unsupervised Motif Acquisition in Speech via Seeded Discovery and Template Matching Combination[END_REF][START_REF] Jansen | Efficient spoken term discovery using randomized algorithms[END_REF][START_REF] Lyzinski | An evaluation of graph clustering methods for unsupervised term discovery[END_REF][START_REF] Oosterveld | A parallelized dynamic programming approach to zero resource spoken term discovery[END_REF]. The focus of these algorithms is not to discover all of the words, but to discover a subset of good candidates. This is why they typically set a lower bound on the duration of the motifs, in order to avoid short words, which tend to have many neighbors and be hypo-articulated. The second class attempts to exhaustively parse the input signal into words, thereby leveraging the additional constraints that word candidates cannot occupy the same portion of speech. These algorithms both segment the speech and construct a lexicon at the same time [START_REF] Kamper | Fully unsupervised small-vocabulary speech recognition using a segmental bayesian model[END_REF][START_REF] Kamper | A segmental framework for fully-unsupervised largevocabulary speech recognition[END_REF]. They are variants of text-based algorithms based on the idea of minimizing the length (or maximizing the probability) of the corpus and its accompanying lexicon [START_REF] Michael | An efficient, probabilistically sound algorithm for segmentation and word discovery[END_REF][START_REF] Goldwater | Nonparametric Bayesian models of lexical acquisition[END_REF][START_REF] Goldwater | A Bayesian framework for word segmentation: Exploring the effects of context[END_REF].

In the present work, we aim at adding a new element to the first class, but simplify the pipeline by getting rid of the DTW step, which requires to compute a very large matrix of framewise similarity of the corpus by itself. The system proposed by Jansen and van Durme [START_REF] Jansen | Efficient spoken term discovery using randomized algorithms[END_REF], which was used as baseline of the Zerospeech Challenge 2015 [START_REF] Versteegh | The zero resource speech challenge 2015[END_REF], proposes to avoid computing the whole matrix by binarizing it using random projections, an indexation technique yielding a large gain in speed, but a decrement in performance due to the approximations performed by the algorithm [START_REF] Jansen | Efficient spoken term discovery using randomized algorithms[END_REF]. Here, we propose to use another indexation tool, based on product quantization [START_REF] Norouzi | Cartesian kmeans[END_REF], and specifically the FAISS library which can perform efficient exact or approximate search on CPUs or GPUs [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF].

DESCRIPTION OF THE PIPELINE

Our pipeline is presented in Figure 1. After extracting speech features, we segment the input speech and construct a fixedlength representation of each segmented term. We then use a KNN algorithm to systematically find a set of matching pairs, which we narrow down by a variant of Non Maximal Suppression to avoid overlap. This is followed by an optional step of graph clustering, before the output is evaluated.

To appear in 2018 IEEE Spoken Language Technology (SLT) Fig. 1. Steps in our KNN pipeline

Features extraction

Because we compare our system with Jansen van Durne's DTW system, we use the same input features, Perceptual Linear Prediction (PLP) features [START_REF] Hermansky | Perceptual linear predictive (PLP) analysis of speech[END_REF]. PLP features are designed to compress the input signal in a way that is inspired by human hearing. Thirteen PLP features are computed every 10 ms based on an analysis window of 25ms, and concatenated with first and second order temporal derivatives.

Term pre-segmentation

Our pipeline extracts every possible segmentation of the input speech into motifs and constructs a fixed length embedding for it. We restrict these motifs to span over the parts of the input data that only contain speech, using the voice activity detection (VAD) temporal alignment in the dataset. In order to avoid unnecessary computations with redundant motifs that only differ in a couple of frames, we only generate a possible segmentation point every a frame, with a > 1. Experiments showed that a = 4 is a good compromise and corresponds approximately to 2 segmentation points per phoneme.

When it comes to discovering potential words, too short motifs are not useful because difficult to cluster, and too long motifs tend to occur only once in a corpus. Therefore, we only consider terms for which the length is between l min and l max . This yields a total number of presegmented motifs T :

T = 1 a (l max -l min ) × N a
Where N is the total number of speech frames in the corpus. For instance, for length bounds between 20 and 50 frames with a step of 4 frames, the number of terms is only 1.8 times the number of frames of the original speech file. In this paper, we test several length bounds.

Fixed-length term embedding

Each term is encoded as a fixed-length vector in order to enable KNN search. Previous work has devised various frame-works for deriving such embeddings [START_REF] Holzenberger | Learning word embeddings: unsupervised methods for fixed-size representations of variable-length speech segments[END_REF].

Here, we use a simple Gaussian smoothed down-sampling method, typically used in [START_REF] Kamper | A segmental framework for fully-unsupervised largevocabulary speech recognition[END_REF] and studied in [START_REF] Holzenberger | Learning word embeddings: unsupervised methods for fixed-size representations of variable-length speech segments[END_REF]. The idea is to represent each segmented term using only d frames: in order to do so, one averages adjacent frames using a gaussian kernel. Let f i be the i-th frame of the output, then

f i = N i • n f d ; r • n f + s • g d (i) • t where g d : i → d 2 -| d 2 -i|, t
is the feature representation of the term being down-sampled, n f is the number of frames of t, r and s two parameters which we here set to 0.07 and 0.1 respectively, as recommended in [START_REF] Holzenberger | Learning word embeddings: unsupervised methods for fixed-size representations of variable-length speech segments[END_REF]. We set the value of d to 20 for all experiments reported in this paper. We had tested several values of d (from 10 to 30), r (from 0 to 1) and s (from 0 to 1), but none of these changes had strong positive or negative impact on our results.

KNN search

The pre-segmented embedded terms are stored in a large index. Then, for each term, we search the entire index for it's k nearest neighbours using FAISS [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF], where the similarity is based on the dot product of the embedded terms. In our pipeline, we use an exact search method, parallelized on 10 CPUs, with k set to 200. Searching the index against itself takes 98% of the overall run-time.

We tested several values for k, ranging from 50 to 500. The idea is using a reasonably large value for k (instead of k = 1) is to take into account self-overlapping pairs which may also tend to be very self-similar. In addition, if a good match is found between two different parts of the corpus, several overlapping matches will also have a good similarity match. Therefore our strategy was to allow a large number of matches to be returned, and prune this number down in a selection step. Experiments showed that all values above 100 yield the same results for a dataset of the size of a few hours. We set the value of k to 200 in order to handle larger datasets while keeping the run-time reasonable.

Selection of top pairs

Generating T × k pairs, where T is the number of previously segmented terms, obviously yields too many pairs, most of which are of poor quality. In addition, as we said above, many pairs are redundant and partially overlapping. In order to reduce the number of pairs, we perform the following 4 steps for each of the input speech files: 2. Thresholding: we keep only the top δ % of the pairs for a given file; in this paper, δ is the main control parameter and manipulated systematically (typically between 0.5% and 3%).

3. Self-overlapping pairs removal: given a pair p = (t 1 , t 2 ), we discard p if t 1 and t 2 overlap more than a given threshold τ self where τ self equals 50% in our experiments. This selection phase is very efficient and discards a large proportion of the pairs (around 90%).

4. Non Maximal Suppression (NMS): to remove the remaining overlapping pairs, we perform a variant of NMS by sequentially inspecting the pairs starting from the most similar ones. We only keep a new pair if it is not overlapping with an already selected pair. We use a threshold τ other of 50%. NMS helps discarding around 50% of the remaining pairs. Practically, we distribute the selected pairs into slots, one for each by pair of utterances in the original corpus (defined as contiguous stretches of speech) and perform NMS within each slot only. This allows for faster computations in our implementation.

In this paper, we define temporal overlap between two pairs of terms p = (t 1 , t 2 ) and p = (t 1 , t 2 ) as follows:

o(p, p ) = l(t 1 ∩ t 1 ) • l(t 2 ∩ t 2 ) l(t 1 ) • l(t 2 )
where l yields the length of a given term and t ∩ t designates the overlapping part of terms t and t . It is important to notice that o(p, p ) is not necessarily equal to o(p , p) and that the order in which pairs are seen during the NMS actually has an impact. Self-overlapping of a given pair p = (t 1 , t 2 ),

with t 1 = [a 1 , b 1 ] and t 2 = [a 2 , b 2 ]
, is computed using the following formula:

so(p) = min(b 1 , b 2 ) -max(a 1 , a 2 ) min l(t 1 ), l(t 2 )

Clustering

Once pairs of word-units have been selected, one can perform a clustering step in order to group those pairs together. The goal is that each cluster would account for one word-unit of the discovered lexicon, and that each member of a given cluster represents one occurrence of the aforementioned wordunit. Several clustering methods were documented [START_REF] Lyzinski | An evaluation of graph clustering methods for unsupervised term discovery[END_REF]. We did not try to use them here and rather focused on the quality of the output pairs of our pipeline. This can be interpreted as if each pair was a cluster on it's own.

EXPERIMENT I

The aim of this Experiment is to test systematically the effect of two important parameters in the KNN pipeline described above: (l min , l max ) the range of durations selected in the presegmentation step, and δ, the percentage of pairs that are kept after the KNN search. The range of duration is important because too long terms can lead to useless computations, while too short terms could give rise to very noisy results. To choose the range of durations to study, we had a peek of the distribution of word durations across the languages of the Zero Resource Challenges, and chose the following partition of durations: 100-200ms, 200-300ms, 300-500ms, 500-1000ms, each of which approximately accounts for 25% of the word tokens across these languages. We also used 200-500ms, which covers the middle range, and about 50% of the word tokens. The δ threshold commands a tradeoff between the number of pairs and the quality of these pairs, and we vary this threshold systematically to explore the whole range of this tradeoff. This is conducted in the Mandarin dataset of the Zero Ressource Speech Challenge 2017 [START_REF] Dunbar | The zero resource speech challenge 2017[END_REF], which we considered to be our development dataset to setup and test our KNN pipeline. Previously mentioned parameters remained fixed to the values described in the pipeline description.

Dataset

The dataset, available in the Zero Resource Speech Challenge 2017, includes 12 Mandarin speakers, each of whom speaks for a duration from 10 to 25 minutes. The overall duration of this dataset is 2 hours and 30 minutes. It has been forced aligned to enable phoneme-based evaluation metrics.

Evaluation

To study the effect of duration and δ, we only focus on the subset of the Zero Resource Speech Challenge 2015 metric devoted to measuring the quality of the similarity-based search, namely:

• Normalized Levenshtein Distance (NED): given a pair p = (t 1 , t 2 ), one can read the phonetic transcriptions of t 1 and t 2 using the gold temporal alignments; the normalized Levenshtein distance between these two transcriptions yields the NED for this pair

• Coverage (COV): it indicates the proportion of input speech which was discovered; for instance, a coverage of 0.5 indicates that half the input phones were involved in computed pairs of terms

• Number of pairs: simply the number of pairs of terms yielded by the pipeline

Results and discussion

Results are reported in Figure 2. They confirm the expected effects of duration range on the tradeoff between number and quality of the pairs. Articulation for short words is on average worse than it is for longer terms. This is indeed what we observe here: for a small COV or number of pairs, longer terms yield much better results that short ones. However, for larger COV and number of pairs, short terms yield a much better NED than long ones. This is because the frequency of a long term is usually low, thus, when a high number of pairs is required (by setting δ to a permissive value for instance), our structure pairs long words with other long words and the chances of these being different occurrences of the same word get lower. On the contrary, small words' frequency is much higher (words lasting between 100ms and 200ms are almost syllables) and it is still plausible to find theoretically good pairs even when δ is very permissive. As expected, results for terms of length ranging between 200ms and 500ms approximately behave like terms of length 300-500ms for low COV and number of pairs, and roughly follow terms of length 200-300ms for high COV and number of pairs. Wider ranges allows for better results but only up to a point. A range between 200ms to 1000ms, for instance, (not shown in the Figure ) yields results which are almost comparable to 200-500ms for all COV values, with much longer computation time. This is why in the following, we stick to the 200-500ms range.

EXPERIMENT II

The aim of this experiment is to compare our KNN pipeline to the Zero Ressource 2015 DTW-based baseline. To do this, we picked our best system as evaluated in the Mandarin dataset in Experiment 1 and applied it without changing any parameter to the two datasets of the ZR15 Challenge: English and Xitsonga.

We first run the same evaluation as in Experiment 1 on the duration range of 200-500ms, systematically varying the threshold parameters of both algorithms in order to compare their quality/quantity tradeoffs. Next, we pick a value for the threshold parameter δ of the KNN system so that the NED is comparable to that of the DTW-baseline and points with higher NED. We then evaluate the results using the standard metrics of the challenge and compare them to state-of-the-art implementations.

Datasets

The English dataset comes from the Buckeye corpus and includes 12 speakers speaking between 16 and 30 minutes each, for a total duration of 5 hours. The Xitsonga corpus includes 24 speakers speaking between 2 and 30 minutes each, for a total duration of 2 hours and 30 minutes.

Evaluation

We use the full set of evaluation metrics of the second Track of the Zero Resource Challenge 2017. This includes Ned and Cov (already described above), and a set of 4 precision/recall/F-score metrics each of them dedicated to a particular aspect of spoken term discovery: Grouping, Type, Token and Boundary. The Grouping precision and recall metrics are similar to cluster purity and collocation scores, respectively, except they are computed on a pair by pair basis and do not require a majority decision (each pair is labelled as belonging to the 'same cluster' or 'different cluster' set). The Type metrics compare the discovered clusters to the gold lexicon (also pair-wise). The token metrics compare the set of discovered terms with the set of gold tokens, and the boundary metrics compare the discovered and gold boundaries. The Fscores are the harmonic means of precision and recall. 2015. We also show the results on Mandarin for comparison purposes. The results show that for every language, the KNN system yields better/more numerous pairs than the DTW system. In English, for a NED of 25% for instance, the KNN system finds between 2 and 2.5 more pairs in English and Xitsonga in the COV metrics. The difference is ever larger in the number of pairs metrics (between 20 and 100). This is all the more remarkable that the KNN system was not fine tuned for these two languages, whereas the DTW system was. For Mandarin, the difference between the two systems even larger, mainly due to very poor performance of the DTW system in this language, suggesting an over-fitting of the parameters for the languages of the Zero Resource Challenge 2015. The interesting discrepancy between COV and number of pairs across the two algorithms suggests that the KNN discover many more pairs per region of speech. This could be because it reports more overlapping pairs (redundant pairs), or because it manages to find more of the pairs related to the same word (less fragmented clusters). Regarding the performance of the KNN algorithm across the three languages, we see similar performances up to 20% coverage, and a divergence thereafter, mandarin giving worst results, followed by English and Xitsonga. Despite these differences, the performance is much more stable than for the DTW algorithm.

Results and discussion

Finally, we selected the value of the δ threshold such that the NED value for the KNN algorithm would cover the range of NED found in the the DTW baseline system. The values tested were 15%, 20% and 30%. We then analyzed the results with the suite of metrics of the Challenge as shown in Table 1.

We found that for all three languages, the KNN algorithm was able to beat the DTW baseline on the grouping precision (corresponding to cluster purity), but not on recall (which corresponds to the inverse of cluster fragmentation). This is not surprising because we did not apply a clustering algorithm and only took the raw pairs from the output of the KNN pipeline. Interestingly, the segmentation and lexical F-scores were better for the KNN algorithm, which is probably due to its larger coverage.

The KNN algorithm was competitive with the state-ofthe-art DTW system [START_REF] Oosterveld | A parallelized dynamic programming approach to zero resource spoken term discovery[END_REF], which performs exact DTW using a highly optimized DTW search over a GPU. The GPU-DTW algorithm presented was set on a different NED/COV tradeoff than our KNN algorithm, resulting in better grouping and boundary performance. Interestingly, our KNN still beast the DTW on type and token F-scores.

The final comparison is with exhaustive algorithms. We compare our KNN algorithm with the Bayesian segmental algorithm of [START_REF] Kamper | A segmental framework for fully-unsupervised largevocabulary speech recognition[END_REF] and the Kmeans version of the algorithm [START_REF] Kamper | An embedded segmental k-means model for unsupervised segmentation and clustering of speech[END_REF]. Unsurprisingly, these exhaustive algorithms outperform KNN on all segmentation and lexical metrics (except for the Type and Token F-Scores in Mandarin). However, this high coverage and good segmentation comes at a cost in clustering purity (very high NED and poor grouping precision). In light of these tradeoffs, further work would be needed to compare these different algorithms back to back.

GENERAL DISCUSSION

We present a new pipeline for spoken term discovery which bypasses the need of performing DTW on the speech data, and rather uses a performant KNN library on fixed-words embeddings. The results, optimized on Mandarin, transfer to two new languages, and beat a DTW-based baseline that had been optimized for these two languages.

The datasets that were used here were small enough that we could used the exact search mode of the KNN library. Using 10 CPUs, our searches run approximatively in real ime (2 to 3 hours for Mandarin -5 hours for the Buckeye) and necessitate the storage of an index of 1 to 2 GB depending on the dataset's size. For larger datasets, we would have to resort to the approximate mode and/or the parallelized GPU mode 1. Zero Resource Track 2 evaluation results for the Topline (gold phonemes plus Adaptor Grammar), the DTW Baseline system, O: parallelized DTW system [START_REF] Oosterveld | A parallelized dynamic programming approach to zero resource spoken term discovery[END_REF], ES-KM: Embedded Segmental Kmeans [START_REF] Kamper | An embedded segmental k-means model for unsupervised segmentation and clustering of speech[END_REF], BES-GMM: BayesSegMinDur's model of [START_REF] Kamper | A segmental framework for fully-unsupervised largevocabulary speech recognition[END_REF], and KNN (our system) with 15%, 20%, or 30% of NED.

of the KNN library. What's important, though, is that because we have separated the problem of representing the speech motifs from the problem of search, the scaling up problems can be addressed by generic systems without having to make particular tweaks about speech.

The pre-segmentation and representation steps are of course critical for the quality and efficiency of the algorithm. Further work would need to explore other pre-segmentation ideas, in particular the idea of using syllabic boundaries as in Rasanen et al. [START_REF] Okko Rasanen | Unsupervised word discovery from speech using automatic segmentation into syllable-like units[END_REF]. If this turns out to be reliable, we could expect a speed up factor of about 10-20. As for the representation, down-sampling is not the most efficient way to represent spoken terms. Other work has shown that recurrent NNs can achieve even better results, on some occasions overtaking DTW methods [START_REF] Holzenberger | Learning word embeddings: unsupervised methods for fixed-size representations of variable-length speech segments[END_REF] and would need to be tested in this framework.

Fig. 2 .

 2 Fig. 2. Results comparison using various lengths for segmented terms on mandarin using PLPs (labels account for length' range in ms)

Figure 3

 3 Figure3shows the NED / COV and NED / Number of pairs graphics for the DTW baseline system and the KNN system on the two test languages of the Zero Ressource Challenge

Fig. 3 .

 3 Fig. 3. Results comparison using various lengths for segmented terms on mandarin using PLPs (labels account for length' range in frames)

  42.5 2.4 12.8 80.5 22.1

							Grouping			Type			Token			Boundary
		Clusters	Pairs	NED Cov	Pr	Re	F	Pr	Re	F	Pr	Re	F	Pr	Re	F
							Mandarin
	Topline	1240	1 742 931	0.0	100 100 100 100 29.3 29.3 29.3 28.1 46.1 34.9 66.2 100 79.7
	Baseline	156	160	30.7	2.9 30.2 96.7 44.7 4.5	0.1	0.2	4.0	0.1	0.1 37.5 0.9	1.8
	ES-KM			88.1 100	/	/	/	2.5	4.1	3.1	2.5	3.4	2.9	36. 47.1 41.1
	KNN-20	25 488	25 488	20.9 17.8 64.8 17.6 27.7 6.5	3.0	4.1	2.7	3.5	3.0 16.1 13.1 14.4
	KNN-30	78 532	78 532	29.8 36.8 50.9 12.5 20.1 6.4	6.4	6.4	2.8	8.6	4.2 16.3 28.8 20.9
							English -Buckeye
	Topline	/	/	0.0	100 99.5 100 99.7 50.3 56.2 53.1 68.2 60.8 64.3 88.4 86.7 87.5
	Baseline	3149	4305	21.9 16.3 21.4 84.6 33.3 6.2	1.9	2.9	5.5	0.4	0.8 44.1 4.7	8.6
	O			39.4 92.1 76.2 100 82.7 5.6	5.1	5.3 10.2 1.9	3.2 71.1 22.5 34.3
	BES-GMM			56.0 100 22.7 29.6 25.5 14.0 28.6 18.8 26.6 2.5 17.0 80.7 50.4 62.0
	ES-KM			71.6 100	/	/	/	/	/	18.9	/	/	18.1	/	/	62.2
	KNN-20	464 491	464 491	20.7 32.7 41.5 15.0 22.1 5.5 17.4 8.4	2.6	5.9	3.6 25.4 22.3 23.7
	KNN-30	1 319 411 1 319 411 31.4 58.9 27.0 13.6 18.1 4.1 29.7 7.3	2.2 10.2 3.6 25.4 40.0 31.1
							Xitsonga
	Topline	/	/	0.0	100 100 100 100 15.1 18.1 16.5 34.1 49.7 40.4 66.6 91.9 77.2
	Baseline	1782	1818	12.0 16.2 52.1 77.4 62.2 3.2	1.4	2.0	2.6	0.5	0.8 22.3 5.6	8.9
	O			39.6 95.5 19.1 100 31.7 1.6	2.2	1.9	1.5	0.5	0.8 49.9 27.6 35.5
	BES-GMM ES-KM			58.6 100 80.3 100	8.3 10.3 9.2 / / /	3.8 ¯9.8 / /	5.5 4.9	4.3 /	4.0 /	4.1 44.5 42.0 43.2 3.7 / / 42.1
	KNN-15	61 818	61 818	15.2 21.2 66.3 5.9 10.8 3.6	3.8	3.7	1.2	4.1	1.9 13.0 16.7 14.6
	KNN-20	351 899	351 899	20.8 54.8 53.0 2.9	5.5	3.3	8.3	4.7	1.3 16.8 2.4 13.0 47.6 20.4
	KNN-30 1.2 Table 1 852 520 1 852 520 30.4 84.8 36.9 1.7 3.3 4.1 17.2 6.7

Sorting: we recover the totality of the KNNs for each segmented term an input file, and sort the resulting pairs of terms by decreasing similarity.

ACKNOWLEDGMENTS

This work was funded by the European Research Council (ERC-2011-AdG-295810 BOOTPHON), the Agence Nationale pour la Recherche (ANR-10-LABX-0087 IEC, ANR-10-IDEX-0001-02 PSL* ), Almerys (industrial chair Data Science and Security), Microsoft Research (joint MSR-INRIA center), Amazon Web Service (AWS Research Credits) and a Google Faculty Award.


Zhi Tao

Tayeb Mohammed-Brahim

Wei Lei

Maxime Harnois

Emmanuel Jacques

Impact of the post-thermal annealing on OFETs using printed contacts, printed organic gate insulator and evaporated C 60 active layer

Keywords: inkjet printing drop-on-demand, Organic FET, physical characterization, electrical characterization, post-thermal annealing, time answer

Towards fully printed organic electronics, bottom-gate, bottom contacts fullerene based organic FETs (OFETs) are fabricated using inkjet printing drop-on-demand technology for the deposition of gate contact, organic gate insulator and source and drain contacts. The last layer of this structure that is the semiconducting fullerene C 60 film, is deposited by thermal evaporation at ambient temperature. Using physical and electrical analysis, the electrical performances are optimized by studying the effect of the thickness of the C 60 film and of the post-thermal annealing of the OFET. A quantitative relationship between the crystalline structure of the C 60 semiconducting layer (grain size and surface roughness) and the electrical performance of the OFETs is demonstrated. The best performance as determined from the values of the electrical parameters of the OFETs are obtained for 90 nm thick C 60 film and after an annealing of the OFET at 200°C. The answer of present nearly fully printed OFETs to a voltage pulse applied to the gate lead to a switching time of 13 ms, and then to application frequency of some 10Hz. This result opens the way to potential main application in electronic circuits for the treatment of analog signals coming from the human body activities.

combined with photolithography processing, the solution process such as printing leads to thicker deposited films. Then, in a bottom-gate, bottom contacts structure of the OFETs, the high aspect ratio between the drain and source contacts and the gate insulator imposes the use of thick semiconducting active layer. Present work will highlight the influence of the thickness of the C 60 active layer on the performance of nearly fully printed C 60 based bottom gatebottom contacts OFETs.

The effect of the post-thermal annealing on the as-fabricated OFETs with different thickness of C 60 active layer will be studied also. In our knowledge, no such work, combining the studies of the thickness and the post-thermal annealing on C 60 OFETs, was previously done. It will lead to determine the optimum OFET's processing parameters.

Consequently, for the first time, a quantitative relationship between the crystalline structure of the C 60 active layer (grain size and surface roughness) and the electrical performance of the OFETs is demonstrated.

Experimental

Transistor fabrication and characterization

Fig. 1 shows a schematic cross-section of the C 60 OFET fabricated following bottom gate and bottom contacts geometry. This figure highlights the problematic of the high aspect ratio between the drain and source contacts and the gate insulator. The thickness of the active layer has to be enough to insure good coverage between the source and drain contacts.

. As illustrated in Fig. 2, after cleaning the glass substrate with acetone, ethanol, and deionized water for 10 mins sequentially, the silver gate electrode was printed on the substrate (keeped at 50°C) with a thickness of 200nm using 256 nozzles Q-class printhead (Dimatix © )(Fig2.b). Before spin coating the gate insulator layer, the substrate with silver gate electrodes is baked at 110°C for 30min allowing nanoparticules based ink sintering. Then, 1.2μm thick uncrosslinked SU8 2000 series photoresist (Bisphenol A Novolak epoxy; MicroChem) is spin-coated at a speed of 4500r/min during 60 seconds (Fig2.c). This layer will be used as polymeric gate insulator. After UV exposure and development (SU8 micro Dev, MicroChem), the SU8 layer was baked at 115°C during 15min.The baked SU8 layer is then submitted to 2mins UV-Ozone treatment to adjust SU8 wetting behavior [START_REF] Robin | Epoxy based ink as versatile material for inkjet-printed devices[END_REF][START_REF] Moon | Morphological impact of insulator on inkjet-printed transistor[END_REF][START_REF] Borgne | Conformal Electronics Wrapped Around Daily Life Objects Using an Original Method: Water Transfer Printing[END_REF] ready for the printing of source and drain contacts. Drain and source electrodes were printed at 50°C using different patterns (width of the electrodes W=4000µm, distance between the electrodes L=150µm, and the thickness of the electrodes D=205nm in Fig2.d). Finally, C 60 film is thermally evaporated as the active layer of the transistor at room temperature under vacuum (2x10 -7 mbar) with a constant deposition rate of 0.15A/s (Fig2.e). The C 60 active layer was deposited with four different thicknesses, 60 nm, 90 nm, 125nm and 250 nm in the purpose to study the effect of its thickness. The OFET is then characterized electrically, as deposited and after annealing at 3 different temperatures, 160°C, 200°C and 250°C, during 5 mins. Electrical characterization was performed in dark and under nitrogen atmosphere using a 2636A Keithley (Fig2.f).

Morphological characterization and topography parameters extraction of the C 60 active layer.

In the purpose to study the effect of the temperature during the final annealing of the OFET, the morphology of the C 60 active layer was analyzed using Atomic Force Microscopy (AFM). An Atomic Force Microscope (AFM VEECO Di Caliber) was utilized in tapping mode on an area of 1x1 µm 2 . To describe the topography of the surface, we use the Power Spectral Density Function (PSDF). Its formulation is given by [26]:

(1)

Here S is the 2-dimensional PSDF, L 2 is the scanned surface area, N is the number of data points in both X and Y direction of scanned area, h mn is the surface profile height at position (m,n), f x and f y are the spatial frequencies in X and Y directions respectively, ΔL is the sampling interval. Based on the AFM images (1µmx1µm), the RMS roughness (σ) and the lateral correlation length (τ) were calculated from these PSDF functions (support files). Besides, the lateral correlation length (τ) is related to the grain size.

Electrical parameter extraction

As the most important electrical parameters, the field effect mobility μ (cm 2 .V.s -1 ), the sub-threshold slope SS (V.decade -1 ), the threshold voltage V TH (V), the contact resistance R c (Ω), the minimum off-state current I off (A), the on/off current ratio I on /I off are utilized to evaluate the performance of the thin-film transistor.

In linear regime, µ, SS and V TH are calculated from the equation 2 of the drain current I D (V DS << (V GS -V TH )):

(2)

Where the W, L and C ins are the width, the length of the channel and the capacitance of the gate insulator per area unit, respectively. The slope of I D -V GS curve at constant V DS is the transconductance g m given by:

(3)

The threshold voltage V TH is the gate voltage axis intercept of I D -V GS curve plotted in linear part. The field effect mobility µ is deduced from the extraction of g m .

The subthreshold slope SS is extracted from I D -V GS curve plotted in semi-logarithmic scale using:

(4)
The contact resistance R c is calculated from the equation of the drain-source resistance R DS : [START_REF] Singh | Inkjet printing-process and its applications[END_REF] where R c is the contact resistance, assumed similar for both source and drain contacts.

Results and discussions

3.1 Effect of the semiconductor annealing temperature on the morphology of the C 60 thin film on the OFET.

Morphology of C 60 active layer as a function of the annealing temperature.

To study only the effect of the final annealing temperature of the OFET, on the morphology of its C 60 active layer, the thickness of this layer was fixed at 125nm. The morphology was studied as deposited and after annealing at 160°C, 200°C, 250°C. As illustrated in Fig. 3, the 2D, 3D and line scan AFM images of C 60 thin film as deposited and after annealing at 160°C, 200°C and 250°C exhibit that C 60 morphology depends on the annealing temperature. From the 2D AFM images, the surface roughness increases and the number of grains decreases in the same area when the annealing temperature increases. Quantitatively, the RMS roughness s and grain size t were calculated using the equation ( 1) and reported in Table 1. Morphological parameters (roughness and grain sizes values) continuously increase when annealing temperatures increase. Furthermore, the grain size increases more when such temperature is below 200°C than between 200°C and 250°C. All the OFET's parameters such as threshold voltage, subthreshold slope, field effect mobility, on-current, improve when the annealing temperature increases. The value of these parameters is reported in Table 2 and plotted as a function of the annealing temperature in Fig. 5a.

Table 2

Main mean electrical parameters of several OFETs having the same size (channel width W=4000 µm, channel length L=150 µm) and 125 nm thick C 60 active layer, measured as-fabricated and after annealing at T A =160°C, 200°C and 250°C. V TH is the threshold voltage, SS the subthreshold slope, µ the field effect mobility, R C the contact resistance, µ after R C the mobility after correction by R C , I off is the minimum current and I on /I off is the ratio between the maximum current and the minimum current Table 2 highlights that the main electrical parameters (i.e., threshold voltage, subthreshold slope, mobility) are strongly impacted by annealing temperature (especially bellow 200°C). It seems that these electrical parameters are governed by the grain size.

T A (°C) V TH (V) SS (V/dec) µ (cm 2 /V.s) R c () µ after R c (cm 2 /V.
The contact resistance decreases after the annealing at 160°C and then stabilizes. Its value stays much lower than the channel resistance and then the corrected mobility by removing the effect of the contact resistance, does not change drastically. This conclusion was expected due to the starting linear increase of the drain current as can be observed in the output characteristics. Moreover, the off-current increases when the annealing temperature increases. The increase seems to be due to the improvement of the active layer after annealing and then to better conductive layer. Considering these preliminary observations, it seems that the optimum annealing temperature is 200°C. The different parameters after this annealing are similar to the 250°C annealing ones. Moreover, the on-current to off-current ratio is higher that its value after 250°C annealing.

To understand the behavior of the main parameters depending on the annealing temperature, we can try to highlight a relationship between the AFM and the electrical characterizations. The mobility and the grain size present similar behavior as shown in Fig. 5b. Both, mobility and grain size strongly increase when temperature increases until 200°C and a little bit when the grain size increases between 200°C and 250°C. The subthreshold slope SS presents a specific behavior as it increases between 200°C and 250°C. Such behavior can be understood from the usual equation of SS [START_REF] Romain | The structure-property relationship study of electron-deficient dihydroindeno [2, 1-b] fluorene derivatives for n-type organic field effect transistors[END_REF] as a function of the defects density in the channel N ep and at the interface between the channel and the gate insulator N SS [START_REF] Rolland | Electrical properties of amorphous silicon transistors and MIS-devices: comparative study of top nitride and bottom nitride configurations[END_REF] (6)

C ins is the capacitance per area unit of the gate insulator and  sc is the relative dielectric constant of the semiconducting active layer.

On the one hand, if the C 60 active layer is considered as a polycrystalline material, N ep is the defect density concentrated mainly in the grain boundaries. The defect density at the interface N SS depends at least partially on the roughness of the interface between the channel and the active layer. Note that, the grain size was previously determined by AFM measurements. Meanwhile, Fig. 5c compares the behavior of SS, the roughness and the grain size determined from AFM measurements. SS decreases whereas the grain size and the roughness increase before 200°C. It seems that this decrease of SS is governed mainly by the decrease of the defect density inside the channel due to the increase of the grain size. Between 200°C and 250°C, SS increases, the grain size increases only a little bit however the roughness increases strongly. In conclusion, as the grain size and then the defect density inside the channel saturates more or less, the increase of SS is mainly attributed to the increase of the roughness. Coming back to the link between the mobility and the grain size, the conduction models in polycrystalline materials can be apply, particularly the conduction in thin-film transistors with polycrystalline active layer. If we assume that polycrystalline materials are composed by a chain of crystalline regions with superficial discrete traps at the grain boundaries, the drain current, I D , and the gate voltage, V G , of these transistors are found [START_REF] Levinson | Conductivity behavior in polycrystalline semiconductor thin film transistors[END_REF][START_REF] Bonnaud | Poly-Si Thin Film and Substrate Materials[END_REF] to be related in the subthreshold regime to: [START_REF] Sirringhaus | High-resolution inkjet printing of all-polymer transistor circuits[END_REF] W and L are the width and the length of the channel. C ins is the capacitance per area unit of the gate insulator. V min is the gate voltage at the minimum of I D from which the channel start to form and then I D start to increase. V D is the drain voltage. q is the electronic charge. t is the thickness of the channel. N T , is the superficial concentration of defects in the grain boundaries.  S is the permittivity of the semiconducting active layer. kT is the thermal energy.

In this model [START_REF] Seto | The electrical properties of polycrystalline silicon films[END_REF], the mobility is thermally activated following the relationship: (8) where E b is the barrier energy at the grain boundary.

The relation [START_REF] Sirringhaus | High-resolution inkjet printing of all-polymer transistor circuits[END_REF] is valid when the electron concentration in the channel (in the case of N-type transistor) is larger than a critical concentration N * =N T /G in which G is the size of the grains. Using this relation [START_REF] Sirringhaus | High-resolution inkjet printing of all-polymer transistor circuits[END_REF], the semi-logarithmic plot of I D /(V G -V min ) versus 1/(V G -V min ) may be fitted with a linear function. The slope of the function is proportional to N T 2 .

As shown in Fig. 5d, semi-logarithmic plot for OFETs annealed at 160°C and 200°C. Both plots have a linear range. The departure from the linearity occurs at larger 1/(V G -V min ) and the slope of the linear functions is lower for 200°C annealed OFET than for 160°C annealed OFET, meaning lower slope for largest grain size OFET. Larger 1/(V G -V min ), corresponding then to larger N * =N T /G, means larger grains in 200°C annealed OFET.

The slope of the linear function can lead us to calculate N T . Besides the physical constant q,  0 and k, the relative dielectric constants are taken 2.7 for SU8 gate insulator [START_REF] Bebiche | OTFTs de type N à base de semiconducteurs π-conjugués: fabrication, performance et stabilité[END_REF] and 5 for C 60 film [START_REF] Chern | Low-frequency dielectric permittivity of C 60[END_REF]. The thickness of the gate insulator is 1.2 µm. The actual unknown is the thickness t of the channel. To jump this difficulty, we can take the channel extending into the full thickness of the active layer that is not realistic, but can give an order of the minimum defect density in the grain boundaries. An order of the maximum defect density can be obtained considering very thin channel, 3 nm for example.

Taking then t=125 nm, the calculated superficial defect density N T at the grain boundaries is 3x10 11 /cm 2 and 2.5x10 11 /cm 2 for 160°C and 200°C annealed OFETs. N T increases to 1.9x10 12 /cm 2 and 1.6x10 12 /cm 2 for 160°C and 200°C annealed OFETs if we consider a channel thickness of 3 nm (~2-unit cells of C60 crystalline network).

The first observation is the low defect density in the grain boundaries of such organic material. For comparison, the same model gives an order of 10 12 /cm 2 for N T in polycrystalline silicon transistors considering a channel thickness of 30 nm [START_REF] Mourgues | Density of states in the channel material of low temperature polycrystalline silicon thin film transistors[END_REF].

The second observation is the slight difference between the values of N T in both OFETs, even if N T is lower in 200°C annealed OFETs. The large difference between the mobility values in these OFETs is mainly due to the large increase of the grain size and not to a decrease of the defect density inside the grain boundaries, leading to much lower N * =N T /G ratio in 200°C annealed OFET. In conclusion, the mobility is primarily governed by the grain size (or the number of grains in the channel) than by the defect density in the grain boundaries. Importantly, this direct quantitative link between the physical structure of the active layer and the electrical parameters of organic transistor was never encountered in the literature in our knowledge.

In a summary of this section, the effect of the annealing temperature on printed C 60 OFET's properties has been studied. Optimum parameters are obtained for 200°C annealing temperature. Furthermore, compare to OFET annealed at 250°C, OFET annealed at 200°C show similar mobility, lower subthreshold slope and higher on-current to off-current ratio.

Effect of the thickness of the C 60 active layer on the OFET annealed at 200°C

Another experimental parameter that can influence the properties of the OFETs is the thickness of the C 60 active layer. Following the previous study, the effect of this thickness on the properties of the OFETs, annealed at 200°C after their fabrication, is presented hereafter. Fig. 6 shows the mean transfer and output characteristics of 4 OFETs having a C 60 active layer with a thickness of 60nm, 90 nm, 125 nm or 250nm. All the OFETS have the same size, with a channel width W=4000 µm and a channel length L=150 µm. The best characteristics are obtained for an active layer of 90 nm thick. The behavior of the electrical parameters as a function of the thickness is shown also in Fig. 6c and their values are presented in Table 3. It seems that the low performance of the 60 nm thick active layer OFETs is probably due to low coverage between the active layer and the 205 nm thick source and drain contacts. The performance of 90 nm thick and 125 nm thick active layer OFETs are more or less similar with higher mobility for 90 nm thick OFET. The bad subthreshold slope of 250 nm thick active layer OFET can be mainly due to the high roughness of the surface. Main mean electrical parameters of several OFETs having the same size (channel width W=4000 µm, channel length L=150 µm) and a C 60 active layer with a thickness t C60 of 60nm, 90 nm, 125 nm or 250nm, all annealed at 200°C. V TH is the threshold voltage, SS the subthreshold slope, µ the field effect mobility, R C the contact resistance, µ after R C the mobility after correction by R C , I off is the minimum current and I on /I off is the ratio between the maximum current and the minimum current. In conclusion, the best conditions to prepare the evaporated C 60 active layer of printed bottom-gate/bottom-contacts OFETs, are obtained with a thickness of 90 nm and a post-thermal annealing equal 200°C.

t C60 (nm) V TH (V) SS (V/dec) µ (cm 2 /V.s) R c () µ after R c (cm 2 /V.
To re-inforce this conclusion, the drain-current of OFETs using 90 nm thick C 60 active layer and annealed at different temperatures, was plotted as a function of the time when constant gate voltage V GS and drain voltage V DS are applied. Under V GS =40V and V DS =20V, the drain current decreases with the time (Fig. 6d). In these conditions, the 200°C annealed OFETs have the slowest decrease leading to more reliable value of the current under constant biasing.

Dynamic behavior of the optimized printed C 60 based OFET

Defining then the best conditions to prepare the evaporated C 60 active layer of printed bottom-gate/bottom-contacts OFETs and knowing their static performances, is a prerequisite step to evaluate their optimum dynamic performance. The dynamic performance is studied through the behavior of resistive loaded OFET under square pulsed voltage applied to its gate.

Static output characteristic of the OFET as an inverter is presented in Fig. 7. The load resistance R is fixed to 500M and the circuit is biased by V DD =20V. In these polarization conditions, the inverter has a gain (maximum of the differentiation dV OUT /dV IN ) of 3 (Fig. 7a). This value depends on the load resistance of course. But, it is in the order of the gain of organic based OFETs inverters [START_REF] Chang | A Circuits and Systems Perspective of Organic/Printed Electronics: Review, Challenges, and Contemporary and Emerging Design Approaches[END_REF]. The minimum output voltage V OL with the maximum input voltage V IN is explained by the resistance value of the channel in on-state compared to the load resistance.

More interestingly than the static characteristic, the dynamic response of the OFET to a voltage pulse gives a better idea of its performance. When applying 20V pulse of 0.5 second duration (1 Hz frequency) to the gate of the OFET, the behavior of the output voltage gives its time response (as shown in fig. 7b). When the gate-source voltage switches from 20V to 0V, the OFET status switches from the on-state to the off-state and then the output voltage has to switch from nearly 0V to nearly 20V. The switch time is defined usually by the time from 10% to 90% of the maximum voltage. In the case of the present OFET, the switching time is 220 ms. This time is due to the load resistance as its value is much lower than the resistance of the channel in off-regime, 500M compared to some hundreds G.

The more significant time is the switching time between the off-state and the on-state that depends nearly only on the transistor. This time corresponds to the decrease of the output voltage from the maximum to the minimum value. The switching time from 90% to 10% of the maximum voltage is 13 ms here. Present OFET can answer to more than 10Hz frequency. However, the frequency can be increased more by optimizing the overlap between the gate contacts and source and drain contacts and by chemically modifying the surface of drain and source electrodes before the deposition of C60 active layer as we have done previously for photolithography processed C60 OFETs where 1.5 cm 2 /V.s was obtained [START_REF] Robin | Improvement of n-type OTFT electrical stability by gold electrode modification[END_REF]. In fact, the frequency can be considered depending on the applications. For example, 100Hz frequency is enough if OFETs are integrated into a treatment electronic circuit of signals coming from human body activities. Such 100Hz frequency can be reached with such process improvement.

Conclusion

The main purpose of present work was devoted to the optimization of the last deposited layer in bottom-gate bottom contacts printed C 60 based OFET. This last deposited layer is a film of C 60 .The first steps of the process were the printing of the silver gate contact, the photoresist SU8 gate insulator, the drain and source silver contacts. All these steps were previously optimized and the results were published in different papers [START_REF] Robin | Epoxy based ink as versatile material for inkjet-printed devices[END_REF][START_REF] Moon | Morphological impact of insulator on inkjet-printed transistor[END_REF] The effect of the post-annealing temperature of the fabricated OFET was studied firstly. Direct correlation between the structure of the C 60 film (grain size, surface roughness) as analyzed from AFM observation and electrical parameters of the OFET was highlighted. More than the correlation, proportionality between the grain size and the field effect mobility was evidenced. The surface roughness is effective only when the grain size does not change a lot. The proportionality between the grain size and the mobility led us to involve the usual Seto model [START_REF] Seto | The electrical properties of polycrystalline silicon films[END_REF] and its derivatives established for polycrystalline films and thin film transistors based on. Treatment of the experimental curves fit well with this model leading us to the calculation of the defect density in the grain boundaries. In our knowledge, such detailed analysis was never made before on the effect of the post-annealing temperature on organic transistors. Determining the 200°C optimum post-annealing temperature, the thickness of the C 60 active layer was optimized at 90 nm and confirmed by electrical stability study of the drain current during continuous polarization of the transistor. Considering the potential main application of such transistors in electronic circuits for the treatment of analog signals coming from the human body activities, the response of present OFETs to a voltage pulse applied to the gate was measured. A switching time of 13 ms was obtained, leading to application frequency of some 10Hz.

Future work on the optimization of the overlap between the contacts and on the chemical modification of the surface of the contacts will be done soon in the goal to improve the stability and to increase the working frequency. The use of active charge (another NMOS OFET) in pseudo-CMOS inverter will be integrated soon in the purpose to get symmetric answer to a voltage pulse (similar rise time and fall time) will be done very soon. Finally, the integration of printed pentacene based P-type OFET is planned to reach the final goal of CMOS electronics.

Figure Captions

Table Captions

Table 1: RMS roughness and grain size calculated using PSDF formulation (1)

Table 2: Main mean electrical parameters of several OFETs having the same size (channel width W=4000 µm, channel length L=150 µm) and 125 nm thick C 60 active layer, measured as-fabricated and after annealing at T A =160°C, 200°C and 250°C. V TH is the threshold voltage, SS the subthreshold slope, µ the field effect mobility, R C the contact resistance, µ after R C the mobility after correction by R C , I off is the minimum current and I on /I off is the ratio between the maximum current and the minimum current

Fig. 1 :

 1 Fig. 1: Cross-section of the C60 OFETs using bottom gate and bottom contacts geometry. The figure highlights the problematic of the high aspect ratio between the drain and source contacts and the gate insulator

Fig. 2 :

 2 Fig. 2: Fabrication process of C 60 OFET

Fig. 3 :

 3 Fig. 3: 2D ( ) AFM images, corresponding 3D enlarged ( ) (a) as-deposited; and after annealing at 160Cº (b), 200°C (c) and 250°C (d)3.2.1 OFET with 125 nm thick C 60 active layer as a function of the annealing temperatureFig.4shows the mean transfer and output characteristics of 4 OFETs as-fabricated

Fig. 4 :

 4 Fig.4: (a)Mean transfer characteristics (V D =20V) and (b) output characteristics (V G =20V) of several OFETs having the same size (channel width W=4000 µm, channel length L=150 µm) and 125 nm thick C 60 active layer, measured as-fabricated and after annealing at 160°C, 200°C and 250°C.

Fig. 5 :

 5 Fig.5: (a) Behavior of the mean threshold voltage V THMEAN , subthreshold slope SS MEAN and field effect mobility, when the annealing temperature increases; (b) Similar behavior of the mobility and the grain size as a function of the annealing temperature; (c) Behavior of the subthreshold slope SS, the surface roughness and the grain size as a function of the annealing temperature; (d) Semi-logarithmic plot of I D /(V G -V min ) versus 1/(V G -V min ) for OFETs annealed at 160°C and 200°C

Fig. 6 :

 6 Fig. 6: (a), (b), (c): Mean transfer and output characteristics of several OFETs having the same size (channel width W=4000 µm, channel length L=150 µm) and a C 60 active layer with a thickness of 60nm, 90 nm, 125 nm or 250nm. The behavior of their mean electrical parameters (Threshold voltage, subthreshold slope and mobility) is also reported. (d) Drain-current of OFETs using 90 nm thick C60 active layer un-annealed and annealed at different temperatures, 160°C, 200°C, 250°C as a function of the time under 40V gate-source voltage and 20V drain-source voltage

Fig. 7 :

 7 Fig. 7: (a) Static output characteristic of printed C 60 based OFET loaded by 500MΩ resistance and (b) its response to 1Hz pulse. The inset shows the gain of the inverter with a maximum value of 3.

Fig. 1 :Fig. 2 :Fig. 3 :Fig. 4 :

 1234 Fig. 1: Cross-section of the C60 OFETs using bottom gate and bottom contacts geometry. The figure highlights the problematic of the high aspect ratio between the drain and source contacts and the gate insulator Fig. 2: Fabrication process of C 60 OFET Fig. 3: 2D ( ) AFM images, corresponding 3D enlarged ( ) (a) as-deposited; and after annealing at 160Cº (b), 200°C (c) and 250°C (d) Fig. 4: (a)Mean transfer characteristics (V D =20V) and (b) output characteristics (V G =20V) of several OFETs having the same size (channel width W=4000 µm, channel length L=150 µm) and 125 nm thick C 60 active layer, measured as-fabricated and after annealing at 160°C, 200°C and 250°C.

Fig. 5 :Fig. 6 :

 56 Fig. 5: (a) Behavior of the mean threshold voltage V THMEAN , subthreshold slope SS MEAN and field effect mobility, when the annealing temperature increases; (b) Similar behavior of the mobility and the grain size as a function of the annealing temperature; (c) Behavior of the subthreshold slope SS, the surface roughness and the grain size as a function of the annealing temperature; (d) Semi-logarithmic plot of I D /(V G -V min ) versus 1/(V G -V min ) for OFETs annealed at 160°C and 200°C Fig. 6: (a), (b), (c): Mean transfer and output characteristics of several OFETs having the same size (channel width W=4000 µm, channel length L=150 µm) and a C 60 active layer with a thickness of 60nm, 90 nm, 125 nm or 250nm. The behavior of their mean electrical parameters (Threshold voltage, subthreshold slope and mobility) is also reported. (d) Drain-current of OFETs using 90 nm thick C60 active layer un-annealed and annealed at different temperatures, 160°C, 200°C, 250°C as a function of the time under 40V gate-source voltage and 20V drain-source voltage

Fig. 7 :

 7 Fig. 7: (a) Static output characteristic of printed C 60 based OFET loaded by 500MΩ resistance and (b) its response to 1Hz pulse. The inset shows the gain of the inverter with a maximum value of 3.

Table 3 :

 3 Main mean electrical parameters of several OFETs having the same size (channel width W=4000 µm, channel length L=150 µm) and a C 60 active layer with a thickness t C60 of 60nm, 90 nm, 125 nm or 250nm, all annealed at 200°C. V TH is the threshold voltage, SS the subthreshold slope, µ the field effect mobility, R C the contact resistance, µ after R C the mobility after correction by R C , I off is the minimum current and I on /I off is the ratio between the maximum current and the minimum current. Investigation of thermal annealing process influence on printed n-type OFET  Investigation of organic semiconductor layer thickness impacts on C60 OFET  Investigation on relationships between OFET stability and C60 crystalline structure  Investigation on Pseudo-CMOS inverter logic circuits

Table 1 :

 1 RMS roughness and grain size calculated using PSDF formulation[START_REF] Hayward | Wearable Technology 2017-2027: Markets, Players, Forecasts[END_REF]

	Annealing	Roughness	Grain size
	Temperature (°C)	(nm)	(nm)
	As-deposited	1.42	4.72
	160	1.75	7.70
	200	1.91	13.46
	250	2.72	14.16


Neil Zeghidour

Nicolas Usunier

Iasonas Kokkinos

Thomas Schatz

Gabriel Synnaeve

Emmanuel Dupoux

LEARNING FILTERBANKS FROM RAW SPEECH FOR PHONE RECOGNITION

We train a bank of complex filters that operates on the raw waveform and is fed into a convolutional neural network for end-to-end phone recognition. These time-domain filterbanks (TD-filterbanks) are initialized as an approximation of melfilterbanks, and then fine-tuned jointly with the remaining convolutional architecture. We perform phone recognition experiments on TIMIT and show that for several architectures, models trained on TD-filterbanks consistently outperform their counterparts trained on comparable mel-filterbanks. We get our best performance by learning all front-end steps, from pre-emphasis up to averaging. Finally, we observe that the filters at convergence have an asymmetric impulse response, and that some of them remain almost analytic.

INTRODUCTION

Speech features such as gammatones or mel-filterbanks (MFSC, for mel-frequency spectral coefficients) were designed to match the human perceptual system [START_REF] Davis | Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences[END_REF][START_REF] Rd Patterson | An efficient auditory filterbank based on the gammatone function[END_REF], and contain invaluable priors for speech recognition tasks. However, even if a consensus has been reached on the proper setting of the hyperparameters of these filterbanks along the years, there is no reason to believe that they are optimal representations of the input signal for all recognition tasks. In the same way deep architectures changed the landscape of computer vision by directly learning from raw pixels [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], we believe that future end-to-end speech recognition system will learn directly from the waveform.

There have been several attempts at learning directly from the raw waveform for speech recognition [START_REF] Palaz | Estimating phoneme class conditional probabilities from raw speech signal using convolutional neural networks[END_REF][START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF][START_REF] Tjandra | Attention-based wav2text with feature transfer learning[END_REF]. [START_REF] Hoshen | Speech acoustic modeling from raw multichannel waveforms[END_REF][START_REF] Sainath | Learning the speech front-end with raw waveform cldnns[END_REF] propose an architecture composed of a convolutional layer followed by max-pooling and a nonlinearity, so that gammatone filterbanks correspond to a particular configuration of the network. [START_REF] Tjandra | Attention-based wav2text with feature transfer learning[END_REF] explore an alternative architecture, with the intention to represent MFSC rather than gammatones. They propose a 4-layer convolutional architecture followed by two networks-in-networks [START_REF] Lin | Network in network[END_REF], pretrained to reproduce MFSC.

We also focus on MFSC because they are the front-end of state-of-the-art phone [START_REF] Tóth | Phone recognition with hierarchical convolutional deep maxout networks[END_REF] and speech [START_REF] Xiong | Achieving human parity in conversational speech recognition[END_REF] recognition systems. Our work builds on [START_REF] Andén | Deep scattering spectrum[END_REF], who introduce a time-domain approximation of MFSC using the first-order coefficients of a scattering transform. This leads us to study an architecture using a convolutional layer with complex-valued weights, followed by a modulus operator and a low-pass filter. In contrast to [START_REF] Tjandra | Attention-based wav2text with feature transfer learning[END_REF], we propose a lightweight architecture that serves as a plug-in, learnable replacement to MFSC in deep neural networks. Moreover, we avoid pretraining by initializing the complex convolution weights with Gabor wavelets whose center frequency and bandwidth match those of MFSC.

We perform phone recognition experiments on TIMIT and show that given competitive end-to-end models trained with MFSC as inputs, training the same architectures by replacing the MFSC with the learnable architecture leads to performances that are better than when using MFSC. Moreover, our best model is obtained by learning everything except for the non-linearities, including a pre-emphasis layer.

TIME-DOMAIN MFSC

We present the standard MFSC and their practical implementation. We then describe a learnable replacement of MFSC that uses only convolution operations in time domain, and how to set the weights to reproduce MFSC.

MFSC computation

Given an input signal x, MFSC are computed by first taking the short-time Fourier transform (STFT) of x followed by taking averages in the frequency domain according to triangular filters with centered frequency and bandwidth that increase linearly in log-scale. More formally, let φ be a Hanning window of width s and (ψ n ) n=1..N be N filters whose squared frequency response are triangles centered on (η n ) n=1..N with full width at half maximum (FWHM) (w n ) n=1..N . Denoting by x t : u → x(u)φ(t -u) the windowed signal at time step t, and f the Fourier transform of function f , the filterbank is the set of N functions (t → M x(t, n)) n=1..N :

M x(t, n) = 1 2π |x t (ω)| 2 | ψn (ω)| 2 dω , (1)

Approximating MFSC with convolutions in time

As in [START_REF] Andén | Deep scattering spectrum[END_REF], we approximate MFSC in the time domain using:

M x(t, n) ≈ |x * ϕ n | 2 * |φ| 2 (t). ( 2
)
where ϕ n is a wavelet that approximates the n-th triangular filter in frequency, i.e. | φn |2 ≈ | ψn | 2 , while φ(t) is the Hanning window also used for the MFSC. The approximation is valid when the time support of ϕ n is smaller than that of φ.

This approximation of MFSC is also known as a firstorder scattering transform. This is the foundation of the deep scattering spectrum [START_REF] Andén | Deep scattering spectrum[END_REF], which cascades scattering transforms to retrieve information that is lost in the MFSC. Deep scattering spectra have been used as inputs to neural networks trained for phone recognition [START_REF] Peddinti | Deep scattering spectrum with deep neural networks[END_REF] or classification [START_REF] Zeghidour | A deep scattering spectrumdeep siamese network pipeline for unsupervised acoustic modeling[END_REF], which showed better performances than comparable models trained on MFSC. In this work, we do not use the deep scattering spectrum. First-order scattering coefficients provide us with both a design for the first layers of the network architecture to operate on the waveform, and an initialization that approximates the MFSC computation.

Given the MFSC center frequencies (η n ) n=1..N and FWHM (w n ) n=1..N , we use (2) to approximate MFSC with Gabor wavelets:

ϕ n (t) ∝ e -2πiηnt 1 √ 2πσ n e -t 2 2σ 2 n . ( 3
)
where η n is the desired center frequency, and the width parameter σ n of the Gabor wavelet is set to match the desired FWHM w n . Since for a frequency ξ we have

φn (ξ) ∝ √ σ n e -1 , the FWHM is 2 √ 2 log 2σ -1 n and we take σ n = 2 √ 2 log 2 wn
. Each ϕ n is then normalized to have the same energy as ψ n . Figure 1 1) is a grouped convolution, meaning that each output filter only sees the input filter with the same index. The decimation of 10ms is implemented in the stride of 160 of this layer. Notice that to approximate the mel-filterbanks, the square of the Hanning window is used and biases in both convolutional layers are set to zero. We keep them to zero during training. We add log compression to the output of the grouped convolution after adding 1 to its absolute value since we do not have positivity constraints on [START_REF] Tóth | Phone recognition with hierarchical convolutional deep maxout networks[END_REF] are trained in an end-to-end fashion.

the weights when learning. Contrarily to the MFSC, there is no mean-variance normalization after the convolutions, but on the waveform. In the default implementation of the TD-filterbanks, we do not apply pre-emphasis. However, in our last experiment, we add a convolutional layer below the TD-filterbanks, with width 2 and stride 1, initialized with the pre-emphasis parameters, as another learnable component.

EXPERIMENTS

Setting

We perform phone recognition experiments on TIMIT [START_REF] John S Garofolo | Timit acoustic-phonetic continuous speech corpus[END_REF] using the standard train/dev/test split. We train and evaluate our models with 39 phonemes. We experiment with three architectures. The first one consists of 5 layers of convolution of width 5 and 1000 feature maps, with ReLU activation functions, and a dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] of 0.5 on every layer but the input and output ones. The second model has the same architecture but a dropout of 0.7 is used. The third model has 8 layers of convolution, PReLU [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF] nonlinearities and a dropout of 0.7. All our models are trained end-to-end with the Autoseg criterion [START_REF] Collobert | Wav2letter: an end-to-end convnet-based speech recognition system[END_REF], using stochastic gradient descent. We compare all models using either the baseline MFSC as input or our learnable TD-filterbank front-end. We perform the same grid-search for both MFSC baselines and models trained on TD-filterbanks, using learning rates in (0.0003, 0.003) for the model and learning rates in (0.03, 0.003) for the Autoseg criterion, training every model for 2000 epochs. We use the standard dev set for early stopping and hyperparameter selection.

Different types of TD-filterbanks

Throughout our experiments, we tried four different settings for the TD-filterbank layers:

• Fixed: Initialize the layers to match MFSC and keep their parameters fixed when training the model

• Learn-all: Initialize the layers and let the filterbank and the averaging be learned jointly with the model • Learn-filterbank: Start from the initialization and only learn the filterbank with the model, keeping the averaging fixed to a squared hanning window • Randinit: Initialize the layers randomly and learn them with the network Table 2 shows comparative performance of an identical architecture trained on the four types of TD-filterbanks. We can observe that training on fixed layers moderately worsens the performance, we hypothesize that this is due to the absence of mean-variance normalization on top of TD-filterbanks as is performed on MFSC. A striking observation is that a model trained on TD-filterbanks initialized randomly performs considerably worse than all other models. This shows the importance of the initialization. Finally, we observe better results when learning the filterbank only compared to learning the filterbank and the averaging but depending on the architecture it was not clear which one performs better. Moreover, when learning both complex filters and averaging, we observe that the learned averaging filters are almost identical to their initialization. Thus, in the following experiments, we choose to use the Learn-filterbank mode for the TD-filterbanks.

Results

We report PER on the standard dev and test sets of TIMIT.

For each architecture, we can observe that the model trained on TD-filterbanks systematically outperforms the equivalent model trained on MFSC, even though we constrained our TDfilterbanks such that they are comparable to the MFSC and do not learn the low-pass filter. This shows that by only learning a new bank of 40 filters, we can outperform the MFSC for phone recognition. This gain in performance is obtained at a minimal cost in terms of number of parameters: even for the smallest architecture, the increase in number of parameters in switching from MFSC to TD-filterbanks is 0.31%. We also compare to baselines from the literature. One baseline trained on the waveform gets a PER of 29.1% on the test set, which is in a range 8.8% -11.1% absolute above our models trained on the waveform. The Wavenet architecture, also trained on the waveform, yields a PER of 18.8, which is higher than our best models despite using the phonetic alignment and an auxiliary prediction loss. Our best model on the waveform also outperforms a 2-dimensional CNN trained on MFSC and an LSTM trained on MFSC with derivatives. Finally, by adding a learnable pre-emphasis layer below the TD-filterbanks, we reach 18% PER on the test set.

ANALYSIS OF LEARNED FILTERS

We analyze filters learned by the first layer of the CNN-8L-PReLU-do0.7 + TD-filterbanks model. Examples of learned filters are shown in Figure 2. The magnitude of the frequency response for each of the 40 filters is plotted in Figure 3. Overall, the filters tend to be well localized in time and frequency, and a number of filters became asymmetric during the learn-ing process, with a sharp attack and slow decay of the impulse response. This is a type of asymmetry also found in human and animal auditory filters estimated from behavioral and physiological data [START_REF] Evan | Efficient auditory coding[END_REF]. In Figure 3, we further see that the initial mel-scale of frequency is mostly preserved, but that a lot of variability in the filter bandwidths is introduced.

A prominent question is whether the analyticity of the initial filterbank is preserved throughout the learning process even though nothing in our optimization method is biased towards keeping filters analytic. A positive answer would suggest that complex filters in their full generality are not necessary to obtain the increase in performance we observed. This would be especially interesting because, unlike arbitrary complex filters, analytic filters have a simple interpretation in terms of real-domain signal processing: taking the squared modulus of the convolution of a real signal with an analytic filter performs a sub-band Hilbert envelope extraction [START_REF] Jl Flanagan | Parametric coding of speech spectra[END_REF].

A signal is analytic if and only if it has no energy in the negative frequencies. Accordingly, we see in Figure 3 that there is zero energy in this region for the initialization filterbank. After learning, a moderate amount of energy appears in the negative frequency region for certain filters. To quantify this, we computed for each filter the ratio r a between the energy in negative versus positive frequency components 1 . This ratio is 0 for a perfectly analytic filter and 1 for a purely real filter. We find an average r a for all learned filters of .26. Filters with significant energy in negative frequencies are mostly filters with an intermediate preferred frequency (between 1000Hz and 3000Hz) and their negative frequency spectrum appears to be essentially a down-scaled version of their positive frequency spectrum.

CONCLUSION

We proposed a lightweight architecture which, at initialization, approximates the computation of MFSC and can then be fine-tuned with an end-to-end phone recognition system. With a number of parameters comparable to standard MFSC, a TD-filterbank front-end is consistently better in our experiments. Learning all linear operations in the MFSC derivation, from pre-emphasis up-to averaging provides the best model. In future work, we will perform large scale experiments with TD-filterbanks to test if a new state-of-the-art can be achieved by training from the waveform.

Fig. 1 .

 1 Fig. 1. Frequency response of filters, and output of MFSC and their time-domain approximation on a sentence of TIMIT.

  (a) shows in frequencydomain the triangular averaging operators of usual MFSC and the corresponding Gabor wavelets. Figures1 (b) and (c) compare the 40-dimensional spectrograms of the MFSC and the Gabor wavelet approximation on a random sentence of the

Fig. 2 .

 2 Fig. 2. Examples of learned filters. Filters' real parts in blue; imaginary part in red.

Fig. 3 .

 3 Fig. 3. Heat-map of the magnitude of the frequency response for initialization filters (left) and learned filters (right).

Table 1 .

 1 Details of the layers for the TD-filterbanks.

	Layer type	Input size Output size Width Stride
	Conv.	1	80	400	1
	L2-Pooling	80	40	-	-
	Square	-	-	-	-
	Grouped conv.	40	40	400	160
	Absolute value	-	-	-	-
	Add 1, Log	-	-	-	-
	Learning mode	Dev PER Test PER
	MFSC		17.8	20.6
	Fixed		18.3	21.8
	Learn-all		17.4	20.6
	Learn-filterbank	17.3	20.3
	Randinit		29.2	31.7

Table 2 .

 2 PER of the CNN-5L-ReLU-do0.7 model trained on MFSC and different learning setups of TD-filterbanks. Since the energy of the Gabor wavelets approximating standard MFSC has a time spread smaller than the Hanning window, the complex wavelet+modulus operations |x * ϕ n | 2 are implemented as a convolutional layer taking the raw wav as input, with a width W = 400 and 2N = 80 filters (40 filters for the real and imaginary parts respectively). This layer is on the top row of Table1. The modulus operator is implemented with "feature L2 pooling", a layer taking an input z of size 2N and outputs z of size N such that z k = z 2 2k-1 + z 2 2k . The windowing layer (third row of Table

TIMIT corpus after mean-variance normalization, showing that the spectrograms are similar.

MFSC specification. The standard setting in speech recognition is to start from the waveform sampled at 16kHz and represented as 16-bit signed integers. The STFT is computed with 512 frequency bins using Hanning windows of width 25ms, and decimation is applied by taking the STFT every 10ms. There are N = 40 filters, with center frequencies (η n ) n=1..N that span the range 64Hz -8000Hz by being equally spaced on a mel-scale. The final features are the log(max(M x(t, n), 1)). In practice, the STFT is applied to the raw signal after a pre-emphasis with parameter 0.97, and coefficients have mean-variance normalization per utterance.

Learnable architecture specification. The time-domain convolutional architecture is summarized in Table

1

. With a waveform sampled at 16kHz, a Hanning window is a convolution operator with a span of W = 400 samples (25ms).

Table 3 .

 3 PER (Phone Error Rate) on TIMIT, in percentages. All models but

	Model

σ 2 n (ξ-ηn) 2

Our model cannot identify if a given filter plays the role of the real or imaginary part in the associated complex filter. We chose the assignment yielding the smallest ra.

ACKNOWLEDGEMENTS

Authors thank Mark Tygert for useful discussions, and Vitaliy Liptchinsky and Ronan Collobert for help on the implementation. This research was partially funded by the European Research Council (ERC-2011-AdG-295810 BOOTPHON), the Agence Nationale pour la Recherche (ANR-10-LABX-0087 IEC, ANR-10-IDEX-0001-02 PSL*).