Leila Zouari

Gérard Chollet

Efficient Codebooks for Fast and Accurate Low Resource ASR Systems

Keywords: Speech recognition, Gaussian selection, codebook 1 GPS : Global Positioning System 2 GSM : Global System for Mobile Communications 3 PDA : Personal Digital Assistant

Today, speech interfaces have become widely employed in mobile devices, thus recognition speed and resource consumption are becoming new metrics of Automatic Speech Recognition (ASR) performance. For ASR systems using continuous Hidden Markov Models (HMMs), the computation of the state likelihood is one of the most time consuming parts. In this paper, we propose novel multi-level Gaussian selection techniques to reduce the cost of state likelihood computation. These methods are based on original and efficient codebooks. The proposed algorithms are evaluated within the framework of a large vocabulary continuous speech recognition task.

Introduction

The proliferation of mobile devices in daily life has created a great demand for efficient and simple interfaces. In particular, speech recognition being a key element of the conversational interface, there is a significant requirement for low-resource and accurate automatic speech recognition systems. Recent mobile devices (GP S 1 , GSM 2 , P DA 3 , . . .) offer a large set of functionalities but their resources are still limited for accurate continuous speech recognition engines. Indeed, state-of-the-art continuous speech recognition systems use Hidden Markov Models (HMM) with many tens of thousands of Gaussian distributions to achieve improved recognition. The computation of

ACCEPTED MANUSCRIPT

the emission probability of these Gaussian distributions is time consuming. As the performance and the speed of speech recognition systems are highly dependent on the number of HMM Gaussians, reducing the number of Gaussians without decreasing the system performance is of major interest [START_REF] Suontausts | Fast Decoding Techniques for Practical Realtime Speech Recognition Systems[END_REF]; [START_REF] Chan | Four Layer Categorization Scheme of Fast GMM Computation Techniques in Large Vocabulary Continuous Speech Recognition Systems[END_REF]; [START_REF] Sankar | Parameter Tying and Gaussian Clustering for Faster, Better, and Smaller Speech Recognition[END_REF]; [START_REF] Sagayama | On the Use of Scalar Quantization for Fast HMM Computation[END_REF]; [START_REF] Takahashi | Four-level Tied-structure for Efficient Representation of Acoustic Modeling[END_REF]; [START_REF] Digalakis | Efficient Speech Recognition using Subvector Quantization and Discrete-Mixture HMMs[END_REF]).

According to previous studies [START_REF] Bocchieri | Vector Quantization for the Efficient Computation of Continuous Density Likelihoods[END_REF]; [START_REF] Mak | Subspace Distribution Clustering Hidden Markov Model[END_REF]; Jurgen et al. (1996); [START_REF] Gales | Use of Gaussian Selection in Large Vocabulary Continuous Speech Recognition using HMMs[END_REF]; [START_REF] Chan | On Improvements to CI based GMM Selection[END_REF]) only few Gaussians dominate the state likelihood computation. Hence, different techniques were developed to select them [START_REF] Aiyer | Rapid Likelihood Calculation of Subspace Clustered Gaussian Components[END_REF]; Xiao et al. (2006); [START_REF] Lee | A New Phonetic Tied-Mixture Model for Efficient Decoding[END_REF]; [START_REF] Filali | Data-driven Vector Clustering for Low Memory Footprint ASR[END_REF]; [START_REF] Sankar | Improved Modeling and Efficiency for Automatic Transcription of Broadcast News[END_REF]). These techniques can be divided into two categories:

• State based methods : These methods aim to reduce the number of Gaussians per state. They are often applied to acoustic models with a high number of Gaussians per state such as semi-continuous or context independent models (Jurgen et al. (1995); [START_REF] Woszczyna | Fast Speaker Independent Large Vocabulary Speech Recognition[END_REF]). • Model based methods : These methods are applied to models with a limited number of Gaussians per state such as triphones [START_REF] Bocchieri | Vector Quantization for the Efficient Computation of Continuous Density Likelihoods[END_REF]; [START_REF] Mak | Subspace Distribution Clustering Hidden Markov Model[END_REF]; [START_REF] Gales | State based Gaussian Selection in Large Vocabulary Continuous Speech Recognition using HMMs[END_REF]). Their objective is to decrease the total number of Gaussians (belonging to all the states).

Gaussian selection is often performed in two steps : codebook construction (by Gaussian clustering) and Gaussian selection. In this paper, we propose several Gaussian selection techniques that reduce the cost of likelihood computation either at the state or model level. These techniques use a novel codebook construction process. The proposed methods are evaluated within the framework of a large vocabulary continuous speech recognition task and are compared to existing methods.

The document is organized as follows: section 2 describes the state based Gaussian clustering and selection. In particular, a weighted Kullback Leibler metric and several clustering criteria are introduced to improve the clustering. They are evaluated in the context of model shortening and used in a state-based Gaussian selection. Furthermore, a multi-level Gaussian selection algorithm is proposed. Section 3 describes our contributions to the model based approaches. Two mains propositions are detailed: contextual Gaussian selection and contextual sub-vector quantization. The conclusions and prospective work are described in section 4.

State-based clustering and selection

The state-based Gaussian selection is performed in two steps : classification and selection. During the first step, state Gaussians are grouped into clusters. Generally, they are organized into a tree structure based on their mean vector values [START_REF] Ortmanns | Fast Likelihood Computation Methods for Continuous Mixture Densities in Large Vocabulary Speech Recognition[END_REF]; Jurgen et al. (1996); [START_REF] Woszczyna | Fast Speaker Independent Large Vocabulary Speech Recognition[END_REF]; [START_REF] Padmanablan | Decision-Tree based Feature Space Quantization for Fast Gaussian Computation[END_REF]) and the Euclidian distance. The second step consists in selecting Gaussians to be used for the likelihood computation.

In this section, we propose a novel Gaussian clustering (ie., codebook construction) algorithm. This algorithm is based on a new metric and several empirical criteria. Then, we evaluate this clustering (and the new metric) within the framework of model shortening. Finally, we investigate the use of the new codebook in a state-based multi-level Gaussian selection.

Gaussian classification

For each Gaussian mixture, distributions are grouped into a binary tree structure and every cut in the tree defines a possible classification. To determine the optimal cut of the tree, two criteria are considered : data driven and dissimilarity based.

Clustering process

The bottom-up clustering algorithm is applied to each mixture of Gaussian distributions as follows :

1. Compute distances between each pair of distributions.

2. Merge the closest distributions : Let g 1 (n 1 , µ 1 , Σ 1 ) and g 2 (n 2 , µ 2 , Σ 2 ) be two Gaussians to which n 1 and n 2 frames have been assigned during the training.

If g 1 and g 2 are merged into g 3 (n 3 , µ 3 , Σ 3 ) then :

n 3 = n 1 + n 2 (1) µ 3 = n 1 n 1 + n 2 µ 1 + n 2 n 1 + n 2 µ 2 (2) Σ 3 = n 1 n 1 + n 2 Σ 1 + n 2 n 1 + n 2 Σ 2 + n 1 n 2 (n 1 + n 2 ) 2 (µ 1 -µ 2 )(µ 1 -µ 2 ) T
(3) g 3 replaces g 1 and g 2 in the set whose size is reduced by one [START_REF] Mokbel | Online Adaptation of HMMs to Real Life Conditions: A Unified Framework[END_REF]). 3. If the number of Gaussians is greater than 1 go to the first step.

Metrics

Two distances are used for Gaussian clustering: the existing likelihood lossbased distance [START_REF] Mokbel | Online Adaptation of HMMs to Real Life Conditions: A Unified Framework[END_REF]) and the proposed weighted relative entropy based metric.

-Loss likelihood based metric : If g 1 and g 2 are merged into g 3 then the likelihood loss (P V ) is the difference between the likelihoods of g 1 and g 2 and the likelihood of g 3 :

P V (g 1 , g 2 , g 3 ) = log Σ 3 (n 1 +n 2 )/2 Σ 1 n 1 /2 Σ 2 n 2 /2
This metric is similar to the loss of entropy based distance used by Digalakis in [START_REF] Digalakis | Genones : Generalized Mixture Tying in Continuous Hidden Markov Model-Based Speech Recognizers[END_REF]). It was successfully used for model adaptation in [START_REF] Mokbel | Online Adaptation of HMMs to Real Life Conditions: A Unified Framework[END_REF]).

-The weighted symmetric Kullback-Leibler divergence (KLP ) : it is expressed as the distance between two probability density functions weighted by the amount of training data.

KLP (g 1 ; g 2 ) = 1 2 tr(n 1 Σ 1 Σ 2 + n 2 Σ 2 Σ 1 ) + 1 2 (µ 1 -µ 2 ) T ( n 1 Σ 1 + n 2 Σ 2 )(µ 1 -µ 2 ) -(n 1 + n 2 )d
where d is the dimension of the parameter vectors and tr is the trace. The information provided by the amount of training data is advantageous only if training and testing data have the same proportions.

Tree cutting

From the root of the tree to the leaves, cuts result in many different classifications (ie. codebooks). Three ways of cutting are proposed:

• Fixed : We consider a constant number of classes. So, the tree is traversed from the leaves until the number of nodes reaches the predefined number of classes.

• Weight based : The number of classes depends on the amount of training data in each class. So the tree is processed (from the root) and processing stops at the node whose children's weight is less than a predefined threshold.

• Distance based : The tree cutting is performed when the distance between two levels reaches a maximum value.

For weight and distance criteria, the number of Gaussians per state is variable. Hence, a mean value is computed.

Fig. 2. Gaussian distributions classification

Codewords and Centroids refer to the nodes resulting from cutting the tree at a specified level. A shortlist is a set of tree leaves having a common codeword.

Model shortening

We propose to investigate the use of the previous clustering algorithm in model shortening. The proposed KLP metric (described in section 2.1.2) is also evaluated in the same context and compared to the loss likelihood distance (used in [START_REF] Digalakis | Genones : Generalized Mixture Tying in Continuous Hidden Markov Model-Based Speech Recognizers[END_REF]).

For the experiments we use parameter vectors with 12 MFCC coefficients, en-ergy, and their first and second derivatives. The acoustic models are context independent. They are trained with 82 hours of the Ester train database [START_REF] Galliano | The Ester Phase II Campaign for the Rich Transcription of French Broadcast News[END_REF]). The dictionary contains 118000 words and the language model is formed by 4 millions of bigrams and 4 millions of trigrams. An hour of Broadcast News extracted from the Ester test data set is used for testing.

In order to compare the different systems, a reference system with 32, 64, 80, 128, 180, 220 and 256 Gaussians per state is produced.

Considering the models with 256 Gaussians per state, the clustering algorithm is applied to each Gaussian mixture. Depending on the experiments, either the likelihood loss or the weighted cross entropy based metric is used. The previously explained tree cutting criteria (fixed, weight based or distance based) are also evaluated.

Fixed classes

The same number of classes is used by the reference system (REF ) and by the loss likelihood (P V ) and weighted Kullback-Leibler (KLP ) based systems.

After clustering, the P V and KLP models obtained are trained. We found that two iterations are sufficient for a suitable parameter estimation.Then speech recognition is applied to the REF , P V and KLP systems using models with 32, 64, 80,128 and 180 Gaussians per state. Results within a confidence interval of 1% are as follows: An analysis of the results given in 1 and Fig. 3 shows that :

• Both the P V and the KLP systems outperform the reference one.

• W ER decreases by about approximately 3% compared to the reference system.

Weight based classes

By using this criterion, we ensure that each cluster has sufficient amount of training data for parameter estimation. As the number of Gaussians per state is variable (it depends on the acoustic variability of each state), a mean value is considered. Results are as reported in Table 2 and Fig. 4. We notice KLP outperforms both the P V and the reference system. Especially, with a mean of only 53 Gaussians per state, its performance is close to that of the REF system with 256 Gaussians per state. Besides, the W ER decreases by about 4.8% compared to the REF system using the same number of Gaussians. 28 Gaussians per state on the KLP system perform better than 64 Gaussians in the reference system.

ACCEPTED MANUSCRIPT

Distance based classes

This criterion prevents the clustering of too distant Gaussians. Gaussians are considered distant when their KLP distance is high or when their merging leads to a significant likelihood loss (if P V based metric is employed).

We consider several levels of the tree and cut when the distance between two levels reaches a maximum value. The obtained results are presented in Table 3. Once again, we notice that the P V and KLP systems outperform the REF system, and that the KLP divergence based system has the lowest WER.

Applying the KLP or P V clustering process, we obtain globally the same performance as the REF system using only about 40% of the total number of Gaussians. These results are encouraging but they are not better than the results of the previous experiments (53 Gaussians) in which only 20% of the initial Gaussians were used.

Weight versus distance

The goal of the previous experiments (sections 2.2.1, 2.2.2 and 2.2.3) is to compare the proposed KLP metric to the P V metric and to the REF system. Therefore we fixed the criterion and varied the distance. In this paragraph, the objective is to compare the distance criterion to the weight criterion. So we fixed the distance (KLP or P V ) and plot the curves of the weight and the distance criteria. An analysis of the results given in Fig. 6 and Fig. 7 show that for the KLP system, the weight criterion outperforms the distance criterion, particularly when the number of clusters is low. In the case of P V clustering, the situation is the contrary. These results can be interpreted as follows:

• When the KLP clustering metric is used, no particular attention is given to the amount of training data available for each cluster. Only similar Gaussians are merged, ensuring that at each level, clusters are as distant as pos- sible. therefore, at some levels, several clusters do not have enough training data, and cutting at these levels is of little value.

• In the case of P V based clustering, the loss of likelihood is minimal at each level. Therefore the resulting clusters are as representative as possible of the training data. Given that no information about cluster similarity is taken into account, many resembling clusters may be present at the same level. In this case the distance based cutting criterion is capable of removing redundant information

Conclusion

The clustering algorithm (presented in section 2.1) is investigated in the framework of model shortening.

Experiments show that the KLP distance performs better than the P V distance. When considering the tree cutting criteria, we notice that for the KLP system, the weight criterion outperforms the distance criterion and the for the P V system, the situation is the contrary.

State-based Gaussian selection

In the previous section (2.2) the clustering algorithm (described in section 2.1) is used to shorten the models by clustering the Gaussian distributions without decreasing the system performance. The second application of the clustering (and codebook construction) algorithm is in Gaussian selection. We investigate a likelihood-based, multi-level Gaussian selection.

The overall algorithm operates in two steps : in the first step, Gaussians are organized into a binary tree. Several cuts of the tree are performed. Each level of cut is characterized by its number of codewords.

In the second step (ie. selection) codeword likelihood is computed and sorted.

Only the most likely codewords are considered when descending to the lowest cutting level. When the leaves of the tree are reached, the corresponding Gaussian distributions are sorted by weight and the best of them contribute to the likelihood computation.

The selection algorithm

Selection is applied during the decoding process. For each node of the decoding graph, the goal is to detect Gaussians that dominate the likelihood computation. It operates as follows :

(1) For the current pruning level, codeword likelihoods are computed. Then they are sorted and the most likely are kept before moving down to the lower level of cut.

(2) When reaching the last level of cut, 2 sets of Gaussian distributions may be selected for the likelihood computation : a) leaves whose ancestors have all been retained. b) leaves selected in a) with large weight values.

The following example (Fig. 8) illustrates an application of this algorithm to a mixture of 24 Gaussian distributions. In this case, two levels of cut are considered : level 1 and level 2. As the codeword 31 is the most likely, it is selected. We then move to the next level of cut (level 2) and compute the likelihood of the corresponding nodes that are 25 and 26. If codeword 26 is more likely, the corresponding leaves which are the Gaussians4,5, 6 and 7 are selected. Finally we may decide to compute the likelihood with all Gaussians or to keep only those with the highest weight values. We thus computed a total of nine likelihoods which is less time consuming than 24.

One-level based selection

The reference system of the previous experiments (section 2.2) is considered. The acoustic models contains 512 Gaussian per state. For each state, the 512 Gaussian distributions are organized into a tree structure, then the tree is cut in a specific level. We experimented cutting the tree at the levels 40 and 120 which correspond respectively to 40 and 120 codewords. Performance is measured in terms of Word Error Rate (W ER) and the percentage of likelihood computation C which is defined as :

C = computed likelihoods all likelihoods (4)
For the reference system : W ER = 35.5% and C= 100%

Shortlist scores :

We vary the number of selected codewords (ie. centroids ; see Fig 2 ) and use the corresponding leaves (ie. shortlists) of the tree for the likelihood computation. The number of selected codewords, the corresponding number of Gaussians (X-axis) and W ER (Y-axis) are reported in Fig. 9. As the number of the selected Gaussians is variable, a mean value is considered. The fraction C is also computed and depicted in Fig. 10. An analysis of the results given in Fig. 9 shows that for the same W ER, the 120 codeword system makes use of fewer Gaussians than the 40 codewords one. In particular, with only 23 Gaussians per state (Fig. 9) it gives exactly the same results as the reference system.The same experiments (Fig. 10) show that the value of C is lower for the 40 codeword system. This is because this fraction takes into account the codebook size. The best tradeoff between C and the W ER is obtained by the selection of 2 codewords (as shown by the arrow in Fig. 10). This corresponds to the value pairs (C,WER)=(15.1%,35.6%). In this case the W ER increases by only 0.1% and the likelihood computation cost is reduced by a factor of seven. The acoustic matching duration of the reference system is 3 times the CPU time. This duration is decreased to 0.43 the CPU time.

Data-based selection :

We take the best system of the previous experiments : 40 codewords among which the 2 likeliest are selected. As the training process is based on the Maximum Likelihood criterion, the likely distributions have large weight values. So, to reduce further the number of selected Gaussians, they are sorted by weight and only the Gaussian distributions with highest weights are retained. After varying the number of selected Gaussians, the results given in Fig. 11 are obtained.

The best tradeoff between C and W ER is (12.4%,35,6%). These results are better than those of the previous experiments. Indeed, for the same value of WER (35.6%) the value of C is reduced. In this case, the likelihood computational cost is decreased by a factor of eight.

Bi-level selection

Now the clustering tree is cut simultaneously at two levels (bi-level) of cut.Two experiments are considered :

• using the levels of cut 40 and 60 which correspond to 40 and 60 codewords.

• using the levels of cut 40 and 120 which correspond to 40 and 120 codewords.

Shortlist scores :

In order to further improve the results of the experiments presented in section 2.3.2.1, all densities of level 40 are and the two best codewords are selected. Then we move to the second level of cut (that is 60 or 120). The corresponding codewords are computed and the most likely of them are kept. Finally, the Gaussians for their codewords are used for the likelihood computation. The 40-120 system gives better results than 40-60. This is foreseeable because the level 120 is lower than the level 60 therefore the classification is more precise. The best tradeoff between C and W ER corresponds to the pair of values (C,WER) = (13,2%,35.6%). This result is better than the one presented in section 2.3.2.1 where the tree was cut at a single level but poor than the result using weight values (section 2.3.2.2).

Data-based selection :

We proceed in the same manner as in section 2.3.3.1. The best settings are considered : two levels of cut 40 and 120, and the best pair of values (C,WER)= (13,2%,35.6%). The optimization of the system consists in keeping only the Gaussians with the highest weight values. When varying the number of selected Gaussians, we obtain the results given in Fig. 13. Here, (C,WER)= (35,6%, 12.2%) is the best tradeoff between C and WER.

As the width of the confidence interval is approximately 0.8%, other tradeoffs are also satisfactory. For example, in the case of pair values where (C,WER)= (35,8%, 11.5%) which denotes a decrease in the likelihood computation cost by a factor of nine with a negligible loss of accuracy (+0.3%). We notice that : a-Two levels selection performs better than one-level selection. So increasing the number of levels is advantageous.

Synthesis

b-The selection of Gaussians with high weight values (data-based selection) improves performance. c-The combination of (a) and (b) gives the best results (best tradeoff between WER and C).

Since both the C values and the WER are very close, there is relatively little diffrence between any of the approaches.

Model based clustering and selection

To reduce the likelihood cost in HMM based ASR, two main approaches are generally used : Gaussian selection [START_REF] Pellom | Fast Likelihood Computation Techniques for Nearest-Neighbor Based Search for Continuous Speech Recognition[END_REF]; [START_REF] Lee | Gaussian Mixture Selection using Context Independent HMM[END_REF]; [START_REF] Bocchieri | Vector Quantization for the Efficient Computation of Continuous Density Likelihoods[END_REF]; [START_REF] Ortmanns | Fast Likelihood Computation Methods for Continuous Mixture Densities in Large Vocabulary Speech Recognition[END_REF]; Herman al. ( 1998))and sub-vector quantization. In both cases, classification is performed by clustering/merging Gaussian distributions [START_REF] Bocchieri | Vector Quantization for the Efficient Computation of Continuous Density Likelihoods[END_REF]; [START_REF] Ortmanns | Fast Likelihood Computation Methods for Continuous Mixture Densities in Large Vocabulary Speech Recognition[END_REF]; [START_REF] Gales | State based Gaussian Selection in Large Vocabulary Continuous Speech Recognition using HMMs[END_REF]; [START_REF] Padmanablan | Partitioning the Feature Space of a Classifier with Linear Hyperplanes[END_REF]). So the contextual information is lost and some distributions will be assigned to codewords of different contexts.

Therefore we propose a context-based classification method. The idea is to use Gaussian distributions of context independent models as codewords. Then the clustering algorithm (previously described in section 2.1) is applied as a further improvement to the codebook. Indeed, this process provides a compact and more efficient codebook (see results in section 2.2). Each codebook is tested within the framework of Gaussian selection and subvector quantization. The performance of the proposed Gaussian selection and sub-vector quantization methods are compared to two exiting methods which are evaluated in the same conditions.

Contextual Gaussian selection

Initially, Bocchieri [START_REF] Bocchieri | Vector Quantization for the Efficient Computation of Continuous Density Likelihoods[END_REF]) proposed a Gaussian selection technique by vector quantization. He generates a vector quantized codebook and attributes a shortlist to each codebook entry. During decoding, the frame is assigned to the nearest codeword/shortlist. Gaussians belonging to this shortlist contribute to that frame likelihood computation. Many extensions of this work have been proposed in the literature [START_REF] Gales | State based Gaussian Selection in Large Vocabulary Continuous Speech Recognition using HMMs[END_REF]; [START_REF] Olsen | Gaussian Selection using Multiple Quantisation Indexes[END_REF]; [START_REF] Leppänen | Gaussian Selection with Non-Overlapping Clusters for ASR in Embedded Devices[END_REF]). They were focused on Gaussian codebook assignments. Here we are rather interested in improving codebook construction. The performance of the proposed methods are compared to the "classic Gaussian selection" method (proposed by [START_REF] Bocchieri | Vector Quantization for the Efficient Computation of Continuous Density Likelihoods[END_REF]) and evaluated in the same conditions.

Gaussian distributions mapping

Large vocabulary continuous speech recognition systems need a significant number of Gaussians to model the different contexts. So Context dependent (CD) models are often used and the corresponding systems are generally slow. Small vocabulary systems make use of Context Independent models (CI) because the acoustic variability is limited. Hence, they have the advantage of being fast.

The CD and CI models may have been trained on the same data, though the have quite different capabilities for capturing it. Therefore, we investigate the use of CI Gaussian distributions as a codebook in a CD model based large vocabulary ASR system.

The proposed contextual Gaussian selection method consists in:

(1) Computing all the distances between the CI and the CD Gaussian distributions.

(2) Assigning each CD Gaussian distribution to the nearest CI. In this case the codebook contains the CI Gaussian distributions and the CD distributions assigned to the same codeword (CI Gaussian distribution) form a shortlist.

Hierarchical mapping

The state based clustering algorithm (described in section 2.1) is applied to the CI distributions in order to reduce their length and improve their representation.

Subsequently, mapping table between the CD distributions and the new codebook (CI distributions after the clustering process) is created.

Contextual selection

We developed a large vocabulary speech recognition system based on the Sphinx training and test tools. The acoustic models are cross-word context dependent with 6108 tied states and 32 Gaussians per state. The parameters of these models are estimated on the Ester training database. Tests are conducted using an hour of Broadcast News extracted from the Ester test set. For this reference system all the Gaussian distributions are used for the likelihood computation so C = 100%. W ER = 28.7%.

Context independent models with 32 Gaussians per state and 3 states each were developed to perform contextual Gaussian selection. They have a total of 3456 Gaussian distributions (36*3*32).

After mapping the CD Gaussian distributions to the CI Gaussian distributions the classical Gaussian selection procedure (proposed by [START_REF] Bocchieri | Vector Quantization for the Efficient Computation of Continuous Density Likelihoods[END_REF] and described in section 3.1) was applied to the new codebook (formed by CI Gaussian distributions). This procedure is called Contextual Gaussian selection. The classical Gaussian selection method was also performed on a codebook obtained by Gaussian clustering (the same codebook as in [START_REF] Bocchieri | Vector Quantization for the Efficient Computation of Continuous Density Likelihoods[END_REF]) for comparative reasons. This codebook contains also 3456 Gaussian distributions. Fig. 16 reports the W ER and C corresponding to varying the number of selected Gaussians for classic and contextual Gaussian selection. We can see that contextual performs better than classic Gaussian selection. The best tradeoff between W ER and C corresponds to the pair of values (29.3%,47.02%) and an absolute loss of accuracy of 0.6%.

Hierarchical selection

Hierarchical clustering (described in section 2.1) is applied to CI models with 64 and 128 Gaussians per state to be reduced to 32 Gaussians per state. Subsequently, classic Gaussian selection is performed using the new codebook. From the results in Fig. 17, we notice that the W ER of the models initially with 64 and 128 Gaussians per state are the lowest. Therefore, Gaussian clustering is also advantageous in sub-vector quantization. Typically, with 47% of computed likelihood the loss of accuracy is less than 0.3%.

Synthesis

From the following table we notice that :

• Contextual Gaussian selection performs better than classic Gaussian selection. The initial speech recognition systems runs in 4 * CPU time. When contextual Gaussian selection is the recognition duration is reduced to 3 * CPU time.

In addition, there is some benefit in terms of slightly imroved recognition.

Contextual sub-vector quantization

Recently, sub-vector quantization based methods were proposed as an alternative to the Gaussian selection approach. They have been successfully applied to reduce acoustic model complexity without significant loss of accuracy [START_REF] Mak | Subspace Distribution Clustering Hidden Markov Model[END_REF]; [START_REF] Tsakalidis | Efficient Speech Recognition using Subvector Quantization and Discrete-Mixture HMMs[END_REF]; [START_REF] Mosur | Sub-vector Clustering to Improve Memory and Speed Performance of Acoustic Likelihood Computation[END_REF]). Several methods for codebook construction have been investigated. They are generally based on clustering techniques. For example Mak [START_REF] Mak | Subspace Distribution Clustering Hidden Markov Model[END_REF]) performs per stream Gaussian clustering by means of Battacharya distance. A speech group at Carnegie Mellon University (CMU) employs the k-means algorithm to cluster sub-vectors (means and variances) into a preset number of codebooks [START_REF] Mosur | Sub-vector Clustering to Improve Memory and Speed Performance of Acoustic Likelihood Computation[END_REF], .. As contextual information is lost by clustering, we are interested (in this subsection) in contextual sub-vector quantization. We subsequently investigate the improvement of the codebook by hierarchical clustering. Results are compared to those obtained by the method described in [START_REF] Mosur | Sub-vector Clustering to Improve Memory and Speed Performance of Acoustic Likelihood Computation[END_REF]) which was evaluated in the same conditions.

Stream-based mapping

The contextual sub-vector quantization method is performed in two steps:

(1) the mean and variance vectors of each CI and CD distribution are divided into streams (i.e. subsets of dimensions).

(2) for each stream, the symmetric Kullback-Leibler distances between the CD and CI distributions are computed. (3) each CD distribution is assigned to the closest CI distribution.

By the end of this process, we obtain a per stream mapping table between the CD and CI distributions.

In [START_REF] Mosur | Sub-vector Clustering to Improve Memory and Speed Performance of Acoustic Likelihood Computation[END_REF], the codebook is created by simultaneously clustering the streams of mean and variance vectors. The K-means algorithm is employed for the clustering.

The parameter vectors are composed of (12MFCC + energy) and their first and second derivatives. Three subdivisions of the parameters vector are considered:

• Only one stream of dimension 39.

• Three streams: (12MFCC+energy) + ∆(12MFCC+energy)+ ∆ ∆(12MFCC+energy).

• Four streams : (energy,∆energy,∆ ∆energy) + (12 MFCC) + (12 ∆(MFCC))

+ (12 ∆ ∆ (MFCC)).

In the following, the performance of the contextual sub-vector quantization method are compared to those of the initial system which are (W ER, C) = (28.7%, 100%) and to those of the existing sub-vector quantization method (described in [START_REF] Mosur | Sub-vector Clustering to Improve Memory and Speed Performance of Acoustic Likelihood Computation[END_REF]).

Contextual Sub-Vector Quantization

For the contextual sub-vector quantization (CSVQ), 36 CI models with 32 Gaussians per state (a total of 108 states) are used for the codebook. The mapping is performed by state (CSVQ-s), by phone (CSVQ-p) or using all the CI distributions (CSVQ-a).

• CSVQ-s : the Gaussian distributions of each monophone state constitute a codebook for the corresponding triphone states.

• CSVQ-p : the Gaussian distributions of a monophone constitute a codebook for the corresponding triphones.

• CSVQ-a : the Gaussians distributions of all the monophones constitute a codebook for all the triphones.

During the decoding process, the likelihood is computed with the CI distributions and the corresponding CD distribution weights.

Fig. 18 reports the W ER according to the computation fraction C for the methods SV Q (the method described in [START_REF] Mosur | Sub-vector Clustering to Improve Memory and Speed Performance of Acoustic Likelihood Computation[END_REF]), CSV Qs, CSV Qp and CSV Qa and for the three sizes of stream (1, 3 and 4). Several observations can be made :

• Multi-stream based methods perform better than one-stream based methods. Indeed, the quantification distorsion (ie. error due to assigning (sub)vectors to codewords) is less important. • The increase in W ER generated by the SV Q, CSV Qs and CSV Qp methods exceeds 1%. From our point of view, this is due to the loss of information about context in the SV Q method. For the CSV Qs and CSV Qp approaches, we can say that the distributions of the CI states and phones are unable to represent all the corresponding CD distributions.

• The best results are obtained by means of the CSVQ-a method. We point out that this method makes use of a codebook formed by all of the CI distributions. The optimal configuration corresponds to the pair of values (W ER, C) = (29.0%, 13.6%), i.e. the likelihood computation fraction is reduced to 13.6% with a small increase in the W ER (+0.3% absolute).

The initial speech recognition systems runs in 4 * CPU time. When CSVQ-a method is performed, this duration is reduced to 2.8 * CPU time.

Hierarchical Sub-Vector Quantization

To improve the results of the CSV Qa method, we applied the clustering algorithm to the CI Gaussian distributions. The length of the CSV Qa codebook is 3456 (ie. 32 Gaussians * 108 states). It was reduced to 540 (ie.

5 Gaussians * 108 states). CSV Qh refers to the new system with a total of 540 Gaussians. xtures. To compare the results of these experiments to the previous ones, we use the same stream definitions. From the results in Fig. 19, we can deduce that :

• CSVQ-h method outperforms SVQ-a.

• Using only one stream is of little interest (the WER increase exceeds 0.9% absolute). • The CSVQ-h method produces an interesting point (WER,C)=(28.8%,17.89%) which compares well with the initial system. In addition, the WER is inside the confidence interval and thus we can conclude that about 17% of densities are computed with no loss of accuracy.

Conclusion

In order to reduce the computation cost in low-resource and large application mobile devices several Gaussian selection and sub-vector quantization techniques were investigated. These methods are based on original and efficient codebooks and operate either at the state or at the model level. Two kinds of codebook have been proposed. The first is formed by context independent Gaussian distributions. The second results from clustering context independent Gaussian distributions. The weighted and symmetric Kullback-Leibler distance is used for the clustering. The evaluation of the codebooks and the selection methods has been conducted within the framework of a large vocabulary continuous speech recognition task. Experiments shows that by using the first codebook, the WER decreases by 3% absolute. For the same WER the likelihood computation cost is reduced to 17%.

As a perspective, we propose to combine these Gaussian selection and subvector quantization techniques for a further improvement of performance.

Fig. 1 .

 1 Fig. 1. Hierarchical clustering of Gaussian distributions.

Fig. 3 .

 3 Fig. 3. W ER vs number of Gaussians : Fixed classes for REF , P V and KLP systems

  Fig. 4. W ER vs number of Gaussians : Weight based classes for P V and KLP systems and fixed classes for REF system.

  Fig. 5. W ER vs number of Gaussians : Distance based classes for P V and KLP systems and fixed classes for REF system.

Fig. 6 .

 6 Fig. 6. KLP system : Weight and distance based tree cutting

  Fig. 7. P V system : Weight and distance based tree cutting

Fig. 8 .

 8 Fig. 8. Bi-level Gaussian selection example

Fig. 9 .

 9 Fig. 9. Computing the likelihood using the best shortlists. The number in this figure correspond to the number of codewords.

Fig. 11 .

 11 Fig. 11. Computing likelihood using the highest weight Gaussians. The arrow shows the best tradeoff between C and the W ER.

Fig. 12 .

 12 Fig. 12. Computing likelihood using two levels of cut and the best shortlists.

Fig. 13 .

 13 Fig. 13. Computing likelihood using the hightest weighted Gaussians.

Fig. 14 .

 14 Fig. 14. Assignment of the CD Gaussian distributions to the CI Gaussian distributions.

Fig. 15 .

 15 Fig. 15. Mapping between the new CI et CD Gaussian distributions. The new CI Gaussian distributions are obtained by clustering the initial CI Gaussian distributions.

  Fig. 16. W ER vs C for classic and contextual Gaussian selection

Fig. 17 .

 17 Fig. 17. W ER vs C for classic, contextual and hierarchical Gaussian selection. The arrow shows the best tradeoff between C and W ER.

Fig. 18 .

 18 Fig. 18. The performance of the SVQ and CSVQ methods for the three streams

Fig. 19 .

 19 Fig.19. The SVQ-a and CSVQ-h results for the three streams

Table 1 Fixed

 1

	32	42.6	40.6	39.5
	64	40.4	38.0	37.5
	80	38.3	37.4	36.9
	128	37.3	36.2	36.2
	180	36.4	36.1	36.2
	220	36.3	35.8	35.5
	256	36.3	-	-
	512	35.5	-	-

classes : W ER for REF , P V , and KLP systems Number of Gaussians REF (%) P V (%) KLP (%)

Table 2 Weight

 2

	KLP	28	40.0
		53	36.6
		150	35.9
		195	36.0
	PV	53	39.5
		101	36.8
		156	36.5

based classes : W ER for P V and KLP systems Metric Number of Gaussians W ER (%)

Table 3 Distance

 3

	KLP	30	40.7
		59	37.7
		101	36.1
		196	35.9
	PV	44	39.4
		94	36.7
		204	35.8

based classes : W ER for the P V and KLP systems Metric Number of Gaussians W ER (%)

  • Hierarchical Gaussian selection gives the best results.

	Method	C (%) ∆ (WER) (%)
	Classic Gaussian selection	50.7	+0.9
	Contextual Gaussian selection	47.0	+0.6
	Hierarchical Gaussian selection	47.5	+0.3
	Table 5
	Model based Gaussian selection results

Acknowledgments

The authors would like to thank Peter Weyer-brown, Kevin McTait and Hemant Misra for their corrections of English orthography and syntax.


Arthur Charpentier

Emilios C C Galariotis

Christophe Villa

Category-based Tail Comovement

Keywords:

published or not. The documents may come

Introduction

During the unfolding …nancial crisis triggered by the US subprime loan failure, stocks have experienced both extreme movement and comovement in returns. While comovement between asset returns has been generally measured using Pearson's correlation, extreme comovement has been addressed using various statistical measures. For example, using multivariate extreme value theory (EVT), [START_REF] Longin | Extreme correlations in international Equity Markets[END_REF] derive extreme correlations for various equity return distributions internationally. Hartmann, Straetmans and de Vries (2004) rely on tail dependence for bivariate linkages between equity and government bond markets in the G-5 industrial countries during market turmoil. While the abovementioned papers and the extant literature implicitly assume asymptotic dependence because of the use of EVT, Poon, Rockinger and Tawn (2004) point out that this is a very simplifying assumption that can lead to estimation errors. They derive a general multivariate framework with two types of extreme value dependence structures that allow for both asymptotic dependence and independence. More speci…cally they make a clear distinction between the four types of dependence: perfect dependent, independent, asymptotically dependent and asymptotically independent, claiming that "for positively related and asymptotically dependent (independent) variables, large values of each variable will occur simultaneously more often (less often) than if the variables are independent (perfectly dependent)." 1 Most papers in this strand of the literature focus on the statistical measurement of extreme comovement and not on explaining the observed patterns based on a proposition that involves a …nancial model/theory. Assuming a frictionless economy and rational investors, traditional …nancial theory maintains that comovement (extreme comovement) in prices is due to comovement (extreme comovement) in fundamental values. Hence, the reason why the stock prices of a given industry comove is that their earnings and hence their intrinsic values, are related, and when one industry constituent reports good …nancial results, it is rather likely that other …rms in the same industry will too. There is little doubt that under the aforementioned assumptions the traditional fundamentals-based view is able to explain standard patterns of comovement. Nonetheless, in the presence of irrational investors, market imperfections and limits to arbitrage, asset prices and fundamental values become disentangled, rendering traditional theory ine¤ective and calling for propositions based on behavioral theories of comovement such as investor sentiment. According to [START_REF] Barberis | Style investing[END_REF] investors group information into categories, based on some common characteristic observable across all assets that belong to the same category. Such an approach simpli…es the investment process, narrowing down the possibilities and making it much easier to allocate funds across a few categories than among numerous assets. The manifestation of this in …nancial markets can be found in style investment, where investors group stocks into styles based on some common characteristic that can be related or unrelated to fundamentals and allocate funds at the category level rather than the individual one. Styles are complemented by counterstyles, for example: large cap versus small cap, growth versus value etc. and investors move funds from style to counter style in anticipation of better investment opportunities, that is, styles become in and out-of-favor. If some investors are noise traders with correlated sentiment operating in a market with limited arbitrage, then as they shift funds between categories, they will induce comovement in prices that is not related to fundamentals. For example, if noise traders become irrationally enthusiastic about internet stocks and channel funds into that category driven by sentiment, the common buying pressure will lead to price comovement for such stocks even if their fundamental values are unrelated. [START_REF] Cornell | Comovement as an investment tool[END_REF] used Yahoo and Amazon to illustrate sentiment based comovement related to category fund allocation for two …rms with di¤erent fundamentals and businesses. A plethora of studies provide evidence that sentiment is correlated across investors and in ‡uences comovement in stock prices (see [START_REF] Baker | Investor Sentiment and the Cross-Section of Stock Returns[END_REF]. [START_REF] Barberis | Comovement[END_REF] provide evidence of style-based comovement associated with the inclusion of stocks in aggregate indices. Finally, [START_REF] Kumar | Retail Investor Sentiment and Return Comovements[END_REF] have shown that systematic trading by retail investors could lead to stock return comovements beyond the usual risk factors. This paper complements the previously mentioned literature. More speci…cally, while the extant literature so far aimed at explaining comovement in normal market states as captured by standard correlation measures, we examine extreme comovement such as the one that can be observed in …nancial crises. To capture such extreme comovement we employ the two types of tail dependence proposed by [START_REF] Poon | Extreme Value Dependence in Financial Markets: Diagnostics, Models, and Financial Implications[END_REF]. More importantly, we take literature further by extending the propositions made by [START_REF] Barberis | Style investing[END_REF]. Simply speaking, Barberis and Shleifer propose a sentiment based theory of comovement, while we, on the other hand, propose a sentiment based theory of extreme comovement. This is motivated by the fact that during …nancial crises, sentiment based investment gains prominence, and the tails of the return distribution are gaining importance too. Our contribution can be viewed as complementary to the one of Barberis and Shleifer, as it extends their proposition so as to cover the full spectrum of dependence. More precisely three main contributions have been derived: 1) we formally prove that the category-based comovement model proposed by [START_REF] Barberis | Style investing[END_REF] implies that returns are asymptotically independent; 2) we formally prove that the category-based comovement model proposed by [START_REF] Barberis | Style investing[END_REF] is also a category-based weak tail-comovement model: in this economy, assets in the same category comove too much in the tail and reclassifying an asset into a new category raises its (weak) tail dependence with that category; 3) we emphasize on the crucial role of the distribution of noise trader sentiment in order to explain both positive correlation and (strong) tail comovement in asset returns. More precisely we show that if the distribution of the change in noise trader sentiment has heavy tails, then assets in the same category comove too much in the tail since tail comonotonicity is found for such assets; and reclassifying an asset into a new category raises its (strong) tail dependence with that category.

The outline of the rest of the paper is as follows. Section 2 introduces . Section 3 details our . In Section ?, we . Section ? concludes and provides directions for future research.

Tail Comovement Measures

Quantile dependence is a measure of the dependence in the tails of the distribution. If Z 1 and Z 2 are random variables with distribution functions F 1 and F 2 , then there is quantile dependence in the lower tail at threshold , whenever

P Z 2 F 1 2 ( ) j Z 1 F 1 1 ( )
is di¤erent from zero while there is quantile dependence in the upper tail at threshold , whenever

P Z 2 F 1 2 ( ) j Z 1 F 1 1 ( )
is di¤erent from zero. Tail dependence obtains as the limit of this probability, as we go arbitrarily far out into the tails. De…nitions of tail dependence for multivariate random vectors are mostly related to their bivariate marginal distribution functions. Loosely speaking, tail dependence describes the limiting proportion that one margin exceeds a certain threshold given that the other margin has already exceeded that threshold. The following approach, as provided in the monograph of Joe (1997), represents one of many possible de…nitions of tail dependence.

The upper tail dependence index between Z 1 and Z 2 is de…ned as

U = lim !1 P(Z 1 F 1 1 ( ) j Z 2 F 1 2 ( )) = lim !1 P(Z 2 F 1 2 ( ) j Z 1 F 1 1 ( ));
and the lower tail dependence index is

L = lim !0 P(Z 1 F 1 1 ( ) j Z 2 F 1 2 ( )) = lim !0 P(Z 2 F 1 2 ( ) j Z 1 F 1 1 ( )):
where 0 1, we have that variables are termed asymptotically dependent if > 0 and asymptotically independent if = 0. However, as pointed out by Poon, Rockinger and Tawn (2004), generally, when = 0 the two random variables are not necessarily exactly independent. Coles, He¤ernan and Tawn (1999) have provided a range of extremal dependence models, derived from a di¤erent form of multivariate limit theory, that describe dependence but have = 0. Although the random variables are asymptotically independent in this case, di¤erent degrees of dependence are attainable at …nite levels of . To this end, they introduce another tail dependence index 2 .

The upper tail dependence index between Z 1 and Z 2 is de…ned as U 2 (0; 1] where

U = lim !1 log(1 ) log P(Z 1 > F 1 1 ( ); Z 2 > F 1 2 ( ))
; while the lower tail dependence index between Z 1 and Z 2 is de…ned as L 2 (0; 1] where

L = lim !0 log( ) log P(Z 1 F 1 1 ( ); Z 2 F 1 2 ( ))
:

Values of > 0, = 1 2 and < 1 loosely correspond respectively to when Z 1 and Z 2 are positively associated in the extremes, exactly independent, and negatively associated. = 0 and 2 (0; 1) signi…es asymptotic independence, in which case the value of determines the strength of dependence within this class (also known as dependence in independence).

The intuition of this coe¢ cient is the following. If Z 1 and Z 2 are independent (in tails), for large enough

P(Z 1 > F 1 1 ( ); Z 2 > F 1 2 ( )) = P(Z 1 > F 1 1 ( )) P(Z 2 > F 1 2 ( )) = (1 ) 2 ;
or equivalently,

log P(Z 1 > F 1 1 ( ); Z 2 > F 1 2 ( )) = 2 log(1 );
while if Z 1 are comonotonic (in tails), for large enough

P(Z 1 > F 1 1 ( ); Z 2 > F 1 2 ( )) = P(Z 1 > F 1 1 ( )) = 1 ;
or equivalently,

log P(Z 1 > F 1 1 ( ); Z 2 > F 1 2 ( )) = log(1 ):
Thus, the limit of the ratio log 1) can be seen as a tail dependence measure, with tail independence when the limit is 2 and tail independence when the limit is 1. In order to have an index increasing with the strength of the dependence, it becomes more natural to consider the inverse of that ratio.

P (Z 1 > F 1 1 ( ); Z 2 > F 1 2 ( ))= log(

Tail Comovement in a Category-based Comovement Model

There are a number of interesting stock market patterns that don't …t neatly into the traditional fundamental view of comovement. One example concerns so-called "twin stocks," which are stocks that are claims to the same cash- ‡ow stream, but are primarily traded in di¤erent locations. The best-known example is Royal Dutch and Shell. They used to be completely independent companies, but in 1907 they agreed to merge their interests while remaining separate entities. Today, shares of Royal Dutch are traded primarily in the United States and in the Netherlands, and are a claim to 60 percent of the combined …rm's cash ‡ow, while Shell shares, traded primarily in the United Kingdom, are a claim to the remaining 40 percent. Since the two shares are claims to exactly the same cash- ‡ow stream, the fundamentals-based view of comovement argues that the prices of the two shares should move in lock-step with one another. In reality, the two stocks seem to have minds of their own-Royal Dutch moves closely with the S&P index, while Shell's movements are closely tied to those of the FTSE index of U.K. stocks. A related example concerns closed-end country funds, whose assets are traded in a di¤erent location from the funds themselves. For example, there are closed-end funds invested entirely in German equities, but whose shares trade primarily in New York. Since a closed-end fund and the assets it holds are claims to very similar cash- ‡ow streams, the price of the closed-end fund and the value of its holdings should move together very closely. However, this is not often the case.

Closed-end country funds tend to move more closely with the national market where they are traded than with the national market where their holdings are traded. In this example, a Germany fund would tend to move more closely with the U.S. market even though all its holdings are German equities. Finally, there is strong evidence that small-cap stocks tend to move together, as do value stocks.

The framework developed by [START_REF] Barberis | Style investing[END_REF] for analyzing trading based comovement is straightforward. Consider an economy that contains a riskless asset in perfectly elastic supply and with zero rate of return, and also 2n risky assets in …xed supply. Risky asset i is a claim to a single liquidating dividend D i;T to be paid at some later time T . This eventual dividend equals D i;T = D i;0 + e i;1 + ::: + e i;T where D i;0 and e i;t are announced at time 0 and time t, respectively, and where e t = (e 1;t ; :::; e 2n;t ) 0 ~N (0; D ) ; i:i:d: over time [START_REF] Barberis | Style investing[END_REF] assume that the cash- ‡ow covariance matrix D takes a speci…c form, although the predictions also hold for more general structures. In particular, we suppose that the cash- ‡ow shock to an asset has three components: a marketwide cash- ‡ow shock, a group-speci…c cash- ‡ow shock that a¤ects assets in one group but not the other, and a completely idiosyncratic cash- ‡ow shock speci…c to the asset. Formally, for i 2 X,

e i;t = M f M;t + S f X;t + q 1 2 S 2 M " i;t
and for j 2 Y ,

e j;t = M f M;t + S f Y;t + q 1 2 S 2 M " j;t
where f M;t is the market-wide shock, f X;t and f Y;t are the group-speci…c shocks, and " i;t and " j;t are idiosyncratic shocks; M and S are constants that control the relative importance of the three components. Each shock has unit variance and is orthogonal to the other shocks. The price of a share of risky asset i at time t is P i;t and the return on the asset between time t 1 and time t is

P i;t = P i;t P i;t 1
The innovation introduced by [START_REF] Barberis | Style investing[END_REF] is to assume that noise traders are attracted to certain groups of assets and that they allocate their funds across those groups rather than at the level of individual assets. For instance, value stocks, small cap stocks and technology stocks could be examples of such groups. If arbitrage is limited, change in noise trader sentiment regarding any one group will lead to price movements that push prices for that group of assets away from their fundamental value. However, this movement, and the subsequent return to fundamental value, are common across all assets in the group. Suppose that, to simplify their decision-making, some investors group the 2n risky assets into two categories, X and Y , and then allocate funds at the level of these categories rather than at the individual asset level. In particular, they place assets 1 through n in category X and assets n+1 through 2n in category Y . A simple representation for asset returns is then Here, u X;t can be thought of as time t noise trader sentiment about the securities in category X. Since the noise traders allocate funds by category, this sentiment level is the same for all securities in category X. The return on a security in category X is a¤ected not only by news about cash ‡ows, e i;t , but also by the change in sentiment about X, u X;t : when noise traders become more bullish about old economy stocks, these stocks go up in price. This model can also be thought of as a reduced-form model for the habitat view of comovement. In this case, X and Y simply have to be reinterpreted as habitats, not categories: instead of representing groups of assets that some investors do not distinguish between when allocating funds, they represent groups of assets that are the sole holdings of some investors. Speci…cally, we can think of assets 1 through n as U.S. stocks and assets n + 1 through 2n as U.K. stocks; there are many investors in both countries who trade only domestic securities. Under the habitat interpretation, u X;t tracks the risk aversion, sentiment, or liquidity needs of investors who invest only in the securities in X. The return of an asset in habitat X is a¤ected not only by news about cash ‡ows but also by the change in risk aversion, say, of these speci…c investors.

P i;
In the propositions 1, 2 and 3, we will assume that

u X;t u Y;t ! N 0 0 ! ; 2 u 1 u u 1 !! i:i:d: over time Proposistion 1.
Under the category-based comovement model, for two assets i and j we have ( P i ; P j ) = (e i ; e j ) = 0 Proposistion 2. Under the category-based comovement model, if two assets i and j, i 6 = j, belong to the same category, i; j 2 X or Y , then ( P i;t ; P j;t ) > (e i;t ; e j;t ) Proposistion 3. Under the category-based comovement model, suppose that asset j, previously a member of category Y , is reclassi…ed as belonging to X. Then ( P j;t ; P X;t ) increases after j is added to category X where P X;t = 1 n P l2X P l;t .

The intuition is straightforward, whether X and Y are categories, habitats, or groups of stocks that incorporate information at similar rates. When asset j enters category X, it is bu¤eted by noise traders' ‡ows of funds in and out of that category.

In the propositions 4 and 5, we will assume that u X has heavy tails. Proposition 4. Under the category-based comovement model, if two assets i and j, i 6 = j, belong to the same category, i; j 2 X or Y , then 1. if u X has right-heavy tails 1 = U ( P i;t ; P j;t ) > U (e i;t ; e j;t ) = 0 2. if u X has left-heavy tails 1 = L ( P i;t ; P j;t ) > L (e i;t ; e j;t ) = 0

We thus observe that even if cash- ‡ow shocks are tail independent, P i and P j are comonotonic in tails. Indeed if = 1, there is tail comonotonicity. 3Proposition 5. Under the category-based comovement model, suppose that asset j, previously a member of category Y , is reclassi…ed as belonging to X. Then 1. if u X has right-heavy tails, U ( P j;t ; P X;t ) increases after j is added to category X where P X;t = 1 n P l2X P l;t .

2. if u X has left-heavy tails, L ( P j;t ; P X;t ) increases after j is added to category X where P X;t = 1 n P l2X P l;t .

Conclusion

Traditional …nancial theory predicts that comovement in asset returns is due to fundamentals. An alternative view is that of Barberis and Shelifer (2003) and Barberis, Shelifer and Wurgler (2005) who propose a sentiment based theory of comovement, delinking it from fundamentals. In their paper they view comovement under the prism of the standard Pearson's correlation measure, implicitly excluding extreme market events, such as the latest …nancial crisis. Poon, Rockinger and Tawn (2004) have shown that under such events di¤erent types of comovement or dependence may co-exist, and make a clear distinction between the four types of dependence: perfect dependent, independent, asymptotically dependent and asymptotically independent. In this paper we extend the sentiment based theory of comovement so as to cover the whole spectrum of dependence, including extreme comovement such as the one that can be observed in …nancial crises. One of the key contributions of this paper is that it formally proves that assets belonging to the same category comove too much in the tail and reclassifying an asset into a new category raises its tail dependence with that category.

Annex A

Proof of Proposition 1: Since e = (e i;t ; e j;t ) is a Gaussian random vector, independent of u X;t which is also Gaussian, P = e + u X is also a Gaussian random vector: since " i and u X are independent. Strong tail dependence is null for non comonotonic Gaussian random vectors, i.e.

( P ) = ( 0 if corr( P ) 2 [ 1; +1); 1 if corr( P ) = +1:
thus, ( P i;t ; P j;t ) = 0 = (e i;t ; e j;t ). This …nishes the proof of Proposition 1.

We now prove the following lemma, which will be useful in the proofs 2 and 3.

Lemma. If X = (X 1 ; X 2 ) has a Gaussian distribution with correlation , then U = L = (1 + )=2 .

Proof of Lemma. U is the upper tail dependence index of (X 1 ; X 2 ) if for some function h de…ned on [0; 1] such that h(1=2) = 1.

P(Z 1 > z 1 ; Z 2 > Z 2 ) [z 1 z 2 ] 1=2 L(z 1 ; z 2 ) if Z
The expression of the weak tail dependence index for Gaussian random vectors can be obtain heuristically as follows. For the bivariate normal distribution, an asymptotic development of Mills ratio gives

P(X > x; Y > y) (x; y; ) (1 2 ) 2 (x y)(y x) :
Set U = 1= log (X), V = 1= log (Y ), and u = 1= log (x), v = 1= log (y) (so that U and V have unit Fréchet distributions),

P(U > u; V > v) ( 1 (e u 1 ); 1 (e v 1 ); ) (1 2 ) 2 (x y)(y x) since x =
1 (e u 1 ) and y = 1 (e v 1 ). A quick limit development yields

x 1 (1 u 1 ) (2 log u) 1=2 ;
thus

( 1 (e u 1 ); 1 (e v 1 ); ) ((2 log u) 1=2 ; (2 log v) 1=2 ; ) / exp (2 log u) + 2 p (2 log u)(2 log v) + (2 log v) 2(1 + 2 ) ! = [uv] 1=(1+ 2 ) exp 2 p (2 log u)(2 log v) 2(1 + 2 )
! where function on the right is slowly varying.

Thus

P(U > u; V > v) = [uv] 1=(1+ 2 ) L(u; v);
where it can be proved that L(u; v) is a slowly varying function. >From the de…nition of the power value is necessarily 1=2 , and thus, for a Gaussian random vector

= 1 + 2 :
Proof of Propositions 2.Since e = (e i;t ; e j;t ) is a Gaussian random vector, independent of u X;t which is also Gaussian, P = e + u X is also a Gaussian random vector. From the above lemma, U ( P i;t ; P j;t ) = L ( P i;t ; P j;t ) = 1 + corr( P i;t ; P j;t ) 2 for i; j 2 X Here, cov( P i ; P j ) = cov(e i ; e j ) + 2 u and var( P i ) = var(e i ) + 2 u :

The proposition therefore follows if.

cov(e i ; e j ) < var(e i )

Using

e i;t = M f M;t + S f X;t + q 1 2 S 2 M " i;t it is easily checked that cov(e i ; e j ) = 2 S + 2 M < 1 var (e i ) = 1
which means that inequality corr( P ) > corr(") does indeed hold and thus following the above lemme that the proposition also hold.

Function h is said to be regularly varying at

1 with index 2 R, denoted h 2 RV 1 if lim t!1 h(tx) h(t)
= x ; for all x > 0:

If = 0,
)) we will say that Z has right heavy tails if its distribution is in the max-domain of attraction of the Fréchet distribution, i.e. F Z 2 RV +1 with 2 (0; 1). But unfortunately, rapid variation is not su¢ cient to characterize light tails. We will say that Z has right light tails if its distribution is in the max-domain of attraction of the Gumbel distribution. A necessary condition is that F Z 2 RV +1 1 . A su¢ cient condition is that (1=h(x)) 0 ! 0 as x ! +1 where h denotes the hazard rate of Z, i.e. h(x) = f Z (x)=F Z (x). For instance, the Student t distribution has heavy tails (with degrees of freedom equal to the tail index ), while the Gaussian distribution has light tails.

FINIR DE REFERENCER LES LOIS CLASSIQUES

Power laws For regularly varying distributions, far out in the tail t ! 1 the distribution behaves like a Pareto distribution. For power laws, distributions have exact Pareto tails.

-stable laws -stable (or Lévy) distributions with in…nite variance 2 (0; 2) have heavy tails. They appear naturally when studying sum of random variables.

Elliptical distributions Tails of standard elliptical distributions can be simply characterized. For instance, the Gaussian distribution has light tails, while the Student t distribution has heavy tails, its tail regular variation index is the number of degrees of freedom (????).

Markov switching processes Consider a random coe¢ cient autoregressive model, e.g. X t = n X t 1 + " t , where ( n ) is a series of random variables. Then X t has heavy tails (see e.g. section 8.4.3 in Embrechts, Kluppelberg & Mikosh (1997)).

GARCH processes ARCH and GARCH processes (with a Gaussian noise) are heavy tailed (see e.g. section 8.4.3 in Embrechts, Kluppelberg & Mikosh (1997) for ARCH(1) processes). Some distributions have heavier tails than the Gaussian distribution, but will not necessarily be called heavy tailed. nonexistence of exponential moments A …rst class of distribution with heavier tails than the Gaussian distribution is the class of distributions such that E(e X ) = 1. Then tail probability P(X > x) declines faster than exponentially. Subexponential distribution A famous class of heavy tailed distribution is obtained when the sum of n random variables is likely to be large if and only if their maximum is likely to be large, i.e.

lim

x!1 P(X 1 + + X n > x) P(maxfX 1 ;

; X n g > x) = 1:

Here the tails decrease more slowly than any exponential distribution.

Proof of Propositions 3.Let P X;t = 1 n P l2X P l;t , and assume that ( u X;t ; u Y;t ) is a Gaussian vector, then ( P j;t ; P X;t ) is a Gaussian vector, for any j (j 2 X or j 2 Y ). Suppose that asset n + 1 is reclassi…ed from style Y into style X, and that at the same time, asset 1 is reclassi…ed from style X into style Y . Berfore reclassi…cation, we have

cov( P X ; P n+1 ) = 2 M + 2 u 2 u
and after cov( P X ; P n+1 ) = 2 M + 2 S + 2 u Therefore, cov( P X ; P n+1 ) does indeed increase after addition.Thus corr( P X ; P n+1 ) does also increase after addition and thus following the above lemme that the proposition also hold..

  [START_REF] Ang | Asymmetric Correlations of Equity Portfolios[END_REF] and Z 2 have unit Fréchet distributions, and where L is bivariate slowly varying function, i.e.

	lim t!1	L(tz 1 ; tz 2 ) L(t; t)	= h	z 1 z 1 + z 2	;

  A random variable Z is said to be right regularly varying with index 2 R if F Z 2 RV +1 , and left regularly varying with index 2 R if F Z 2 RV 1 . From classical results on extreme values (see e.g. Embrechts et al. (

then h is said to be slowly varying. If = +1, then h is said to be rapidly varying at 1, i.e. lim t!1 h(tx) h(t) = 0 for all x 2 (0; 1) and lim t!1 h(tx) h(t) = 1 for all x > 1.

[START_REF] Bae | New Approach to Measuring Financial Contagion[END_REF] focus on analyzing the joint occurrences of extreme events using a multinomial logistic model in order to study the propagation of large-return shocks within and across regions.

In Poon, Rockinger & Tawn (2002), is denoted , while is (1 + )=2. This weak dependence function is closely related to bivariate regular variation (as de…ned in Resnick (?)).

Z 1 and Z 2 are said to be comonotonic if there exists ' strictly increasing such that Z 2 = '(Z 1 ). Then one variable increases i¤ the other increases with probability one. If Z 1 and Z 2 are …nite variances, then the correlation between between Z 1 and Z 2 exists, and it is maximal when Z 1 and Z 2 are comonotonic (so called Hoe¤ding upper bound for correlation).

Proof of Proposition 4: A proof will be given here only for right tail, and upper tail dependence, since the proof remains mainly unchanged for left tail and lower tail dependence.

f u X (u) F e i ;e j F 1 P i (t) u; F 1 P j (t) u du:

(1) With a change of variable u = F 1 u X (t)x, we get

Let g t denote the integrated function, so that

Recall that

:

For the …rst term, note that

and therefore

>From Karamata's theory (see e.g. Proposition 1.5.8 in Bingham et al. (1987), also called Von Mises'conditions), since u X is right regularly varying with index 2 (0; 1), this term is simply

For the second term, t) . Since " and u X are independent, the quantile function of " + u X is lower than the quantile function of " + u X in the case of comonotonicity, for u large enough. Hence, there is t 0 < 1 such that for all t 2 (t 0 ; 1),

where the + exponent is for comonotonic pairs, while ? denotes independent pairs. The last equality is obtained from the property of additivity of the quantile function for comonotonic variables. Thus,

:

Since u X has heavier tails than " i , then (see Proposition 1.5.7 in Bingham et al. and Proposition VIII.8.1 in Feller (1971)),

Hence,

Thus, ( for all x > 1; lim t!1

; and therefore

Moreover since e i are gaussian U (e i ; e j ) = 0; we have U ( P i ; P j ) > U (e i ; e j ):

Proof of Proposition 5. It is possible to rewrite P X;t as follows

where e X;t is normal (as a weighted sum of components of a Gaussian vector) and independent of u X;t (since e ;t is independent of u X;t ).

So the proof of the previous Proposition can be used to prove that ( P X;t ; P n+1;t ) is equal to one after reclassi…cation while ( P X;t ; P n+1;t ) was zero before reclassi…cation. Thus the proposition hold.