Badr Abdullah

Irina Illina

Dominique Fohr

DYNAMIC EXTENSION OF ASR LEXICON USING WIKIPEDIA DATA

Keywords: Automatic speech recognition, out-ofvocabulary words, word embedding, lexicon extension

Despite recent progress in developing Large Vocabulary Continuous Speech Recognition Systems (LVCSR), these systems suffer from Out-Of-Vocabulary words (OOV). In many cases, the OOV words are Proper Nouns (PNs). The correct recognition of PNs is essential for broadcast news, audio indexing, etc. In this article, we address the problem of OOV PN retrieval in the framework of broadcast news LVCSR. We focused on dynamic (document dependent) extension of LVCSR lexicon. To retrieve relevant OOV PNs, we propose to use a very large multipurpose text corpus: Wikipedia. This corpus contains a huge number of PNs. These PNs are grouped in semantically similar classes using word embedding. We use a two-step approach: first, we select OOV PN pertinent classes with a multi-class Deep Neural Network (DNN). Secondly, we rank the OOVs of the selected classes. The experiments on French broadcast news show that the Bi-GRU model outperforms other studied models. Speech recognition experiments demonstrate the effectiveness of the proposed methodology.

INTRODUCTION

Automatic speech recognition (ASR) systems usually operate on a fixed lexicon and cannot correctly transcribe words that are not in the lexicon. For a system that must recognize speech about current events, this can pose a problem, because new words are continually introduced based on the occurring events. A particular issue is proper nouns (PNs): the names of important people or locations. Recognizing them can be vital to understand the occurring events. Context-independent extension of the LVCSR lexicon based on word frequency statistics does not necessarily improve the LVCSR performance because missing OOV PNs can be rare words and thus are difficult to retrieve.

Methods for OOV retrieval can be categorized in OOV detection approaches and vocabulary selection approaches. Detection based approaches [START_REF] Qin | OOV word detection using hybrid models with mixed types of fragments[END_REF], [START_REF] Chen | Variablespan out-of-vocabulary named entity detection[END_REF] aim to detect/locate OOV words in the LVCSR hypothesis, followed by a search for the matching OOV word. They need careful selection of sub-word units and LM estimation. Recently, in the acousticto-word model based on the Connectionist Temporal Classification (CTC) of [START_REF] Li | Acoustic-To-Word Model Without OOV[END_REF], OOV words are taken into account by character-based CTC. Vocabulary selection based approaches propose a relevant vocabulary for speech recognition based on additional text data [START_REF] Sun | Learning OOV through semantic relatedness in spoken dialog systems[END_REF]. These methods are more dynamic as they propose context specific vocabulary [START_REF] Maergner | Unsupervised vocabulary selection for real-time speech recognition of lectures[END_REF], [START_REF] Oger | Ondemand new word learning using world wide web[END_REF], [START_REF] Ohtsuki | Unsupervised vocabulary expansion for automatic transcription of broadcast news[END_REF]. In the present paper, we study this last class methodology. In our approach, our aim is not to identify OOV regions in the ASR hypothesis but to use the global context of the test document.

In this article, we propose a document-specific multiclass approach to dynamically extend the LVCSR lexicon, based on the semantic context of the document. For each test document, we will dynamically generate an extended lexicon, containing all words of the original lexicon and a list of highly relevant OOV PNs. To retrieve OOV PNs we propose to use a very large text corpus: Wikipedia articles, covering a very large number of topics and containing a huge number of proper names. We call this a multipurpose corpus. This corpus is employed to learn a context model that captures relationships between a document and the OOV PNs of the document. As a context model, we propose multi-class DNN approaches. These approaches are capable of learning task specific words and context representations. The retrieved OOV PNs are added to the original lexicon and a second-pass decoding is performed.

In previous works, different methods for OOV PN retrieval have been explored. Latent Dirichlet Allocation (LDA) [START_REF] Oger | Ondemand new word learning using world wide web[END_REF] showed that generated unsupervised context representations are not optimal for our task. The Neural Bag-of-Words (NBOW) model [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF] and the Neural Bag-of-Weighted-Words (NBOW2) model [START_REF] Sheikh | Improved neural bag-ofwords model to retrieve out-of-vocabulary words in speech recognition[END_REF] give very good results. However, in these works, the diachronic text corpus of medium size was used. Moreover, the diachronic text corpus was close in term of topics and date to the test corpus. This is not always possible. Some systems can be deployed in a very large number of tasks. Compared to our previous works [11][12][10], the main contributions of this paper are: (a) the use of a very large multi-topic text corpus to retrieve OOV PNs; (b) the use of multi-class DNN approaches to represent semantic context (CNN and Bi-GRU); (c) the comparison of the proposed models with the fastText approach; (d) the evaluation of the proposed approach in terms of PN error rate on audio files. The advantage of our approach is that the LVCSR lexicon is dynamically adapted to each spoken document without sretraining the other components of the system. The proposed approach can be performed for any language.

The rest of the paper is organized as follows. Section 2 presents the background and describes the proposed approaches. The experiment protocol and the experimental data are described in section 3. The OOV PN retrieval results are discussed in section 4.

PROPOSED METHODOLOGY

The general procedure for retrieving and adding OOV proper nouns to the ASR lexicon is as follows (see Figure 1):

(1) Context model is learned from the multipurpose text corpus.

(2) Given a spoken document to be recognized, the ASR is run to obtain an automatic transcription of the document (first-pass hypothesis).

(3) Based on the semantic context of the first-pass hypothesis, a list of likely relevant OOV PNs is produced. This is performed using the context model and our retrieval methodology. (4) A grapheme-to-phoneme (G2P) conversion is applied to get pronunciations for each word in the proposed list of OOV PNs. OOV PNs and their generated pronunciations are added to the lexicon. The new lexicon is called the extended lexicon. Probabilities for the new OOV PNs are added to the language model. (5) A second-pass ASR decoding is performed using the extended lexicon.

The estimation of the language model probabilities and the generation of word pronunciations for OOV PNs are two large research problems. They are beyond the scope of this article.

For training the context model and retrieving OOV PNs, we propose to use Wikipedia articles: a multipurpose corpus. This corpus has several advantages: it has large lexical coverage and a huge number of topics. An interesting feature of Wikipedia is the huge number of proper nouns. There are approximately 1.7 million OOV PNs in the French Wikipedia (see section 3). Extracting a short (compared to the usual size of a lexicon 100K-200K) list of relevant OOV PNs from 1.7M possible candidates is not trivial. In this article, we address this problem by partitioning OOV PNs from Wikipedia in classes according to their semantic context. We propose a two-step approach:

(1) Partition OOV PNs in classes and learn the context model in term of OOV PN classes using the multipurpose corpus. Using a semantic model, select the OOV PN classes that could occur in the semantic context of the document to be recognized (first-pass hypothesis).

(2) Rank the OOV PNs of the selected classes based on their relevance to the document to be recognized. In the following, we will present these two steps.

Context model and OOV PN class prediction

In order to group OOV PNs from Wikipedia into classes, we propose to apply K-means clustering. K-means is a popular clustering algorithm that aims to partition data points into K clusters. In our problem, the data points are OOV PN word vectors in some space.

As word vector space, we propose to exploit continuous-space representation proposed in [START_REF] Le | Distributed representations of sentences and documents[END_REF] [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF], word2vec. This representation takes into account semantic information of each word. Semantically similar words will be mapped to the same area of the vector space. We apply K-means clustering on the word embeddings space to obtain classes. A single Wikipedia document can cover a mixture of topics, and so might contain OOV words from different classes. Mapping a document to one or more classes is a multi-class classification problem.

After OOV PN class creation, the context model based on DNN is built. The training of this DNN requires a large text corpus containing a huge number of proper names. In this article we use Wikipedia data. The input of the DNN network consists of the in-vocabulary words of the document, the output is the class labels of the OOV PNs present in this document. The size of the output layer is equal to the number of classes. Note that the frequency of an OOV class in a document is not taken into account, only its presence. We used two DNN models: CNN and Bi-GRU.

The first layer of the DNN is an embedding layer: the weight of this layer can be initialized with word embedding computed by word2vec. For multi-class multi-label classification, we chose sigmoid as the activation of the output layer and binary cross-entropy as the loss function.

The trained context model is used to predict the OOV PN classes that are likely to occur in the ASR transcription. Since we formulate our task as a multi-class classification problem, the top-Q classes that have the highest output values are considered as predicted classes.

Convolutional Neural Network

CNN has been successfully applied for text classification [START_REF] Kim | Convolutional Neural Networks for Sentence Classification[END_REF]. Since language is temporal in nature, a one-dimensional convolutional operation can be applied to extract meaningful features from word sequence.

Our network has the following configuration. The input is the sequence of words, represented as one-hot vectors. The first layer is an embedding layer, the second is a convolutional layer and the third layer is a max-over-time pooling layer. The last layer is a dense (fully-connected) layer with a sigmoid as activation function. The last layer has as many neurons as OOV PN classes.

Bi-directional Gated Recurrent Unit Network

It was shown that Recurrent Neural Networks suffer from the vanishing gradient problem [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF] [START_REF] Jozefowicz | An empirical exploration of recurrent network architectures[END_REF]. To address this problem, gated recurrent structures have been proposed: Gated Recurrent Unit (GRU) [START_REF] Cho | Learning Phrase Representations using RNN Encoder--Decoder for Statistical Machine Translation[END_REF]. A GRU is characterized by two gates: reset gate and update gate. Two gates adaptively control how much information from the hidden state should be carried to the next state and how much information from the current input should be taken in the next hidden state. It was shown that processing the sequence in two directions (i.e., forward and backward: Bi-directional GRU) can improve the performance of many sequence modeling applications [START_REF] Graves | Speech recognition with Deep Recurrent Neural Networks[END_REF].

Our Bi-GRU has a similar configuration as the CNN (see Figure 2.). The input is the sequence of words, represented as one-hot vectors. The first layer is an embedding layer; the second is a Bi-GRU layer. In the bidirectional recurrent structure, the last hidden states of GRU in the forward direction and backward direction are merged (by concatenation in our study) to form a single vector that represents the word sequence. The last layer is a dense layer with a sigmoid as activation function. The last layer has as many neurons as OOV PN classes.

OOV PN ranking

The goal of our system is to select a list of PNs that are highly relevant to the spoken document. Since every predicted OOV class consists of many OOV proper nouns, we need a ranking procedure. That is, for each spoken document, we need to retrieve OOV PNs based on their relevance to the semantic content of the document. This could be achieved by quantifying the semantic similarity between the spoken document and each OOV proper noun of the selected classes. To this end, we represent an OOV PN word (oov) by its word embedding (voov) and a document (VDoc) by the average of the word embedding (vw) of the invocabulary words (w) of the document (global semantic context representation):

We use the cosine distance as a measure of semantic similarity between these two vectors:

EXPERIMENTAL SETUP

Data

For the multipurpose and the development text corpuses we used French Wikipedia articles. For the test text and audio corpus, we used French news articles from the Euronews channel.

Text corpora

The multipurpose corpus is necessary to build word embeddings and to train DNN models to infer OOV PNs relevant to the development and the test corpora. We used French Wikipedia articles. We observed that the length of the articles varies greatly. We decided to remove the articles with less than 10 words (after pre-processing).

We used 1% of the Wikipedia data (about 14K documents) as a development set to tune the hyperparameters of our models. The development corpus is used to evaluate the methodology proposed in this paper and to adjust the involved parameters. For the test corpus, we used the text articles from the Euronews website channel [20], written from January 2014 to June 2014. We selected only the articles containing at least one OOV PN word. The system is evaluated with the adjusted parameters on the test corpus.

From Table 1 we can observe that the Wikipedia corpus contains 1.7M unique OOV PNs (OOV PN unigrams) with respect of our 96K LVCSR vocabulary (see section 3.1.5). This number of OOV PNs is very large and it is impossible to add all these words to the LVCSR vocabulary. We assume that the Wikipedia corpus contains a large number of OOV PNs used in the Euronews corpus. The OOV PNs rate in the Euronews corpus is 2.6% (7.9K out of 301K).

Pre-processing of the text data

To train and evaluate proposed models, the text corpus was preprocessed: the TreeTagger part-of-speech tagging tool is used to automatically tag PNs. The TreeTagger is reported to perform with an accuracy of 95.7% on French data [START_REF] Allauzen | Training and evaluation of pos taggers on the french multitag corpus[END_REF]. The PNs and non-PN words that occur in the lexicon of our LVCSR system are tagged as IV and the remaining PNs are tagged as OOV. For training the DNN models, words in the multipurpose text corpus are lemmatized. Moreover, PNs and non-PNs occurring less than 5 times are discarded. A stop-list of common words and non-content words is applied and a part-of-speech based filter is employed to choose words tagged as PN, noun, adjective, verb and acronym. Other words are discarded. The context models are trained and evaluated with this filtered corpus. For word2vec wordembedding this pre-processing is not performed.

Wikipedia corpus (train and development corpora)

Test audio corpus

In order to evaluate the recognition performance of the proposed approaches, we collected a test audio corpus. This corpus is composed of 296 French videos from the Euronews channel, corresponding to a total duration of 6 hours (corpus size 60K words). OOV PN rate is 1.3% [START_REF] Sheikh | How diachronic text corpora affect context based retrieval of OOV proper names for audio news[END_REF].

OOV PN retrieval performance measures

We used several evaluation measures:

(1) For OOV PN class evaluation, we used the recall at top-Q predicted classes (Recall@Q in figures) and the Fmeasure at top-Q predicted classes (F-measure@Q in figures) [START_REF] Weston | #TagSpace: Semantic embeddings from hashtags[END_REF].These metrics are calculated for each document and averaged over all evaluation documents.

(2) For OOV PN ranking, we computed the recall (Recall): for each document, the OOV PNs of the best predicted classes are sorted. After this, the recall for top-N words (operating point) is computed and averaged over all evaluation documents [START_REF] Manning | Introduction to Information Retrieval[END_REF].

Recognition system

The (3) the top-N words are added to the original lexicon and to the language model. Finally, we obtain an extended lexicon of (96K+N) words for each test file. We perform a second pass speech recognition to recognize the OOV PNs by updating the LVCSR system using the extended lexicon and modified language model. To update the pronunciation lexicon, automatic G2P converters are used [START_REF] Bisani | Joint-sequence models for grapheme-to-phoneme conversion[END_REF], [START_REF] Illina | Multiple Pronunciation Generation using Grapheme-to-Phoneme Conversion based on Conditional Random Fields[END_REF]. Up to three pronunciations per new OOV PN are generated using a CRF [START_REF] Iyyer | Deep unordered composition rivals syntactic methods for text classification[END_REF]. Regarding the language model, we added only unigram probability for the new OOV PNs. The mass probability attributed to the new OOV PN unigrams has been subtracted from the probability of the <unk> probability [START_REF] Sheikh | Modelling Semantic Context of OOV Words in Large Vocabulary Continuous Speech Recognition[END_REF]. Thus, the total unigram probability adds up to 1.

Context model configurations

As training set, the multipurpose train text corpus (Wikipedia) is used. For each model, the parameter set is adjusted on the development text corpus (one part of Wikipedia) and the best parameter sets are given:  Word embedding is used for three different purposes:

(1) to initialize the embedding layer of the neural models;

(2) to build document embeddings by taking the average of the word embeddings of in-vocabulary of the document; (3) to rank OOV PN list. For computing the word embeddings, we use the skip-gram word2vec with 400 dimensions, context window of 20 words, and hierarchical softmax. We only consider words that occur at least 5 times in the training corpus (Wikipedia). Finally, we obtained the embeddings for about 562K OOV PNs.  fastText. As baseline, the standard text classification tool fastText [START_REF] Bojanowski | Enriching Word Vectors with Subword Information[END_REF] is used. As input, the raw text is used. As output, the class labels are given. Therefore, one document can have several class labels.  CNN. The first layer is composed of an embedding layer whose weights are initialized by the word2vec embeddings. We used one convolutional layer followed by a max pooling layer and a ReLU dense layer. A dropout of 0.2 is applied before the output layer. Our preliminary experiments showed that trainable word embeddings slightly improve the performance of CNN. Thus, we set the embedding layer to be trainable.

 Bi-GRU. Like the CNN, the first layer is composed of an embedding layer whose weights are initialized by the word2vec embeddings. The second layer is a bidirectional GRU with Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], early stopping and 0.2 dropout rate. The standard backpropagation algorithm with stochastic gradient descent is used to train the network. To speed up the training, we used static word embeddings (the first layer is frozen).

EXPERIMENTAL RESULTS

OOV PN class prediction results

The classification consists in finding for each document which OOV PN classes this document could contain. It is multi-class labeling because one document can correspond to several OOV PN classes. The classification is performed using the context models proposed in this paper.

For defining the classes, we used a K-means clustering for grouping OOV PNs into classes. We used word embedding and cos-distance to cluster OOV PN words. We created 1000 classes. It is a good compromise between the number of OOV PNs in the Wikipedia corpus (1.7M) and the average number of words per class. A smaller number of classes decreases the model performance.

CNN

CNNs require careful tuning of many hyperparameters. We explored two important design choices: the window size and the number of filters. In these experiments, we keep all hyperparameters of the model fixed and we investigate the impact of various values of a studied parameter. The results of this experiment on the development set (Wikipedia) are shown in Table 2. We cannot observe any trend regarding the change of the window size and the classification performance. Regarding the impact of the number of filters, from Table 2 it can be observed that increasing the number of convolutional filters leads to consistent performance gains. Note that the number of filters represents the dimensionality of the document representation due to the max-pooling operation. CNN with 1000 filters and window size of 3 is used in the following experiments.

Bi-GRU

Since our Wikipedia multipurpose data is large, training many Bi-GRU models is time consuming. We only report the result for 400-dimensional Bi-GRU and 800-dimensional Bi-GRU. The dimensionality of the model refers to the size of the hidden state. For example, the 400-dimensional Bi-GRU consists of two GRUS each with 200 hidden state, one GRU processes the input sequence in the forward direction while the other GRU processes the input sequence in the backward direction. The result of the two GRU layers is concatenated. From Table 2, it can be observed that the 800dimensional Bi-GRU outperforms the 400-dimensional BI-GRU. Bi-GRU shows the best performance compared to the other models investigated in this article. This good performance of Bi-GRU is probably because the hidden states of gated recurrent architectures have a large capacity for encoding sequential information due to the gate mechanism.

Dev set (Wiki)

Recall@5 Recall@10

Discussion

Figure 3 summarizes the OOV PN classification results on the Euronews test set using the best configurations for each model. We observe that the fastText model, used as baseline, is outperformed by the proposed models. CNN models give slightly inferior results compared to Bi-GRU. The best performance is obtained by the Bi-GRU context model. In terms of F-measure, this model obtains 33.2% using the best class.

OOV PN ranking results

In the previous sections, we computed the results at the class level. In this section, for each test document, we assume that we have retrieved one or several OOV PN classes. From these classes, we extract a list of ranked OOV PNs. Figure 4 shows the OOV PNs ranking results in terms of recall on the Euronews text test set. In Figure 4, the x-axis represents the number of OOV proper nouns (in logarithmic scale) that could be added to the LVCSR lexicon before running second-pass decoding. The y-axis represents the OOV PN retrieval performance measured by the average recall on the Euronews test set. From Figure 4 we can observe that Bi-GRU performs better than the extension of the LVCSR lexicon based on fastText and CNN. We can also notice that Bi-GRU, CNN and fastText outperform the adding of the most frequent OOV PNs according to occurences on the Wikipedia corpus (method called freq in the figure).

Compared to [START_REF] Sheikh | Modelling Semantic Context of OOV Words in Large Vocabulary Continuous Speech Recognition[END_REF], presented results are better in terms of recall. It is important to note, that in [START_REF] Sheikh | Modelling Semantic Context of OOV Words in Large Vocabulary Continuous Speech Recognition[END_REF] a small diachronic corpus close to the test corpus was used to retrieve OOV PNs. In the present approach, the task of retrieval is more difficult because we employed a very large multipurpose corpus (Wikipedia). The advantage of this corpus is that it is multi-topic and covers a very large number of PNs. Thus, it can be used not only for the broadcast news domain but for other domains.

Speech recognition results

In this section, the list of relevant OOV PNs, that are retrieved according to the proposed methodology, is used for speech recognition. We extract top-4000 OOV PNs from the list of retrieved OOV PNs per test audio document. We construct the extended lexicon (original lexicon and 4K OOV PNs) per document and update the language model per document. We perform a second pass speech recognition with the updated LVCSR system.

Table 3 displays the word error rate (WER) and proper name error rate (PNER) results on the test audio corpus using original LVCSR and LVCSR with extended lexicon (using the Bi-GRU context model and second-pass recognition). PNER is obtained by first aligning the reference and hypothesized word level transcriptions and then calculating substitution, deletion, insertion errors, and the error rate only on the proper name terms. From Table 3, we can see that adding the new PNs into the lexicon and language model had a positive impact on the WER. The WER showed an improvement. This improvement is due to the recognition of OOV PNs and the reduction of insertion and deletion errors. The PNER reduction is larger compared to WER (6% relative improvement). This PNER decrease shows that new proper nouns were appropriately added to the lexicon.

Method

WER PNER LVCSR with original lexicon 20.3 39.9 LVCSR with extended lexicon (using Bi-GRU) 19.9 37.4 Table 3. WER (%) and PNER (%) performance of OOV PN retrieval for Euronews audio test set using second-pass recognition.

CONCLUSION

In this work, we address the problem of improving recognition of an ASR system by adding new OOV PNs to the lexicon. We propose to retrieve OOV PNs dynamically for each test file without retraining the other components of the system. For this, a multipurpose corpus is used: Wikipedia. The great advantages of this corpus are a large set of OOV PNs and a multi-topic domain. Thus, this corpus can be used for any domain of speech recognition. The proposed approach can be performed for any language. We proposed a global semantic context model relying on DNNs. The Bi-GRU model shows the best performance. On the test data, the models we proposed yielded improvements in recall over the method based on fastText; in addition, on the French broadcast news transcription, Bi-GRU model led to a decrease in PN error rate. This shows that new proper nouns were appropriately selected.

ACKNOWLEDGMENT

This work is funded by the Chist-Era AMIS (Access Multilingual Information opinionS) project. Experiments presented in this paper were partly carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria, CNRS, RENATER and other Universities and organizations (https://www.grid5000.fr).

Figure 1 .

 1 Figure 1. Illustration of our approach for recognition OOV PNs [11].

Figure 2 .

 2 Figure 2. The Bi-GRU model.

.

Figure 3 .

 3 Figure 3. OOV PN classification results in term of recall and Fmeasure. Euronews text test data.

Figure 4 .

 4 Figure 4. Recall performance of OOV PN retrieval for Euronews text test set.

Table 1 .

 1 Statistics

	Number of documents	1.4M
	Corpus size (word count)	228M
	Vocabulary size	2.4M
	OOV PN unigrams	1.7M
	Total count of OOV PNs	20M
	Euronews corpus (test corpus)
	Number of documents	3.1K
	Corpus size (word count)	301K
	Vocabulary size	23.8K
	OOV PN unigrams	3.9K
	Total count of OOV PNs	7.9K

about text data after pre-processing. K denotes thousand and M denotes million.

  Kaldi-based Automatic Transcription System (KATS) employs context dependent DNN-HMM phone models trained on 200-hour broadcast news audio files. The original lexicon contains about 96K words. Using the Pocolm toolkit [25], bigram and trigram language models are estimated on the "Le Monde" + Gigaword corpus and used to produce the word lattice. The original lexicon of 96K words is augmented by adding the retrieved OOV PN word list as follows. For each development/test file: (1) we create a ranked list of OOV proper nouns; (2) we keep only top-N words of this list;

Table 2 .

 2 CNN. The impact of the window size and of the number of filters on the classification results for the CNN. Impact of dimensionality for Bi-GRU. Development set (Wikipedia).

	Window size	CNN
	3	45.4	54.5
	4	45.1	54.7
	5	45.0	54.0
	6	44.8	54.1
	Nbr of filters
	300	45.4	55.5
	600	46.1	55.4
	1000	46.2	55.5
	Dimension	Bi-GRU
	400 (2x200)	46.5	56.0
	800 (2x400)	47.2	56.7


Arnaud Ralko
email: arnaud.ralko@neel.cnrs.fr

Ioannis Rousochatzakis

Resonating valence bond physics is not always governed by the shortest tunneling loops

It is well known that the low-energy sector of quantum spin liquids and other magnetically disordered systems is governed by short-ranged resonating valence bonds (RVB). Here we show that the standard minimal truncation to the nearest-neighbor valence bond (NNVB) basis fails completely even for systems where it should work the most, according to received wisdom. This paradigm shift is demonstrated for the quantum spin-1 2 squarekagome, where strong geometric frustration, similar to the kagome, prevents magnetic ordering down to zero temperature. The shortest tunneling events bear the strongest longer-range singlet fluctuations, leading to amplitudes that do not drop exponentially with the length of the loop L, and to an unexpected loop-six valence bond crystal (VBC), which is otherwise very high in energy at the minimal truncation level. The low-energy effective description gives in addition a clear example of correlated loop processes that depend not only on the type of the loop but also on its lattice embedding, a direct manifestation of the long-range nature of the virtual singlets.

Introduction -The search for quantum spin liquids has been an active topic in condensed matter physics for many years [1][2][3][4][5][6][18][19][20][21]. Recently, this search has gained new impetus from the discovery [7][8][9][10][11][12][13][14][15] of a series of layered spin- 1 2 kagome antiferromagnets (AFMs), whose ideal nearest-neighbor (NN) Heisenberg limit stands out as the prime candidate for a Z 2 topological spin liquid [16,17].

One of the earliest fundamental insights from the theory side is that the low-energy sector of all magnetically disordered AFMs should be governed by resonances between sufficiently short-ranged singlet or valence bond (VB) pairings [6,[18][19][20][21]. Casting this very physical picture into an effective Hamiltonian has been challenging for many years [22][23][24][25][26][27][28][29][30][31]. While one aspect of the problem, the non-orthogonality of the VBs [23], was essentially resolved recently [28][29][30][31], the really serious problem turns out to be the truncation to the NNVB basis, a problem that goes back to Zeng and Elser [24]. This is the second study that focuses on the impact of virtual singlets outside the NNVB basis to the tunneling physics. Ref. [31] has dealt with the 2D kagome, where the delicate competition between different resonances leads to a VBC [32][33][34][35] which is very fragile [29,30,36], and virtual singlets are essential to stabilize the Z 2 spin liquid [31].

The aim of this work is to show that virtual singlets have a strong qualitative effect in all disordered AFMs, even when the minimal NNVB truncation is expected to work the most, according to received wisdom. To this end, we have chosen an extreme case of an AFM, the square-kagome [37][38][39][40][41][42] of Fig. 1, which features a huge tunneling energy separation at the minimal truncation level. This happens because this system manifests the shortest resonances possible, the squares [ 4 in Fig. 1 (b)], with NNVB amplitudes that are five times stronger than the second-shortest, 'loop-six' events. This energy separation leads to the very robust 'pinwheel' VBC of Fig. 1 (c, left) [37,40], well separated in energy from competing RVB states.

Yet, even for such a system, virtual singlets change the physics qualitatively because they have very different impact on processes with different loop length L: The square loops hold the largest density of defect triangles (triangles without VBs [43]) and thus bear much stronger quantum fluctuations, which suppress the loop-four amplitude by a striking factor of ∼10. This leads to 'renormalized' amplitudes that do not drop exponentially with L, and to an unexpected 'loop-six' VBC [Fig. 1 (c, right)] which is otherwise very far in the parameter space in the absence of virtual singlets. The result that virtual singlets change the physics qualitatively even in such a system simply tells us that the minimal NNVB truncation is probably inadequate for all standard [44] disordered AFMs.

The 'dressed' NNVB description of the square-kagome offers in addition a clear example of correlated loop processes, which depend not only on the type of the loop but also on the particular VB environment of the loop. This embedding dependence is a direct manifestation of the long-range nature of virtual singlets, and adds another twist in the way we think about the effective RVB dynamics of disordered magnets.

Model -The square-kagome features two symmetry inequivalent sites (A and B), two NN bonds (AA and AB), and two void loops (AA-squares and AB-octagons), see Fig. 1(a). Hence the most natural spin- 1 2 Heisenberg model comprises two NN exchange constants, J AA and J AB ,

H Heis = ij J ij S i • S j , (1)
where ij denotes NN sites. We target the nature of the disordered phase labeled 'Y a ' in [40], which appears in the highly frustrated region x ≡ JAB JAA ∼ 1. This is precisely where the RVB picture is most relevant, because at x = 1 we deal with a system of corner-sharing equilateral triangles, and such systems hold the two key RVB ingredients [24][25][26]43]: i) a manifold of energetically favorable short-ranged VB states, and ii) a mechanism that drives the resonances between them. The first derives from the fact that each triangle can be minimized exactly by pairing two out of the three spins into a singlet. This leads to an extensive number of NNVB states where a large portion of the triangles ( 3 4 ) host a singlet and are thus satisfied locally. The second ingredient stems from the quantum fluctuations around the remaining, void or 'defect' triangles [43].

JAA J AB a) ⇤ 4 ⇤ 6a ⇤ 6b ⇤ 8 b) |P W ± i |P R + i c)
The square-kagome has 22 topologically distinct dominant processes, 4 around AA-squares and 18 around ABoctagons [40]. The former are the most crucial and are shown in Fig. 1 (b): The loop-four or 'perfect square' resonance 4 , two loop-six processes 6a and 6b , and finally the loop-eight or 'square pinwheel' process 8 , involving NNVB states with 2, 1 and 0 defect triangles, respectively. Similarly, the octagonal processes [45] consist of: the 'perfect octagon' resonance 8 , four loop-10 10a-d , eight loop-12 12a-h , four loop-14 14a-d , and finally the 'octagonal pinwheel' process 16 , with 4, 3, 2, 1 and 0 defect triangles, respectively.

To accommodate the effect of virtual singlets on the quantum dimer model (QDM) [22-29, 50, 51] we write down an embedded QDM [31] H QDM = ,e t e (|1 e 2 e |+h.c.)+V e (|1 e 1 e |+|2 e 2 e |), (2) where we sum over all loops that involve a resonance between two NNVB states |1 e and |2 e [blue and red ovals in Fig. 1 (b)], and all possible VB environments e for given .

The corresponding tunneling amplitudes and potential energies are denoted by t e and V e (measured from E 0 = - 3 8 N , N is the number of sites).

Minimal truncation & main competing phases -Within the minimal NNVB truncation method, the parameters t e and V e are independent of the embedding e [31]. Their values are reported in [40] based on a method which is equivalent to the infinite order overlap expansion of [28,29]. Namely, for each process we consider a minimal cluster that can host only the two NNVB coverings, |1 e and |2 e , involved in the process, see top line of Fig. 2. For these minimal clusters, the NNVB problem amounts to diagonalizing the Hamiltonian

H NNVB = O -1 2 H Heis O 1 2

NNVB

, where O is the overlap matrix in the NNVB basis {|1 e , |2 e }. The matrix elements of O and H NNVB can be found following standard rules [22][23][24][25][26]28].

The parameters involving the AA-squares are:

t 4 = -1, V 4 = 1 2 , t 6a = t 6b = 1 5 (2x-1), V 6a = V 6b = 1 20 (2x-1), and t 8 = 8 21 (1 -x), V 8 = -1 21 (1 -x).
For the remaining ones see Table I of [40].

Although the RVB description is only relevant around x = 1, it is useful to examine the phase diagram as a function of x, in order to establish the main competing phases of the problem. Figure 3 shows exact diagonalization (ED) results from the 48-site cluster on a torus, using the parameters from the above 2×2 NNVB truncation. The low-energy spectra (a-b) and ground state (GS) expectation values of various loop densities (c) reveal three VBC phases: i) the 'even pinwheel' of Fig. 1 (c,left) for x < 1, ii) the 'odd pinwheel' of Fig. 1 (c,left) for 1 < x < 1.75, and iii) the 'loop-six' VBC of Fig. 1 (c,right) for x > 1.75. The potential terms do not alter this picture qualitatively, as shown in Fig. 3. Without potential terms the transition between the 'odd pinwheel' and the 'loop-six' crystals shifts to x 2.34.

All three states belong to the trivial topological sector, and break the translation symmetry in the same way, with a doubling of the unit cell. This symmetry breaking manifests in the spectra by an almost exact degeneracy between two states with momenta 0 and (π, π). The extremely small splitting [not visible in Fig. 3 (a,b)] shows that finite-size effects are already very weak for the 48-site cluster.

Let us look at these states in more detail. The two 'pinwheel' states maximize the number of 4 resonances with even parity (i.e. with the symmetric combination of the two NNVB states involved, since t 4 = -1 < 0 [START_REF]Our singlet-orientation convention is clockwise in void squares and octagons[END_REF]), denoted by |PR + in Fig. 1 (c,left). This amounts to a percentage of 1 2 resonating AA-squares. The remaining half carry 'pinwheels', denoted by |PW ± , whose parity is fixed by the sign of t 8 = 8 21 (1x), i.e., it is odd (even) for x < 1 (> 1). At x = 1, the two parities give rise to an extensive degeneracy of 2 N 12 (on top of the two-fold degeneracy from translational symmetry breaking), which can be clearly seen in the exact spectra of Fig. 3 (a,b). This is analogous to the 36-site VBC of the kagome [30,[32][33][34][35], which maximizes the 'perfect hexagon' resonances, and features a similar extensive degeneracy due to 'hexagonal pinwheels'. Here the degeneracy is lifted for x = 1 due to the inequivalent bond structure [40].

Turning to the 'loop-six' crystal, this state maximizes the 6a resonances, denoted by |L6 -in Fig. 1 (c,right). Here all AA-squares carry such a resonance and the parity is odd since t 6a = 1 5 > 0. We emphasize here that at the level of the minimal NNVB truncation, t 6a = t 6b , yet the favored 'loop- 6b have a very small excitation energy, as can be seen in the low-energy excitation spectrum of Fig. 3 (a,b).

R = 0 R = 1 R = 2 ⇤ (1) 6b ⇤ (2) 6b ⇤ (2) 6a ⇤ (1) 6a ⇤ 4 ⇤ 8 ⇤ (0) 6a ⇤ (0) 6b
Fluctuations are in fact very weak for all three VBC phases. As shown in Fig. 3 (c), the GS loop densities of 4 and 8 are almost exactly equal to 1/2 for x < 1.75, while that of 6a is almost exactly equal to 1 for x > 1.75. The same fact is also

⇤ 6a ⇤ 4 , ⇤ 8 V 0 = 0 ⇡/L ⇡/L V 0 6 = 0
E (0) pinwheel /N = (V 4 -|t 4 | + V 8 -|t 8 |) /12 , E (0) loop-six /N = (V 6a -|t 6a |) /6 , (3)
of the ideal product states between the resonating building blocks without extra quantum fluctuations. The latter are very weak because they arise from sequences of longer loop events that necessarily involve the octagons, whose tunneling amplitudes are very small, see [40]. 'Dressed' NNVB description -Having established the main competing phases of the problem we now focus again on the highly frustrated point x = 1. As announced above, the longerrange singlets that are virtually excited around the defect triangles destabilize the 'pinwheel' crystals in favor of the resonating 'loop-six' state, i.e. they effectively shift the boundary between the two phases in a drastic way. Essentially, the ratio -t 4 /t 6a drops from the NNVB value of 5 below 2, which is the transition value obtained from (3) with t 8 = 0 and by neglecting the potential terms (see below).

Let us see how such a drastic reduction takes place here. Following [31], we extract the QDM parameters from the ED spectra of appropriate finite Heisenberg clusters, using the lowest two exact eigenvalues with the right symmetry. This method is essentially equivalent to enlarging the truncation basis in order to include the virtual singlets accommodated by the clusters. Figure 2 shows the clusters considered here for the four processes around AA-squares (for the octagon processes see [45]). The clusters shown at the top line capture the virtual singlets that are closest to the central loops.

To capture the contribution from other virtual singlets we proceed as in [31] and add sawtooth chains of length R at the positions of the defect triangles of the minimal R = 0 clusters. These chains provide the only escaping paths of the virtual singlets (or the spinons forming these singlets [START_REF] Hao | [END_REF]), provided we do not encounter extra defect triangles nearby. Such extra defect triangles become first relevant for the loop-six pro- cesses, and can be studied by examining different variations of the corresponding clusters for each given R, that accommodate zero, one or two extra defect triangles, see Fig. 2. The corresponding effect on the octagonal processes is disregarded here since these processes are anyway too weak to affect the low-energy physics quantitatively. The extracted tunneling amplitudes for the square and the octagonal processes are given in Table I and Table 2 of [45], respectively, and deliver a number of clear insights (the potential terms are much smaller and can be disregarded [45]). First, except for the minimal R = 0 embedding of t 4 (and the processes 8 and 16 whose amplitudes are zero irrespective of R), the amplitudes are clearly different from the ones resulting from the minimal 2×2 truncation (NNVB). The differences show that the 2×2 truncation give in most cases a very poor approximation, so the virtual singlets cannot be ignored.

Second, the shortest amplitude t 4 shows a remarkable reduction by about ten times from R = 0 to R = 3. By contrast, the amplitudes t (e) 6a drop by about 2-3 times (depending on e), and show a relatively faster convergence with R compared to t 4 . Virtual singlets have therefore a much stronger tendency to penalize the shortest loop events because they correspond to the highest local density of defect triangles. With the ratio -t 4 /t (e) 6a dropping below the transition value of 2 above R = 2, it is immediately clear that the RVB physics will no longer be governed by the shortest resonances.

Third, our results show a ∼ 20% variation of (e) 6a and a much stronger ∼ 40-60% variation of (e) 6b depending on e, i.e. on whether there are zero, one or two extra defect triangles nearby. Such a strong dependence on one class of the dominant processes of the problem must be taken into account explicitly in the effective description. The presence of such correlated loop processes, which depend strongly not only on the type of the loop but also on the embedding, highlights the qualitatively non-trivial impact of virtual singlets, which effectively sample the immediate VB environment of a loop.

Fourth, virtual singlets have a different impact on t 6a and t 6b , again in contrast to the minimal NNVB truncation. For all R we systematically find that t preference for loop-six resonances of the 6a type. In the minimal NNVB model discussed above, such events were selected by weak virtual octagonal processes whereas here they are favored directly by virtual singlets.

Finally, the processes around octagons are also renormalized substantially by virtual singlets, see [45]. But, these processes remain anyway harmless, which is checked by a full ED of the 'dressed' QDM including all 22 processes. Fig. 4 shows the connected dimer-dimer (or density-density in the QDM setting) GS correlation profiles using the parameters from the minimal NNVB truncation and from the R = 2 clusters, with the explicit embedding dependence for the loopsix processes. At this level, we have access to all amplitudes except t 8 , whose corresponding cluster is too large for ED. In the calculations shown here we have taken the conservative assumption t R=2 8 = t R=1 8 , which does not affect the results. The data confirm that the 'loop-six' crystal is stabilized already at R = 2, and taking the R = 3 amplitudes renders this phase even more robust.

Outlook -We have shown that the minimal truncation to the NNVB basis fails qualitatively even in a system where it is expected to work the most, according to received wisdom. This is strong evidence that virtual singlets have a qualitative impact in all standard disordered AFMs. Not only do the virtual singlets 'dress' the basic RVB building blocks, they also completely alter fundamental notions that derive from our experience with the minimal NNVB truncation: The tunneling amplitudes do not drop exponentially in the length L of the loops, and the tunneling dynamics cannot always be approximated as a sum over strictly local loop processes, as initially envisaged by Rokhsar and Kivelson [23]. i Supplemental material A. General remarks for the NNVB manifold of the square-kagome antiferromagnet As shown in [1] by a generalization of the so-called arrow representation [2], the dimension of the NNVB Hilbert space is 2×2 N/3 , where N is the total number of sites. Similar to the non-bipartite, triangular and kagome lattices, the NNVB manifold splits into four topological sectors for periodic boundary conditions in both directions. These sectors can be labeled by the parities (p x ,p y ) of the number of dimers intersecting any horizontal or vertical line [3][4][5][6]. Here, the sectors (0,1) and (1,0) are connected to each other by the four-fold symmetry of the lattice.

B. Basic elements of the minimal 2×2 NNVB truncation

The basic elements of the minimal 2×2 NNVB truncation method in the square-kagome AFM are given in [1]. For convenience they are repeated here in Table II. Let us consider any of the finite spin-1/2 Heisenberg clusters of the first row of Fig. (2) of the main text (or Fig. 5 for the octagonal processes) and consider the elementary tunneling process p between the two dimer coverings |1 p and |2 p that can be accommodated by this cluster. For the dimer orientations [7], we take the singlets to be oriented clockwise in each void square and octagon. The transition graph [8] of our elementary process involves a single nontrivial loop of length L p , surrounding a single square or a single octagon. Then, within our convention, the overlap between the two NNVB states is given by

ω p = 1 p |2 p = (-1) Lp 2 2 1-Lp 2 . (4)
The diagonal and off-diagonal matrix elements of the Hamiltonian,

E 0,p = 1 p |H Heis |1 p = 2 p |H Heis |2 p , v p = 1 p |H Heis |2 p , (5)
can be found using standard rules [8][9][10][11][12][13][14]. In the basis {|1 p , |2 p } we have

H NNVB = (O -1 2 H Heis O -1 2 ) NNVB = 1 ω p ω p 1 -1 2 E 0,p v p v p E 0,p 1 ω p ω p 1 -1 2 = E 0,p + V p t p t p E 0,p + V p (6)
with

t p = +h p ω p 1 -ω 2 p , V p = -t p ω p , where h p = v p /ω p -E 0,p . (7)
The quantity h p corresponds to Eq. ( 12) of Schwandt et al [15], modulo a factor of 3/4 related to the redefinition of the Hamiltonian; see passage below Eq. ( 10) in [15]. So Eq. ( 7) is consistent with Eq. ( 40) of [15]. In other words, the above 2×2 truncation is equivalent with the infinite-order cluster expansion of [15].

4 +2 -1 -3 -3 2 -1 + 1 2 6a,b 6 -2 -2 -3x -3 4 (2x + 1) + 1 5 (2x -1) + 1 20 (2x -1) 8 8 +2 -3 -3(2x -1) -3x + 8 21 (1 -x) -1 21 (1 -x) AB-octagon 8 8 +2 -3 -6x -3x -8 21 x + 1 21 x 10a-d 10 -2 -4 -3 2 (3x + 1) -3 4 (4x + 1) + 4 85 (2x + 1) + 1 340 (2x + 1) 12a-f 12 +2 -5 -3(x + 1) -3 2 (2x + 1) -16 341 + 1 682 14a-d 14 -2 -6 -3 2 (x + 3) -3 4 (4x + 3) + 16 1365 (3 -2x) + 1 5460 (3 -2x) 16 16 +2 -7 -6 -3(1 + x) -128 5461 (1 -x) + 1 5461 (1 -x)
TABLE II. QDM parameters tp and Vp (in units of J AA ) for the four processes around AA-squares and the 18 processes around AB-octagons, using the minimal 2 × 2 NNVB truncation approach. Lp is the length of the loop in the transition graph of |1p and |2p . For the dimer orientations we follow the convention that singlets are oriented clockwise in each even-length loop. All values correspond to the clusters of the first row of Fig. 5. If we use clusters of any other row, E0,p and vp will change, but the overlap ωp and the combination hp = vp/ωp -E0,p do not change, leading to the same tp and Vp.

C. Potential terms

Here we discuss the potential terms of the effective QDM. As explained in [16], the largest contributions to the potential energy arise from processes involving a single defect triangle, and as such they can be absorbed in a global energy shift, since the total number of defect triangles is a constant in the NNVB basis. The remaining contributions can be divided into binding energies among two, three, etc. defect triangles. In the present case, the most important processes (i.e. the ones around the AAsquares) involve up to two nearby defect triangles, for which the binding energy is about 0.01 [16], i.e. 10 times smaller than the dominant tunneling amplitudes. This is why we can safely disregard the potential terms in the effective QDM description of the square-kagome.

D. Finite clusters for the octagonal processes

Figure 5 shows the finite-size spin- 1 2 Heisenberg clusters that are used to extract the tunneling parameters of the 18 different octagonal processes (for the processes around AA-squares see main text).

FIG. 1 .

 1 FIG. 1. (Color online) (a) The square-kagome has two inequivalent sites (A and B) and two NN exchange couplings JAA and JAB. (b) The dominant processes around AA-squares. Blue and red ovals denote the two NNVB states involved in each process. (c) The two types of competing VBCs, the 'pinwheel' (left) and the 'loop-six' (right). (d) Virtual excursions outside the NNVB basis. Here, the virtual singlet at the top is excited by applying the Hamiltonian on a defect triangle (shaded green) of one of the two NNVB states (red).

FIG. 2 .

 2 FIG. 2. (Color online) Finite spin-1/2 Heisenberg clusters from which we extract the tunneling amplitudes for the four dominant processes on the lattice, see text. The shaded (green) triangles indicate the closest defect triangles give rise to an embedding dependence.

FIG. 3 .

 3 FIG. 3. (Color online) ED results of (2) using the minimal NNVB parameters, as a function of x = JAB/JAA, with and without potential terms V . (a-b) Low-energy spectra measured from the GS. The 32 states (2×2 N 12 ) that become degenerate at x = 1 (see text) are shown in grey, and the number d is the degeneracy of each level. (c) GS expectation values of various loop densities. (d) Comparison between the GS energy with (3), see text.

  demonstrated in Fig.3 (d) which shows that the exact energies of the lowest eigenstates (symbols) lie practically on top of the corresponding energies (lines)

FIG. 4 .

 4 FIG. 4. (Color online) Connected dimer-dimer correlation profiles in the GS of (2) at x = 1, using the NNVB tunneling parameters (left) and the ones extracted from the R = 2 clusters of Fig. 2 (right), which includes the explicit dependence on the extra defect triangles (see text). The reference dimer is the horizontal segment at the bottom left corner. The red and grey (left) or blue and grey (right) denote positive and negative values.

FIG. 5 .

 5 FIG. 5. Finite clusters used to extract the octagonal tunneling parameters of the effective Hamiltonian. The gray triangles are the defect triangles involved in each tunneling process.

TABLE I .

 I Tunneling parameters t for the square processes (for the octagonal processes see[45]), as extracted from the finite spin-1/2 Heisenberg clusters of Fig.2, as described in the text. The numbers from the minimal NNVB truncation are also shown for comparison.

		NNVB R=0	R=1	R=2	R=3
	4	-1	-1	-0.385403 -0.208442 -0.133124
	(0) 6a	+1/5	+0.266044 +0.151461 +0.125542 +0.113368
	(1) 6a	+1/5	+0.266044 +0.131499 +0.102100 +0.089369
	(2) 6a	+1/5	+0.266044 +0.124130 +0.093471 +0.081068
	(0) 6b (1) 6b (2) 6b	+1/5 +1/5 +1/5	+0.116951 +0.097390 +0.088453 +0.082762 +0.116951 +0.063010 +0.057934 +0.054884 +0.116951 +0.002895 +0.029735 +0.030038
	8	0	0	0	0	0

Acknowledgements -We are grateful to F. Mila for enlightening discussions and suggestions. This work is supported by the French National Research Agency through Grants No. ANR-2010-BLANC-0406-0 NQPTP.


est destinée au dépôt

… but need precise input variables ; it could also be further improved by integrating stresses effects.

  Modelling nitrogen partitioning to filling seeds in pea (Pisum sativum L.) : a tool to predict seed N concentration Annabelle Larmure, Caroline Surleau, Nathalie G. Munier-Jolain, Vincent Faloya To cite this version: Annabelle Larmure, Caroline Surleau, Nathalie G. Munier-Jolain, Vincent Faloya. Modelling nitrogen partitioning to filling seeds in pea (Pisum sativum L.) : a tool to predict seed N concentration. 5th European Conference on Grain Legumes and ICLGG, Jun 2004, Dijon, France. 85, 2004, Legumes for the benefits of agriculture nutrition and the environment : their genomics, their products, and their improvement.  low root of the mean squared error of prediction (MSEP = 2.7 mg g -1 )

Evaluation of the crop model component


PhD Esther Wiersinga-Post
email: j.e.c.wiersinga-post@med.umcg.nl

Sonja Tomaskovic

Lavinia Slabu

Remco Renken

Femke De Smit

Hendrikus Duifhuis

Decreased BOLD responses in audiovisual processing

Keywords: audiovisual binding, audiovisual integration, crossmodal, deactivation, fMRI, human brain imaging, inhibition, McGurk effect, multimodal, sparse sampling

Audiovisual processing was studied in a functional magnetic resonance imaging study using the McGurk effect. Perceptual responses and the brain activity patterns were measured as a function of audiovisual delay.

In several cortical and subcortical brain areas, BOLD responses correlated negatively with the perception of the McGurk effect. No brain areas with positively correlated BOLD responses were found. This was unexpected as most studies of audiovisual integration use additivity and super additivity -that is, increased BOLD responses after audiovisual stimulation compared with auditoryonly and visual-only stimulation -as criteria for audiovisual integration. We argue that brain areas that show decreased BOLD responses that correlate with an integrated audiovisual percept should not be neglected from consideration as possibly involved in audiovisual

Introduction

To perceive objects and events in the environment, organisms are equipped with different senses each specialized for the detection of certain physical aspects of the outside world. How the brain manages to combine information that enters through separate sensory chan-nels and forms a unified percept of the outside world is still not well understood.

This study investigates which brain areas are involved in audiovisual (AV) binding using functional magnetic resonance imaging (fMRI) combined with behavioral measurements of the McGurk effect [START_REF] Mcgurk | Hearing lips and seeing voices[END_REF]. This effect is very suitable for studying AV binding, as the auditory percept of McGurk stimuli depends on whether AV information is integrated or processed separately. Stimuli that easily elicited the McGurk effect can for instance be made by dubbing the syllables /aba/ and /apa/ on movies of a mouth pronouncing the non-labials /aga/ and /aka/. As the consonants /b/ and /p/ are easily confused with the consonants /d/ and /t/ respectively [START_REF] Miller | An analysis of perceptual confusions among some English consonants[END_REF], and because these are visually very similar to the visemes /g/ and /k/ [START_REF] Fisher | Confusions among visually perceived consonants[END_REF], individuals sensitive to the McGurk effect will perceive /ada/ and /ata/ when listening to these incongruent stimuli [START_REF] Mcgurk | Hearing lips and seeing voices[END_REF]. Accordingly, the perception of /ada/ or /ata/ indicates that auditory and visual information is fused, whereas the perception of /aba/ or /apa/ indicates that auditory and vsual information is processed separately.

In our experiment, the synchronization of the McGurk stimuli was varied to manipulate the strength of the McGurk effect (i.e. the strength of AV binding). We hypothesize that the brain areas involved in the AV binding process will show brain activation patterns correlated to the strength of the McGurk effect. Brain activation patterns as measured using the BOLD response, were therefore correlated to the strength of the AV binding as indicated by the perceptual responses.

To use a McGurk response as evidence for the fusion of auditory and visual information, it is important that the auditory stimulus is easily discerned. If not, the 'changed' auditory percept could simply be caused by the absence of auditory information. To ensure the audibility of the relatively brief and weak consonants in the speech stimuli, we used a sparse sampling paradigm [START_REF] Eden | Utilizing hemodynamic delay and dispersion to detect fMRI signal change without auditory interference: the behavior interleaved gradients technique[END_REF][START_REF] Hall | Sparse' temporal sampling in auditory fMRI[END_REF]. Stimuli in this paradigm are presented in periods of rela-tive silence between scanning periods. This ensures that the auditory stimuli are not masked by scanning noise and that the brain activation obtained depends on the stimulus and not on the scanning noise.

Methods

Participants

Twenty healthy volunteers with normal hearing and normal or corrected-to-normal vision participated in this study. None of them reported neurological or language impairments. Four (two females) out of the initial pool of 20 individuals were excluded from the fMRI analysis because of the absence of the McGurk effect as revealed in the psychophysical study. One individual did not complete the scanning experiment and one individual was excluded because of excessive motion during scanning. We analyzed a total of 14 individuals aged between 22 and 45 years (five females). Before participa-tion, individuals signed an informed consent form. The experiment was approved by the local Medical Ethical Committee, Groningen.

Stimuli

Movies were made with a digital video camera [Panasonic 3 CCD 24 digital zoom (Panasonic, One Panasonic Way, Secausus, NJ07094, USA)] and external microphone (Sennheiser) in which a female Dutch speaker uttered the syllables /aba/, /aga/, /apa/ and /aka/. The video frame rate was 25 Hz and the sounds were digitized at a sampling rate of 48 kHz with 16-bit resolution. The movies were edited in Adobe Premiere (Adobe Premiere, 345 Park Avenue, San Jose, California, 95110-2704 USA). Incongruent stimuli were produced in which the audio signals of the /aba/ and /apa/ utterances were dubbed onto the videos of the speaker saying /aga/ and /aka/, respect-ively. Stimuli with AV delays ranging from -510 ms (audio leads video) to + 420 ms (video leads audio) were prepared. The duration of the acoustic syllables was around 730 ms. The total length of the AV movies was 2.00 s with the auditory utterance temporally centred. Only the mouth was shown. To avoid onset and offset information because of the opening and closing of the mouth, the movies started and ended with an open mouth.

The psychophysical experiment used 14 AV stimuli with different delays (-510, -390, -270, -190, -110,-70, -30, + 10, + 50, + 90, + 130, + 210,+ 290 and + 410 ms). AV stimuli were selected for the fMRI experi-ment that resulted in minimal (-510 and +410 ms), in-termediate (-110 and + 210 ms) and maximal (+ 50 ms) strength McGurk effects. Furthermore, audio-only (A) stimuli (speech sounds /aba/ or /apa/ combined with a blank screen with a fixation cross), and visual-only (V) stimuli (a mouth pronouncing /aga/ or /aka/) were used. The baseline stimulus was a fixation cross on the screen.

Stimuli were presented in the psychophysical and the fMRI experiments against a continuously present pink noise background of 40 dB SPL. Along with the use of ear mufflers, which reduced the scanner noise by approxi-mately 20 dB, this noise background was high enough to mask environmental noise during the relatively silent periods between the acquisitions. The signal-to-noise ratio of the auditory stimuli was 20 dB.

Psychophysical experiment

Individuals were seated in a sound insulated booth (Industrial Acoustic Company, Sohlweg 17, D-71372 Niederkrchten, Deutschland). Auditory stimuli were presented through headphones [Telephonics TDH-49P (Telephonics, 815 Broad Hollow Road, Farmingdale, New York 11735, USA)] and visual stimuli were pre-sented on a computer screen at 60 cm distance. The two AV stimuli with 14 different AV delays were presented seven times in pseudo-random order, resulting in a total of 196 stimulus presentations per session. Individuals had to indicate whether they heard the sound /aba/, /ada/, / apa/ or /ata/ by pressing corresponding keys on a keyboard. Each individual performed three sessions.

Functional magnetic resonance imaging experiment

Functional MR scans were acquired using a 3 T scanner (Philips Intera, Philips Medical Systems, Eindhoven, The Netherlands). Prior to the functional images, a T1-weighted anatomical scan of the entire brain was produced (160 slices; field of view = 256 mm; scanning matrix = 256 256; transverse slice orientation; repeti-tion time = 25 ms; echo time = 4.6 ms; flip angle = 301). Functional MR images were acquired using echo planar imaging; (46 slices; slice thickness = 3.5 mm; field of view = 224 mm; scanning matrix = 64 64; transverse slice orientation; repetition time = 20 s; echo time = 35 ms; flip angle = 901). Eight different stimulus conditions were used: 1 to 5 -AV with delays of -510, -110, 50, 210 and 410 ms; 6 -A; 7 -V; and 8 -baseline. Each experiment consisted of two functional series in which 74 volumes were scanned (18 volumes at baseline and 56 volumes where the seven remaining stimulus conditions were presented eight times).

To avoid interference with scanner noise during acoustic stimulus presentation, we applied the sparse sampling method [START_REF] Eden | Utilizing hemodynamic delay and dispersion to detect fMRI signal change without auditory interference: the behavior interleaved gradients technique[END_REF][START_REF] Hall | Sparse' temporal sampling in auditory fMRI[END_REF] with a relatively long repetition time of 20 s. Between two acquisitions, four stimuli of one stimulus condition were presented in pseudo-random order.

The entire experiment, including one anatomical scan and two functional series, lasted approximately 50 min. After each stimulus presentation, except after a baseline stimulus, individuals had to indicate what was said (/aba/, /ada/, /apa/ or /ata/) by pressing a button on a response box with their left or right index fingers, or their middle finger. The individuals had to respond within 1 s. During this second, a window with four buttons corresponding to the possible answers was displayed.

Visual stimuli were projected onto a screen located at the base of the scanner through a LCD projector (Barco G300j, President Kennedypark 35, 8500 Kortrijk, Belgie). Individuals viewed the movies through a mirror angled above the head. Auditory stimuli were presented through the headphones of the scanner's sound system.

Data analysis

The SPM2 software (http://www.fil.ion.ucl.ac.uk/spm) was used to process and analyze the data. The echo planar imaging images were realigned to the first volume in the run. These images were co-registered with the individuals' own anatomical images. They were normalized to the standard brain (MNI) and spatially smoothed using a Gaussian kernel with a full width half maximum of 10 mm.

For each individual, weighted effect sizes (con values) were obtained for all stimulus conditions (1-7) against baseline [START_REF] Gentilucci | Automatic audiovisual integration in speech perception[END_REF]. A block design was used for this analysis, without convolution with the hemodynamic response function, as sparse sampling was used. To correlate the individuals' McGurk responses with the brain activation levels, a second level analysis was applied. The con values for conditions 1-5 were correlated to the average McGurk responses across individuals (the regressor is shown in the last graph of Fig. 1, black symbols). We also analyzed the data using the individual mean McGurk responses as regressors (see Fig. 1, 1-14, black symbols). To the resulting t-map, a threshold of P = 0.001 (cluster size Z 20) was applied. For these regions, the con values per individual were extracted using the MARSBAR toolbox (http://marsbar. sourceforge.net).

Results

Psychophysics

The mean percentage of McGurk responses -that is the mean of the /ada/ responses following auditory /aba/ stimulation and the /ata/ responses after auditory /apa/ stimulation -clearly depends on AV synchronization (Fig. 1). The McGurk response percentages peak in the AV delay range of -30 to +130 ms. Four out of the initial pool of 20 individuals always heard /aba/ or /apa/, independently of the synchronization of the AV stimulus and were excluded from the fMRI experiment (data not shown in Fig. 1).

Functional magnetic resonance imaging Psychophysics

We analyzed the whole brain and selected those brain areas in which the BOLD response showed significant correlation with the McGurk response percentages (last graph of Fig. 1, black symbols). Positive correlations were not found. Negatively correlated BOLD responses were found in the supramarginal gyrus, the superior frontal gyrus, parts of the cerebellum and the thalamus (see Table 1 and Fig. 2). All these brain areas show decreased con values after synchronous AV stimulation (see stimulus 3 in Fig. 2b) compared to the maximum response after monosensory stimulation (see stimulus 6 or 7 in Fig. 2b). All con values, except for those in the thalamus, are however positive compared to the rest condition. We also analyzed the fMRI-data using the individual average McGurk response as a regressor (Fig. 1, 1-14, black symbols). The results of this analysis are comparable with the results described (not shown).

Discussion

Audiovisual binding as a function of audiovisual delay

The aim of this study was to correlate the brain activity patterns with a perceptual measure of AV binding as evoked by the McGurk effect. Our psychophysical experiment showed that four out of 20 individuals were insensitive to the McGurk effect. These individuals always perceived the auditory stimulus, /aba/ or /apa/, independently of AV synchronization, and apparently did not combine auditory and visual information in the identification task. Many other studies mention that typically 10-20% of individuals are insensitive to the McGurk effect (e.g. [START_REF] Munhall | Temporal constraints on the McGurk effect[END_REF][START_REF] Sekiyama | Auditory-visual speech perception examined by fMRI and PET[END_REF]) Gentilucci and Cattaneo [START_REF] Gentilucci | Automatic audiovisual integration in speech perception[END_REF] even report that over 50% of individuals are insensitive. Therefore, non-McGurk individuals form a substantial part of the normal population. In previous fMRI and PET studies that used the McGurk effect to study AV binding [START_REF] Sekiyama | Auditory-visual speech perception examined by fMRI and PET[END_REF][START_REF] Olson | A comparison of bound and unbound audio-visual information processing in the human cerebral cortex[END_REF][START_REF] Jones | Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect[END_REF], individuals were not screened to see whether they were sensitive to the McGurk effect or not, which might have resulted in contamination of these brain imaging data by including individuals who did not bind AV information.

The psychophysical data as shown in Fig. 1 are comparable with those described in the literature [START_REF] Munhall | Temporal constraints on the McGurk effect[END_REF][START_REF] Wassenhove Van | Temporal window of integration in auditory-visual speech perception[END_REF], with prevalent AV fusion over temporal asynchronies from -30 to + 130 ms. A window of AV integration of around 200 ms and with an asymmetrical prevalence for audio lags was also found in other studies, such as of a simultaneity judgment task [START_REF] Stone | When is now? Perception of simultaneity[END_REF] and in an electrophysiological study of the cat superior colliculus [START_REF] Meredith | Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors[END_REF]. This asymmetric temporal window of integration could be caused by the difference in speed between mechanotransduction and phototransduction, acoustic mechanotransduction being around 60 ms faster than phototransduction [START_REF] Wallace | Development of multisensory neurons and multisensory integration in cat superior colliculus[END_REF]. Adaptation of the central nervous system to differences in the speed of light and sound resulting in an audio lag for sources at a distance from the perceiver has also been proposed to account for the facilitated AV integration of stimuli with leading visual stimulation [START_REF] Meredith | Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors[END_REF].

Brain regions with BOLD responses correlated to AV binding

Our results show that the BOLD response is correlated to the strength of the McGurk effect in a select number of the brain areas: the supramarginal gyrus and superior frontal gyrus, thalamus and parts of the cerebellum. As the McGurk effect indicates AV binding, we hypothesize that these brain areas play a role in AV processing. Several other studies suggest that these brain areas are involved in AV or multisensory processing. The supramarginal gyrus are thought to be involved in AV speech perception [START_REF] Bernstein | Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing[END_REF][START_REF] Miller | Perceptual fusion and stimulus coincidence in the cross-modal integration of speech[END_REF], and the supramarginal gyrus and superior frontal gyrus in AV motion perception [START_REF] Baumann | Neural correlates of coherent audiovisual motion perception[END_REF], whereas several nuclei of the thalamus function as multimodal gateways of sensory information [START_REF] Komura | Auditory thalamus integrates visual inputs into behavioral gains[END_REF]. Further, the cerebellum is mentioned in many studies of AV processing and is thought to be activated during asynchrony detection [START_REF] Bushara | Neural correlates of auditory-visual stimulus onset asynchrony detection[END_REF] and AV speech perception [START_REF] Skipper | Listening to talking faces: motor cortical activation during speech perception[END_REF], for instance.

Decreased BOLD responses as a function of AV binding

Additivity and super-additivity are used as criteria to indicate AV integration sites [START_REF] Calvert | Multisensory integration: methodological approaches and emerging principles in the human brain[END_REF][START_REF] Laurienti | On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies[END_REF] in most brain imaging studies. We were, however, unable to find the brain areas in which the BOLD response positively correlated with the number of McGurk responses. On the other hand, our results are comparable with those of Jones and Callan [START_REF] Jones | Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect[END_REF], Miller and d'Esposito [START_REF] Miller | Perceptual fusion and stimulus coincidence in the cross-modal integration of speech[END_REF] and Bernstein et al. [START_REF] Bernstein | Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing[END_REF], who for example found in the supramarginal gyrus larger BOLD responses to temporally offset or incongruent stimuli than to synchronized or congruent AV stimulation. Bernstein hypothesized that in these brain areas temporally incongruent stimuli would give rise to a larger BOLD response because of the mismatch between auditory and visual information. However, as the monosensory A and V stimuli produce BOLD responses that are comparable with those produced by the offset AV stimuli in our study, it appears that the offset stimuli do not cause increased BOLD responses. Instead, the synchronous, integrated AV stimuli seem to produce decreased BOLD responses. Assuming that decreased BOLD responses represent decreased neuronal activity, this finding agrees with recent evidence based on single-cell recordings in cats, that activity is decreased in a substantial part of multisensory neurons in response to simultaneous multimodal stimulation [START_REF] Meredith | Crossmodal projections from somatosensory area SIV to the auditory field of the anterior ectosylvian sulcus (FAES) in Cat: further evidence for subthreshold forms of multisensory processing[END_REF]. The positive con values found after auditory-only, visual-only and AV stimulation can be explained by the assumption that these brain areas also contain neurons that are activated during AV or monosensory stimulation.

So far, we can only speculate on why we did not find a single brain area in which the BOLD response increased with AV binding. One difference between this study and many other studies of AV integration is that we used an active AV integration task, rather than a passive paradigm. There is evidence that attention demanding tasks can result in decreased brain activity levels in a network of brain areas [START_REF] Mckiernan | A parametric manipulation of factors affecting task-induced deactivation in functional neuroimaging[END_REF][START_REF] Shulman | Common blood flow changes across visual tasks: II. decreases in cerebral cortex[END_REF]. However, as the task and probably also attention did not vary with the different stimuli used in our study, the decreased BOLD response is probably not related to attention. Another important difference between our study and almost all other studies of AV integration is that we used a sparse sampling paradigm. Although we are convinced of the usefulness of sparse sampling because of the importance of a well-defined and easily discerned auditory stimulus, the price paid for using this paradigm is the reduction of statistical power and/or a longer duration for the scanning experiment.

Perhaps we were unable to identify relatively small positively-correlated BOLD responses because of the reduced statistical power of our experiment. However, it is clear that negativelycorrelated BOLD responses were much stronger than the possible positive correlations.

Conclusion

Two cortical brain areas, the supramarginal gyrus and the superior frontal gyrus, part of the thalamus and parts of the cerebellum showed activity patterns that correlated with the perception of the McGurk effect, used to indicate AV binding. BOLD responses decreased with increased AV binding in all these brain areas.

Figure. 1

 1 Figure. 1 Behavioral results for the 14 individual participants that were included in the fMRI experiment. Mean results for all individuals with standard deviations are shown in the last graph. Audiovisual (AV) delay in the synchronization of the AV stimuli is shown on the horizontal axis. The vertical axis represents the percentage of McGurk responses, that is, the mean with standard deviation of the /ada/ and /ata/ responses. Grey symbols represent the psychophysical experiment data. Black symbols represent the data recorded during the fMRI experiment.

Table 1 .

 1 Figure 2. (a) Brain areas in which the BOLD response showed a significant negative correlation with the number of McGurk responses (P = 0.001, uncorrected, cluster size Z 20). See Table 1 for details of brain areas 1-7. (b) Mean con values (weighted effect sizes versus baseline) in brain areas 1-7 for the seven stimuli used. Error bars indicate standard errors. Stimuli: 1 = AV -510 ms, 2 = AV -110 ms, 3 = AV50 ms, 4 = AV210 ms, 5 = AV410 ms, 6 = A, 7 = V.

Acknowledgements

The authors wish to thank the subjects for participating and Professor Pim van Dijk and the anonymous reviewers for their valuable comments on an earlier draft of this manuscript.