William N Havard

Jean-Pierre Chevrot

Laurent Besacier

Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech

In this paper, we study how word-like units are represented and activated in a recurrent neural model of visually grounded speech. The model used in our experiments is trained to project an image and its spoken description in a common representation space. We show that a recurrent model trained on spoken sentences implicitly segments its input into word-like units and reliably maps them to their correct visual referents. We introduce a methodology originating from linguistics to analyse the representation learned by neural networks -the gating paradigm -and show that the correct representation of a word is only activated if the network has access to first phoneme of the target word, suggesting that the network does not rely on a global acoustic pattern. Furthermore, we find out that not all speech frames (MFCC vectors in our case) play an equal role in the final encoded representation of a given word, but that some frames have a crucial effect on it. Finally, we suggest that word representation could be activated through a process of lexical competition.

Introduction

Neural models of Visually Grounded Speech (VGS) sparked interest in linguists and cognitive scientists as they are able to incorporate multiple modalities in a single network and allow the analysis of complex interactions between them. Analysing these models does not only help to understand their technological limitations, but may also yield insight on the cognitive processes at work in humans [START_REF] Dupoux | Cognitive science in the era of artificial intelligence: A roadmap for reverseengineering the infant language-learner[END_REF] who learn from contextually grounded speech utterances (either visually, haptically, socially, etc.). This is with this idea in mind that one of the first computational model of visually grounded word acquisition was introduced by [START_REF] Roy | Learning words from sights and sounds: a computational model[END_REF]. More recently, [START_REF] Harwath | Unsupervised learning of spoken language with visual context[END_REF] and Chrupała et al. (2017) were among the first to propose neural models integrating these two modalities.

While Chrupała et al. (2017) and [START_REF] Alishahi | Encoding of phonology in a recurrent neural model of grounded speech[END_REF] focused on analysing speech representations learnt by speech-image neural models from a phonological and semantic point of view, the present work focuses on lexical acquisition and the way speech utterances are segmented into lexical units by a neural model.

More precisely, we aim at understanding how word-like units are processed by a VGS architecture. First, we study if such models are robust to isolated word stimuli. As such networks are trained on raw speech utterances, robustness to isolated word stimuli would indicate that a segmentation process was implicitly carried out at training time. We also explore which factors influence the most such word recognition. In a second step, to better understand how individual words are activated by the network, we adapt the gating paradigm initially introduced to study human word recognition [START_REF] Franc ¸ois Grosjean | Spoken word recognition processes and the gating paradigm[END_REF] where our neural model is inputted with speech segments of increasing duration (word activation). Finally, as some linguistic models assume that the first phoneme of a target word activates all the words starting by the same phoneme, we investigate if such a pattern holds true for our neural model as well (word competition). As far as we know, no other study has examined patterns of word recognition, activation and competition in models of VGS.

This paper is organised as follows: section 2 presents related works and section 3 details our experimental material (data and model). Our contributions follow in section 4 (word recognition), section 5 (word activation) and section 6 (word competition). Section 7 concludes this work.

Related Work

In this section we explore what is known about word recognition in humans. We then review recent works related to the representation of lan-guage in VGS models. A few words are also said about modified inputs and adversarial attacks as they are related to the analysis methodology used in part of this work.

Word Recognition in Humans

Many psycholinguistic models try to account for how words are activated and recognised from fluent speech. The process of word recognition "requires matching the spoken input with mental representations associated with word candidates" [START_REF] Dahan | Chapter 8 -spoken word recognition[END_REF]. One of the first model trying to account for how humans recognise and extract words from fluent speech is the COHORT model by [START_REF] William D Marslen-Wilson | Processing interactions and lexical access during word recognition in continuous speech[END_REF]. In this model, word recognition proceeds in 3 steps: access, selection and integration. Access denotes the process by which a set of words (a cohort) becomes activated if their onsets are consistent with the perceived spoken input. As soon as a word form becomes inconsistent with the spoken input, it is removed from the initial cohort (selection phase). A word is deemed recognised as soon it is the last one standing in the cohort. Integration consists in checking if the word's syntactic and semantic properties are consistent with the rest of the utterance. However, COHORT supposes a full match between the perceived input and the word forms and does not account for word frequency in the access phase. REVISED COHORT [START_REF] William | Functional parallelism in spoken word-recognition[END_REF] later relaxed the constraints on the cohort formation to take into account these facts. There is no active competition per se between words in the COHORT model. That is, the strength of activation of a word does not depend on the value of the activation of the other words, but only on how well the internalised word form matches the perceived spoken input. TRACE [START_REF] James | The trace model of speech perception[END_REF]) is a connectionist model of spoken word recognition consisting of three layers of nodes, where each layer represents a particular linguistic unit (feature, phoneme and word). Layers are linked by exitatory connections (e.g. fricative feature node would activate /f/ phoneme node which would, in turn, activate words starting with this sound), and nodes within a layer are linked by inhibitory connections, thus inducing a real competition between activated words. Contrary to the COHORT model which does not allow words embedded in longer words to be activated, TRACE allows such activation. SHORTLIST [START_REF] Norris | Shortlist: a connectionist model of continuous speech recognition[END_REF] is another model which builds upon COHORT and TRACE by taking into consideration other features such as word stress. 1To sum up, models of spoken word recognition consider that a set of words matching to a certain extent the spoken input is simultaneously activated and these models involve at some point a form of competition between the set of activated words before reaching the stage of recognition.

2.2 Computational Models of VGS [START_REF] Roy | Learning words from sights and sounds: a computational model[END_REF] were among the first to propose a computational model, known as CELL, that integrates both speech and vision to study child language acquisition. However, CELL required both speech and images to be pre-processed, where canonical shapes were first extracted from images and further represented as histograms; and speech was discretised into phonemes. More recently, CNN-based VGS models [START_REF] Harwath | Unsupervised learning of spoken language with visual context[END_REF][START_REF] Harwath | Jointly discovering visual objects and spoken words from raw sensory input[END_REF][START_REF] Kamper | Semantic speech retrieval with a visually grounded model of untranscribed speech[END_REF] and RNN-based VGS models (Chrupała et al., 2017) which do not require speech to be discretised into sub-units were introduced. Chrupała et al. ( 2017) investigated how RNN-based models encode language, and showed such models tend to encode semantic information in higher layers, while form is better encoded in lower layers. [START_REF] Alishahi | Encoding of phonology in a recurrent neural model of grounded speech[END_REF] studied if such models capture phonological information and showed that some layers do capture such information more accuratly than others. [START_REF] Kádár | Representation of linguistic form and function in recurrent neural networks[END_REF] introduced omission scores to interpret the contribution of individual tokens in text-based VGS models. More recently, [START_REF] Havard | Models of visually grounded speech signal pay attention to nouns: A bilingual experiment on english and japanese[END_REF] studied the behaviour of attention in RNN-based VGS models and showed that these models tend to focus on nouns and could display language-specific patterns, such as focusing on particules when prompted with Japanese. Recently, [START_REF] Harwath | Jointly discovering visual objects and spoken words from raw sensory input[END_REF] showed that CNN-based models could reliably map word-like units to their visual referents, and [START_REF] Harwath | Towards visually grounded sub-word speech unit discovery[END_REF] showed such networks were sensitive to diphone transitions and that these were useful for the purpose of word recognition. However, none of the aforementioned works studied the process by which words are recognised and activated. This present work aims at bridging what is known about word activation and recognition in humans and the computations at work in VGS models.

Modified Inputs and Adversarial Attacks

As will be shown later, the gating method used in this article modifies the input stimulus to better understand the behaviour of the neural model.

We can draw a parallel with approaches recently introduced to show the vulnerability of deep networks to strategically modified samples (adversarial examples) and to detect their over-sensitivity and over-stability points. It was shown that imperceptible perturbations can fool the neural models to give false predictions. Inspired by the researches for images [START_REF] Su | One pixel attack for fooling deep neural networks[END_REF], efforts on attacking neural networks for NLP applications emerged recently (see [START_REF] Wei | Generating textual adversarial examples for deep learning models: A survey[END_REF] for a survey). However, while a lot of references can be found for textual adversarial examples, fewer papers addressed adversarial attacks for speech (we can however mention the work of [START_REF] Wu | Spoofing and countermeasures for speaker verification: a survey[END_REF] addressing spoofing attacks in speaker verification and of [START_REF] Carlini | Audio adversarial examples: Targeted attacks on speechto-text[END_REF] attacking Deep-Speech end-to-end ASR system).

Experimental Settings

Model Type

Even though the methodologies developed in this work could also be applied to CNN-based VGS models, the present work will solely focus on the analysis of the representations learned by a RNNbased VGS model. Indeed, from a cognitive perspective, RNN-based models are more realistic than CNN-based models as the speech signal -or in our case, a sequence of MFCC vectors -is sequentially processed from left-to-right, whereas in CNN-based models the network processes multiple frames at the same time. This will thus allow us to explore if RNN-based models display human-like behaviour or not.

Model Architecture

The model we use for our experiment is based on that of Chrupała et al. (2017) and later modified by [START_REF] Havard | Models of visually grounded speech signal pay attention to nouns: A bilingual experiment on english and japanese[END_REF]. It is trained to solve an image retrieval task: given a speech query, the model should retrieve the closest matching image. The model consists of two parts: an image encoder and a speech encoder. The image encoder takes VGG-16 pre-calculated vectors as input instead of raw images. It consists of a dense layer which reduces the 4096 dimensional VGG-16 input vector into a 512 dimensional vector which is then L2 normalised. The speech encoder takes 13 Mel Frequency Cepstral Coefficients (MFCC) vectors instead of raw speech.2 It consists of a convolutional layer (64 filters of length 6 and stride 3) followed by 5 stacked unidirectional GRU layers [START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF], with 512 units each. Two attention mechanisms [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] are used: one after the 1 st recurrent layer and one after the 5 th recurrent layer. The final vector produced by the speech encoder corresponds to the dot product of the weighted vectors outputted by each attention mechanism. The model is trained to minimise the following triplet loss function as implemented by Chrupała et al. (2017):

L(u, i, α) = u,i u max[0, α + d(u, i) -d(u , i)] + i max[0, α + d(u, i) -d(u, i )]
(1) The loss function encourages the network to minimise the cosine distance d between an image i and its corresponding spoken description u by a given margin α while maximising the distance between mismatching image/utterance pairs. For our experiments, we set α = 0.2.

Data

The data set used for our experiments is based on MSCOCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]. MSCOCO is a data set used to train computer vision systems, and features annotated images, each paired with 5 human written descriptions in English. MSCOCO's images where selected so that the images would contain instances of 80 possible object categories. We trained our model on the spoken extension introduced by Chrupała et al. (2017). This extension provides spoken version of the human written captions. It is worth mentioning that this extended data set features synthetic speech (female voice generated using Google's Text-To-Speech (TTS) system) and not real human speech.

Model Training and Results

We trained our model for 15 epochs with Adam optimiser and an initial learning rate of 0.0002. The training set comprises 113,287 images with 5 spoken captions per image. Validation and test set comprise 5000 images each. 3 Model is evaluated in term of Recall@k (R@k) and median rank r. That is, given a spoken query, which corresponds to a full utterance, we evaluate the model's ability to rank the unique paired image in the top k images. We obtain a r of 28.

Full results are shown in Table 1. Even though our results are lower than the original implementation by Chrupała et al. (2017), our model still performs far above chance level, showing it did learn how to map an image and its spoken description.

Model

R@1 R@5 R@10 r Synth. COCO 0.056 0.182 0.284 28 Table 1: Recall at 1, 5, and 10 results and median rank r on a speech-image retrieval task (test part of our datasets with 5k images). Chrupała et al. (2017) with RHN reports median rank r = 13. Chance median rank r is 2500.5.

Word Recognition

Harwath et al. ( 2018) observed that CNN-based models can reliably map word-like units to their corresponding visual reference. Chrupała et al. (2017) and more recenlty [START_REF] Merkx | Language Learning Using Speech to Image Retrieval[END_REF] showed that RNN-based utterance embeddings contain information about individual words, but did not show for what type of words this behaviour holds true and if the model had learnt to map these individual words to their visual referents. [START_REF] Havard | Models of visually grounded speech signal pay attention to nouns: A bilingual experiment on english and japanese[END_REF] showed that the attention mechanism of RNN-based VGS models tends to focus on the end of words that correspond to the main concept of the target image. This suggests that such models are able to isolate the target word forms from fluent speech and thus segment their inputs into sub-units. In the following experiment we test if a RNN-based VGS network can reliably map isolated word-like units to their visual referents and explore the factors that could influence such mapping.

Isolated Word Mapping

We selected a set of 80 words corresponding to the name of 80 object categories in the MSCOCO data set. 4 We expect our model to be very efficient with the selected 80 words, as these are the main objects featured in MSCOCO. We generated speech signals for these 80 isolated words using Google's TTS system and then extracted MFCC features for each of the generated words. We evaluate the ability of the model to rank images containing an object instance corresponding to the target word among the first 10 images (P@10).5 Contrary to (Chrupała et al., 2017) who uses Recall@k, we use Precision@k as there are several images that correspond to a single target word. It is to be noted that at training time, the network was only given full captions and not isolated words. Thus, if the network is able to retrieve images featuring instances of the target word, it shows that implicit segmentation was carried out at training time.

Results are shown in Figure 1. 40 words out of the 80 target words have a P@10≥ 0.8. This shows that the network is able to map isolated words to their visual referent despite never having seen them in isolation and that the network implicitly segmented its input into sub-units.

Factors Influencing Word Mapping

We explore here the factors that could come at play in the recognition of isolated words. We explore 2 types of factors: speech related factors and image related factors. For the former we consider

Word Activation

In this section we describe how individual words are activated by the network. To do so, we perform an ablation experiment (similar to that of [START_REF] Franc ¸ois Grosjean | Spoken word recognition processes and the gating paradigm[END_REF] which was conducted on humans) where the neural model is inputted only with a truncated version of the 80 target words (see Section 5.1). Such a method is also called gating in the literature.

Gating

The gating paradigm "involves the repeated presentation of a spoken stimulus (in this case, a word) such that its duration from onset is increased with each successive presentation" (Cotton and Grosjean, 1984). In our case, it means the neural model is fed with truncated version of a target word, each truncated version comprising a larger part of the target word. Truncation is either done left-to-right (model only has access to the end of the word) or right-to-left (model only has access to the beginning of the word). Truncation is operated on the MFCC vectors computed for each individual word, meaning that MFCC vectors are iteratively removed either from the beginning of the word or the end of the word, but not from both sides at the same time. Each truncated version of the word is then fed to the speech encoder which outputs an embedding vector. As in our previous experiment, model is evaluated in terms of P@10.

COHORT model, in its initial version [START_REF] William | Functional parallelism in spoken word-recognition[END_REF], stipulates that word onset plays a crucial role in word recognition whereas other models of spoken word recognition give less importance to word onset. This importance of exact word onset matching was later revised in later CO-HORT models. The aim of this experiment is to test whether word onset plays a role in word recognition for the network or not. If it is the case, we expect the network to fail recovering images of the target word if the word is truncated left-to-right.

Figure 2a shows evolution of P@10 averaged over the 80 test words. As can be seen from the graph, precision evolves differently according to which part of the word was truncated. When the target words are truncated left-to-right, precision drops quickly. However, when truncated right-toleft, precision remains high before gradually dropping. These results show that the model is robust to truncation when it is carried out right-to-left but not when it is carried out left-to-right. Figure 2b shows the evolution of P@10 for one of the target words ("giraffe"). When MFCC vectors corresponding to the first phoneme are removed (/Ã/), precision plummets from 1 to 0. However, when MFCC vectors belonging to the end of the word are removed, precision plateaus at 1 until /Ç/ is reached and then plunges to 0. This shows the model successfully retrieved giraffe images when only prompted with /ÃÇ/ but not when prompted with /Çaef/ even though the latter comprises a longer part of the target word.

These results suggest that the model does not rely on a vague acoustic pattern to activate the semantic representation of a given concept, but needs to have access to the first phoneme in order to yield an appropriate representation.

Activated Pseudo-Words

Such ablation experiments also enables us to infer on what units the network relies to make its predictions. Indeed, Figure 3 are the pseudo-words that were internalised by the network for the word "baseball bat". When truncation is done left-to-right (blue curve), we notice that at the beginning precision is quite high (≈ 0.6), then reaches 0 when only /O:lbaet/ is left, but suddenly increases up to 0.9 when the only part left is /baet/. This suggests that the network mapped both "baseball bat" as a whole and "bat" as referring to the same object. We observed the same pattern for the word "fire hydrant" where both "fire hydrant" and "hydrant" are mapped to the same object.

However, Figure 2b shows that when only prompted with /ÃÇ/ the network manages to find pictures of giraffes. This suggests that the pseudowords internalised by the network could be /ÃÇaef/ as a whole but might also be /ÃÇ/. We thus need to take caution when stating that the network has isolated words, as the words internalised by the network might not always match the human gold reference.

Gradual or Abrupt Activation?

Figure 2b shows that removing or adding one MFCC vector may yield large differences in the network performance. Precision decreases steeply and not steadily. This suggests that little acoustic differences yield wide differences in the final representation. Thus, in this section we analyse how representation is being constructed over time and explore if some MFCC vectors play a more important role than others in the activation of the final representation.

We progressively let the network see more and more of the MFCC vectors composing the word, iteratively feeding it with MFCC vectors starting from the beginning of the word until the network Figure 4: Figure 4a shows evolution of the cosine similarity between the embeddings produced for each truncated version of the target word and the embedding for the full word using a model with randomly initialised weights.

Figure 4c shows the same measure with the embeddings produced by a trained model. Figure 4b shows peaks indicating the inflection points of curve 4a (green) and 4c (red). For our experiments, we only considered inflection point to be significant if the resulting peak was higher than 0.025 (blue).

has had access to the full word. We then compute the cosine similarity between the embedding computed for each of the truncated version of the word and the embedding corresponding to the full word. The closer the cosine similarity is to 1, the more similar the two representations are. Thus, if each MFCC vector equally contributes to the final representation of the word, we expect cosine similarity to evolve linearly. However, if some MFCC vectors have a determining factor in the final representation we expect cosine similarity to evolve in steps rather than linearly. To detect steps that could occur in the evolution of cosine similarity, we approximate its derivative by computing first order difference. High steps should thus translate into peaks (e.g. Figure 4b). We compute the evolution of cosine similarity for the 80 target words encoded with the best trained model (e.g. Figure 4c) and also consider a baseline evolution by encoding the 80 target words with an untrained model (e.g. Figure 4a). 6 To avoid micro-steps of yielding peaks and thus creating noise, we smooth cosine evolution curves with a gaussian filter. We consider peaks higher than 0.025 as translating a high step in the evolution of cosine similarity. On average, they are 1.35 peaks per word for the trained model against 0.1 peak per word for our baseline condition (untrained model), showing that cosine evolution is linear in the latter but not in the former. Thus, in our baseline condition (untrained model), each MFCC vector equally contributes to the final representation, whereas in our trained model some MFCC vectors are more de-6 Thus consisting only of randomly initialised weights cisive for the final representation than others. Indeed, some MFCC vectors trigger a high step in the cosine evolution suggesting that the embedding suddenly gets closer to its final value. Figure 4c shows the evolution of the cosine similarity for the word "giraffe". As it can be seen, cosine similarity does not tend linearly towards 1, but rather evolves in steps. Adding the MFCC vectors corresponding to the transition from /Ç/ to /ae/ triggers a large difference in the embedding as the cosine similarity suddenly jumps to a higher value, showing it is getting closer to its final representation. However, cosine similarity plateaus once /ae/ is reached up until final silence, suggesting the final /aef/ plays little to no role in the final representation of the word.

Word Competition

As presented in Section 2.1, some linguistic models assume that the first phoneme of target word activates all the words starting by the same phoneme. The words that are activated but which do not correspond to the target words are called "competitors". As the listener perceives more and more of the target word some competitors are deactivated as they do not match what is being perceived. For example, considering the following lexicon: /beIbi/ (baby), /beIzIk/ (basic), and /beIzbO:l/ (baseball), the first sound /b/ would activate all three words, once /beIz/ is reached, "baby" would not be considered a competitor anymore, and once /beIzb/ is reached the only word activated would be "baseball" as it is the only word whose beginning corresponds the perceived sounds. Figure 5: Illustration of lexical competition between 5a "meat" and "meter" (target) and 5b "plate" and "player" (target). Numbers in 1 st x-axis corresponds to the number of MFCC frames; 2 nd x-axis corresponds to timealigned phonemic transcription of the target word; y-axis shows number of images for which at least one caption (out of 5) contains the target or competitor word. Vertical colour bars are projection of phoneme boundaries of the target word. Horizontal colour bars show chance score for each word (<2).

We test if the network displays such lexical competition patterns. To do so, we select a set of 29 word pairs according to the following criteria: i) words should at least appear 400 times or more in the captions of the training set, so that the network would have been able to learn a mapping between this word and its referent; ii) words forming a pair should at least start with the same phoneme;7 iii) words should not be synonyms and clearly refer to a different visual object (thus excluding pairs such as "motorcycle" and "motorbike").

For each word pair, we select the longest word as target and progressively let the network see more and more of the MFCC vectors composing this word (as in Section 5.3). At each time step the network produces an embedding, which we use to rank the images from the closest matching image to the least matching image. 8 Then, for the 50 closest matching images, we check if at least one of the caption contains either the target word or the competitor. As the competitor is embedded in the longer word, we expect the network to produce an embedding close to that of the competitor at the beginning and then when the acoustic signal does not match the competitor anymore, we expect the network to be able to find only the target word.

Figure 5 shows example of competition between two word pairs. Figure 5a shows that when prompted with the beginning of the word "meter" /mi:t/ the representation activated by the network is close to that of "meat" as the closest maching images's captions contain the word "meat". Representation of the word "meter" seems to be activated only when /Ç/ is reached, and consequently triggers the total deactivation of the word "meat". Figure 5b displays a different pattern. As in the previous example, the beginning of the word "player" /pleI/ triggers the activation of the word "plate". When /Ç/ is reached, the target word becomes activated and competitor "plate" starts to deactivate. However, the deactivation is not full, so that when the whole word "player" is entirely processed by the network, the word "plate" still remains highly activated. (REVISED) COHORT [START_REF] William D Marslen-Wilson | Processing interactions and lexical access during word recognition in continuous speech[END_REF][START_REF] William | Functional parallelism in spoken word-recognition[END_REF] and TRACE (Mc-Clelland and Elman, 1986) both state that competing words are all activated at the same time, that is when the first phoneme is perceived. However here, the two competing words are activated sequentially but not at the same time. Also, in some cases, competing words that do not match the input anymore still remain highly activated.

In this paper, we analysed the behaviour of a model of VGS and showed that a RNN-based model of VGS is able to map isolated words to their visual referents. This result is in line with previous results, such as that of [START_REF] Harwath | Towards visually grounded sub-word speech unit discovery[END_REF] which uses a CNN-based network. This shows that such models perform an implicit segmentation of the spoken input in order to extract the target words. However, the mechanism by which implicit segmentation is carried out and what cues are being used is still to be explained. We also demonstrated that not all words are equally well recognised and showed that word frequency and number of neighbouring object in an image partly explain this phenomenon.

Also, we introduced a methodology originating from linguistics to analyse the representation learned by neural networks: the gating paradigm. This enabled us to show that the beginning of a word can activate the representation of a given concept (e.g. /ÃÇ/ for "giraffe"). We explain this by the fact that the network has to handle a very small lexicon, where word forms rarely overlap and thus the network needs not see the full word to make its decision. More importantly, we showed that the network needs to have access to the first phoneme in order to activate the representation of the target word, thus showing that it does not respond to a vague acoustic pattern. Word onsets thus play a crucial role in the process of word activation and recognition for our network. Though word onsets are also important for humans, they are not as crucial as for our network. Indeed, humans are able to recover the missing information. In future work, we would like to test if sentential context has an effect in word recognition. We also demonstrated that our model is able to map multiple pseudo-words to the same referent such as humans do (Section 5.2). However, it is not clear how and when acoustics interface with meaning and this still remains an open question.

Finally, we showed that there is a form of lexical competition in the network. Indeed, small words embedded in longer words are activated. However, we showed that, contrary to humans where words sharing the same beginning are all activated at the same time, words are activated sequentially by the network. Also, some stay partially activated eventhough the input does not match that of the activated word.

Ultimately, we would like to highlight the fact that the gating paradigm could also by applied to understand the temporal dynamics of the representations learned by other speech architectures such as those used in speech recognition for instance.

A.1 Precision@10 for the 80 test words

(c) (d) (e) (f) (g) (h) (i) (j) (k) (l)
Figure: Evolution of Precision@10 for each ablation step for the words (a) "fork", (b) "microwave", (c) "motorcycle", (d) "pizza", (e) "scissors", (f) "sink", (g) "surfboard, (h) "toilet", (i) "traffic light", (j) "train", (k) "wine glass", (l) "zebra". It should be noted that "fork" and "microwave" do not display the same behaviour as the other words, this could be explained by the fact these two words both have a very low Precision@10. Figure 13: Competition plots between 13a "baseball" and "baby" and 13b "bed" and "bench". Figure 14: Competition plots between 14a "glass" and "grass" and 14b "kitchen" and "keyboard". Figure 15: Competition plots between 15a "meat" and "mirror" and 15b "mirror" and "meter". Figure 16: Competition plots between 16a "plane" and "plate" and 16b "shore" and "shower".

A.3 Activation Sample

Figure 1 :

 1 Figure 1: Precision@10 for the 80 isolated words corresponding to MSCOCO categories.

  Figure 2: 2a Evolution of Precision@10 averaged over 80 test words as a function of the percentage of MFCC vector removed for each word. 2b Evolution of Precision@10 for each ablation step of the word "giraffe", with time-aligned phonemic transcription /ÃÇaef/ at the bottom. "SIL" signals silences. For both 2a and 2b, blue line displays scores when ablation was carried out left-to-right, meaning that at any given part on the blue curve, model has only had access to the rightmost part of the word. (e.g. /Çaef/ without initial /Ã/). Red line displays scores when ablation was carried out right-to-left, meaning that at any given part on the red curve, model has only had access to the leftmost part of the word. (e.g. /ÃÇ/ without final /aef/ ).

Figure 3 :

 3 Figure 3: Evolution of P@10 for each ablation step of the word "baseball bat" with time aligned phonemic transcription /beizbO:l#baet/ at the bottom.

Figure

  Figure: Precision@10 for each of the 80 test words

Figure 1 :Figure 2 :Figure 3 :Figure 4 :Figure 5 :Figure 6 :Figure 7 :Figure 8 :Figure 9 :Figure 10 :Figure 11 :Figure 12 :

 123456789101112 Figure 1: Evolution of cosine similarity for the word "fork". Figure 1b shows peaks indicating the inflection points of curve 1a (untrained model, green) and 1c (trained model, red)

Table 2 :

 2 Factors influencing word recognition performance in our model. Spearman's ρ between Precision@10 and mentioned variables as well as p-value. Avg. Size). Results are shown in Table 2. We observe a weak negative correlation between precison and average number of neighbouring objects, thus suggesting that objects that have a low number of neighbouring objects are better recognised by the network. It also seems that bigger objects yield better precison than smaller objects as we observe a weak positive correlation. Word frequency seems to play an important role as we observe a moderate positive correlation. However, we observe no correlation between precison and the length of the target words nor with object frequency in the images. Correlation values, however, remain relatively low, suggesting some other factors could also influence word recognition.

			Spearman's ρ	p-value	Effect
		Avg. Neighbour	-0.3906	0.0003 ***	Weak
	Images	Avg. Size	0.3154	0.0043 **	Weak
		Avg. Freq	0.1187	0.4675	None
	Text	Word Freq. # Syllables	0.5514 -0.1211	1.148e-07 *** Moderate 0.2844 None
	word frequency (Word Freq.) and word length (#
	syllables). Concerning image related factors we
	consider object instances frequency in the images
	(Avg. Freq.), average number of neighbouring ob-
	ject instances (Avg. Neighbour), average area of
	each object (

For a review of spoken word recognition model, reader can consult[START_REF] Dahan | Chapter 8 -spoken word recognition[END_REF] and[START_REF] Weber | Models of spoken-word recognition[END_REF]

12 mel frequency cepstral coefficients + log of total frame energy, vectors extracted every 10ms on a 25ms window

The train/dev/test correspond to those used in(Chrupała et al., 2017)

List available at https://github.com/ amikelive/coco-labels/blob/master/ coco-labels-2014_2017.txt

Evaluation is performed on the test set containing 5000 images

Phonemic transcription found in CMU Pronouncing Dictionnary was used

 8 That is, we compute the cosine distance between the embedding produced at time step t and all the images (5000) of our collection.

Acknowledgments

This work was supported by grants from Neu-roCoG IDEX UGA as part of the "Investissements d'avenir" program (ANR-15-IDEX-02).


Farouk Frigui
email: ffrigui@enit.fr

Jean-Pierre Faye

Carmen Martin

Olivier Dalverny

François Peres

Sébastien Judenherc

APPLICATION OF A COMPLETE STRUCTURAL HEALTH MONITORING CHAIN ON THE CANADIAN RIVIÈRE-AUX-MULETS BRIDGE NUMERICAL MODEL SUBJECTED TO A SEISMIC LOAD

Keywords: Structural Health Monitoring (SHM), Operational Modal Analysis (OMA), dynamic behavior, seismic load

The surveillance of civil engineering structures has become increasingly important over the last decade. Nowadays, several methods and means of Structural Health Monitoring (SHM) exist. They help to detect, localize and quantify damage, and predict the remaining service life of structures. One of the major challenges in the SHM field of civil engineering structures is to define a global and reliable methodology of damage detection and localization. In this paper, a complete chain of surveillance is set up on a three-dimensional Finite Element (FE) model of the Canadian Rivière-Aux-Mulets bridge, created with the ABAQUS software. The structure is damaged by a true seismic signal and a set of Vibration-Based Damage Detection Methods (VBDDM) is applied to detect and to localize the damage.

INTRODUCTION

Structural damage in civil engineering is essentially related to the environment where the structure is located. In fact, temperature change, aging of materials, human activities and natural disasters have a direct influence on the performance of structures. Therefore, civil engineering supervision has become very important over the last decade. Today, there are several methods of monitoring allowing the understanding of the structure's dynamic behaviour. These methods, also called Vibration-Based Damage Detection Methods (VBDDM), consist in the monitoring of the temporal evolution of dynamic characteristics such as eigen-frequencies and mode shapes [START_REF] Ndambi | Damage assessment in reinforced concrete beams using eigenfrequencies and mode shape derivatives[END_REF]. These dynamic characteristics can be identified on a real structure from its output response using Operational Modal Analysis (OMA) techniques, under white noise assumption.

In this context, we present the monitoring of the numerical model of the Canadian Rivièreaux-Mulets bridge subjected to a seismic load. The experimental SHM chain was carried out numerically. OMA algorithms were implemented in MATLAB to identify eigen-frequencies and mode shapes from these responses: the Stochastic Subspace Identification (SSI-COV) method was used to identify eigen-frequencies and the Frequency Domain Decomposition (FDD) was used to identify mode shapes. Identification results were compared to that of Abaqus. Finally, the detection and the localization of damages were made by applying the VBDDM methods, namely: eigen-frequencies method (∆݂ሻ, Modal Assurance Criterion (MAC), Mode Shape Curvatures Method (MSCM), Curvature Damage Factor (CDF) and the Flexibility Curvature method (FC).

DAMAGE DETECTION AND LOCALIZATION:

The surveillance methods, most commonly used in civil engineering, are based on the relation that exists between the change of the physical properties, and the variations of dynamic characteristics: variations of eigen-frequencies, mode shapes and damping. Thanks to this causal link, monitoring the evolution of these characteristics represents an accurate method of health monitoring [START_REF] Carden | Vibration based condition monitoring: a review[END_REF]. These techniques help choosing actions of rehabilitation on damaged structures, leading consequently to an optimization of maintenance costs. This principle has received considerable attention during the past two decades. However, the problem lies in the setup of a correct correlation between the variations of the dynamic characteristics, the appearance of the damage and its location.

Here in, we present a non-exhaustive list of detection and localization methods commonly used in civil engineering. The implementation, advantages and disadvantages of each method are presented.

Eigen-frequencies changes (∆݂ሻ

The damage induces a change in the structure's behaviour, particularly a shift of the eigenfrequencies. This dynamic characteristic reflects the overall behaviour of the structure. Thus, the monitoring of the frequencies is a sensitive damage indicator. Eigen-frequencies method is computed as follows [START_REF] Salawu | Detection of structural damage through changes in frequency: a review[END_REF]:

∆f ൌ f ୧ ୳ െ f ୧ ୢ f ୧ ୳ (1)
Where ݂ ௨ and ݂ ௗ are respectively the eigen-frequencies of the i th mode. u and d denote respectively the undamaged and the damaged state.

Since the eigen-frequency is an intrinsic property of the structure and no spatial information are required for its application, this technique addresses the overall structure and, as a result, allows only detection of damage [START_REF] Farrar | Vibration-based structural damage identification[END_REF].

Modal Assurance Criterion (MAC)

Since the mode shapes are sensitive to the structure's state, their follow-up allows detecting damages. In order to compare the mode shapes of a healthy structure to those of the damaged one, the MAC can be used. It's computed as follows [START_REF] Allemang | A correlation coefficient for modal vector analysis[END_REF]:

MAC ୨୩ ൌ ሾ∑ ሾᴪ ୳ ሿ ୧ ୨ ሾᴪ ୢ ሿ ୧ ୬ ୧ୀଵ ሿ ଶ ∑ ሺሾᴪ ୳ ሿ ୧ ୨ ሻ ଶ ୬ ୧ୀଵ ∑ ሺሾᴪ ୢ ሿ ୧ ୩ ሻ ଶ ୬ ୧ୀଵ (2)
With ᴪ ௨ and ᴪ ௗ are respectively the undamaged and the damaged sets of mode shapes. n is the number of modes.

MAC is used to determine the similarity of two mode shapes. If the mode shapes are identical, then MAC will have a value of 1. If the mode shapes are different due to the damage, the MAC value will be less than 1 [START_REF] Pastor | Modal assurance criterion[END_REF]. The most interesting values are those of the diagonal because they reflect the correlation between the shapes of the same mode. This method has two major disadvantages. In the first place, for low damage and according to its position, it can be detected by the MAC method using higher order modes [START_REF] Lam | Localization of damaged structural connections based on experimental modal and sensitivity analysis[END_REF]. These modes are more sensitive to damage and difficult to identify experimentally. In the second place, it only allows detection of damages without localization.

Mode Shape Curvature Method (MSCM)

This method is very sensitive to small disturbances caused by damages [START_REF] Salane | Identification of modal properties of bridges[END_REF]. It can be computed using the central difference approximation as follows [START_REF] Pandey | Damage detection from changes in curvature mode shapes[END_REF]:

ᴪ " ୧,୨ ൌ ሺᴪ ୧ାଵ,୨ െ 2ᴪ ୧,୨ ᴪ ୧ିଵ,୨ ሻ h ଶ (3)
Δᴪ " ୧,୨ ൌ |ᴪ " ୧,୨ ୳ െ ᴪ " ୧,୨ ୢ | (4)
Where ᴪ , and ᴪ " , are respectively the displacement and the mode shape curvature at the ݅ ௧ node and ݆ ௧ mode. h is the distance between two consecutive measurement nodes. u and d denote respectively the undamaged and damaged structure.

Unlike the methods presented above, this technique allows to locate the damage. However, the choice of the mode is very important. Indeed, some modes are less sensitive to damage than others and can induce misleading information (Foti, 2013)

Curvature Damage Factor (CDF)

In order to avoid the problem of choosing an appropriate mode in MSCM and to reach global information concerning the damage, one can use the Curvature Damage Factor (CDF) [START_REF] Wahab | Damage detection in bridges using modal curvatures: application to a real damage scenario[END_REF]. It consists in averaging the variations of the mode shape curvature with respect to the number of considered modes N:

CDF ൌ ∑ |ᴪ " ୧,୨ ୳ െ ᴪ " ୧,୨ ୢ | ୬ୀଵ N (5)

Flexibility Curvature (FC)

Damage in the structure induces and increases in its flexibility. Thus, follow-up flexibility curvature can allow damage detection and localization. Flexibility matrix can be computed as follows [START_REF] Pandey | Experimental verification of flexibility difference method for locating damage in structures[END_REF]:

F ൌ 1 w ୧ ଶ ᴪ ୧ ᴪ ୧ ୲ ୬ ୧ୀଵ (6)
Where ݓ is the ݅ ௧ eigen-frequency and ᴪ is the mode shape matrix mass-normalized to unity (ᴪ ௧ ܯᴪ ൌ 1ሻ.

The flexibility converges rapidly by increasing the frequency, thus a few lower modes provide a good estimation of the flexibility matrix. Generally, the first two modes are sufficient.

The flexibility curvature method is computed following the steps below:

ߛ ఫ ᇱᇱ തതതത ൌ ఊ ണషభ തതതതതതାఊ ഥ ೕశభ ିଶఊ ഥ ೕ మ (Zhang, 1995) (9)
With ܨ ௨ is the initial flexibility matrix corresponding to the undamaged structure and ܨ ௗ is the final flexibility matrix corresponding to the damaged structure. h is the distance between two consecutive measurement nodes.

OPERATIONAL MODAL ANALYSIS (OMA) TECHNIQUES:

The SHM techniques, previously presented, use natural frequencies and mode shapes. These characteristics can be identified experimentally using Operational Modal Analysis (OMA) methods [START_REF] Cunha | From input-output to output-only modal identification of civil engineering structures[END_REF].

In this section, we present two methods essentially: the Stochastic Subspace Identification technique (SSI) for the identification of eigen-frequencies and the Frequency Domain Decomposition (FDD) for the identification of mode shapes. Both techniques were chosen for their efficiency and robustness [START_REF] Greiner | Operational modal analysis and its application for SOFIA telescope assembly vibration measurements[END_REF][START_REF] Ghalishooyan | Operational Modal Analysis Techniques and their Theoretical and Practical Aspects: A Comprehensive Review and Introduction[END_REF].

The Stochastic Subspace Identification (SSI):

In the case of a time-invariant linear dynamic model, the state-space model in discretized domain can be written as [START_REF] Basseville | Output-only subspace-based structural identification: from theory to industrial testing practice[END_REF]:

x ሺ୩ାଵሻ ൌ Ax ሺ୩ሻ w ሺ୩ሻ y ሺ୩ሻ ൌ Cx ሺ୩ሻ q ሺ୩ሻ ( 10
)
Where ݔ ሺାଵሻ is the (2n×1) state vector at the time instant (k+1)∆T, ∆T is the sampling period, ݕ ሺሻ is the (l ×1) output vector at the time instant k∆T. A is the transition matrix, C is the observation matrix. w ϵ ܴ ଶ×ଵ and q ϵ ܴ ×ଵ are respectively the process and the measurement noises and are assumed to be white noises.

The transition matrix A contains all the modal information. Indeed, its eigenvalues ߣ are related to the eigen-frequencies through the next equation [START_REF] Peeters | Stochastic subspace techniques applied to parameter identification of civil engineering structures[END_REF][START_REF] Peeters | Reference based stochastic subspace identification in civil engineering[END_REF]:

f ୧ ൌ ඨ Re ൬ ݈݊ ሺλ ୧ ሻ ΔT ൰ ଶ Im ൬ ݈݊ ሺλ ୧ ሻ ΔT ൰ ଶ 2π ൘ (11)
Where Re and Im denote respectively the real part and the imaginary part.

The purpose of the SSI algorithm is to identify the transition matrix A from the structure's response. This can be achieved using the covariance matrix of the outputs ݕ ሺାሻ and ݕ ሺሻ . It can be written as [START_REF] Xie | Modal parameter identification of flexible spacecraft using the covariance-driven stochastic subspace identification (SSI-COV) method[END_REF]:

ᴧ ୧ ൌ CA ୧ିଵ G (12)
Where ܩ ൌ ߑܣ ௫௫ ܥ ௧ ܵ, ߑ ௫௫ is the covariance matrix of the state vector ݔ ሺሻ and ܵ is the covariance matrix of ݓ ሺሻ and ݍ ሺሻ . t denotes the transpose operator.

The SSI's algorithm is setup following these steps (Kuts, 2016):

• Step 1: Gathering the covariance matrices of the outputs into the Hankel block matrix H as follows:

H ൌ ൦ CA G CA ଵ G … CA ୯ିଵ G CA ଵ G ⋮ ⋮ CA ୯ G ⋮ ⋮ ⋮ ⋮ CA ୮ G CA ୮ାଵ G … CA ୮ା୯ିଵ G ൪ (13)
•

Step 2: Decomposition of the Hankel block matrix H into three matrices U, S and V using the Single Value Decomposition (SVD):

H ൌ USV ୲ ൌ U ଵ S ଵ .ହ S ଵ .ହ V ଵ ୲ ൌ O * K ( 14
)
Where O is the observability matrix and K is the controllability matrix. • Step 3: Identification of the observability matrix O using equation ( 14)

O ൌ U ଵ S ଵ .ହ (15)
• Step 4: Computing the shifted matrices ܱ ↑ and ܱ ↓ . ܱ ↑ is obtained by removing the last block row and ܱ ↓ is obtained by removing the first block row of the observability matrix O as follows:

O ↑ ൌ ۉ ۈ ۇ C CA CA ଶ ⋮ CA ୮ିଵ ی ۋ ۊ , O ↓ ൌ ۉ ۈ ۇ CA CA ଶ ⋮ ⋮ CA ୮ ی ۋ ۊ (16)
• Step 5: Identification of the transition matrix

A A ൌ O ↑ O ↓ # (17)
Where # denotes the Moore-Penrose pseudo-inverse.

• Step 6: Identification of eigen-frequencies using equation [START_REF] Foti | Dynamic identification techniques to numerically detect the structural damage[END_REF].

The Frequency Domain Decomposition (FDD):

Under assumption of a white noise excitation, mode shapes can be estimated from the spectral density [START_REF] Brincker | Modal identification from ambient responses using frequency domain decomposition[END_REF]. The inputs ݔሺݐሻ and the outputs ݕሺݐሻ are related through the next equation:

G ୷୷ ሺjwሻ ൌ ሾFRFሺjwሻሿሾG ୶୶ ሺjwሻሿሾFRFሺjwሻሿ ୲ (18)
Where ܩ ௫௫ ሺ݆ݓሻ and ܩ ௬௬ ሺ݆ݓሻ are the input and the output Power Spectral Density (PSD) matrices. ܩ ௫௫ ሺ݆ݓሻ is constant given that excitation is considered as a zero-mean white noise. ሻݓ݆ሺܨܴܨ is the Frequency Response Function matrix.

Under the assumption of a lightly damped structure and white noise excitation, ܩ ௬௬ ሺ݆ݓሻ can be written as follows:

G ୷୷ ሺjwሻ ൌ d ୩ ф ୩ ф ୩ ୲ jw െ λ ୩ d ୩ * ф ୩ * ф ୩ * ୲ jw െ λ ୩ * ୩ ⋲ሺୱ୳ୠሺ୵ሻሻ ( 19
)
Where ݀ is a scalar constant and ሻݓܾሺݑݏ is a set of modes that contribute at a particular frequency. ф is the shapes matrix, ߣ ൌ െߪ ݓ݆ ௗ are poles and ݓ ௗ is the damped natural frequency. The superscripts * and t denote respectively the complex conjugate and the transpose operator.

The FDD algorithm is executed following these steps:

•

Where ܲܵܦሺ݆ݓሻ denotes the power spectral density and ሻݓ݆ሺܦܵܥ denotes the Cross Spectral Density. ሻݓ݆ሺݕݕܩ is the Power spectral density matrix of 4 measured degree of freedom. • Step 2: Performing SVD decomposition of ሻݓ݆ሺݕݕܩ at eigen-frequencies [START_REF] Wang | A frequency and spatial domain decomposition method for operational strain modal analysis and its application[END_REF]:

G ୷୷ ሺjwሻ ൌ ሾфሿሾΣሿሾфሿ ୌ (21)
Where the diagonal value of ሾߑሿ are the singular values and the columns of ф are the singular vectors. • Step 3: If only one mode is dominating at a particular frequency, then only one singular value will be dominating. Thus the mode shape Ѱ݅ corresponds to the first eigenvector [START_REF] Gade | Frequency domain techniques for operational modal analysis[END_REF]:

Ѱ݅ ൌ ф ଵ ( 22
)
2.

NUMERICAL SIMULATION:

The Rivière-aux-Mulets bridge, located on Highway 15 in Sainte-Adèle in the north-west of Montreal -CANADA, was modelled on Abaqus software and the damage was introduced by a true seismic load (Figure 1). The model consists of three concrete piers and three concrete spans. The left and right spans are 40.8 m long and the central span is 80.4 m long [START_REF] Talbot | Approches expérimentales et numériques pour l'analyse dynamique d'un pont routier[END_REF]. The north and south abutments have not been modelled. The span-abutments and span-pier connections are modelled by spring elements. The L'Aquila earthquake accelerogram is used in this numerical simulation. The earthquake occurred in the Abruzzo region in Italy. Its magnitude rose to 6.3 on the moment magnitude scale. The earthquake damaged about 10,000 buildings making this earthquake the deadliest in Italy. The signal recorded at the AQV station was used to damage the numerical model (Figure 2). The purpose being to damage the structure, the signal amplitude is reduced to 50% and only 10 seconds of the signal are used to excite the model in the ݔ ሬሬԦ and ݖ ሬሬԦ directions. Eleven sensors were considered to record the response of the structure to white noise excitation before and after the damaging event in the ݕ Ԧ direction (Figure 1). They were placed in a linear and equidistant manner to have an accurate reflection of the mode shapes. The numerical model takes into account the complex and non-linear behaviour of the concrete. The recognition of crack patterns is made through the elasto-plastic damage model: Concrete Damage Plasticity (CDP). The CDP is recommended for analysis of concrete structures subjected to cyclic loadings and is governed by the following equation [START_REF] Jankowiak | Identification of parameters of concrete damage plasticity constitutive model[END_REF]:

σ ൌ ሺ1 െ dሻD ୣ୪ : ሺε െ ε ୮୪ ሻ ( 23
)
Where ߪ is Cauchy stresse tensor, d the scalar stiffness degradation variable, ܦ the undamaged elastic stiffness of the material, ߝ the strain tensor and ߝ the plastic strain tensor.

The major mechanisms of rupture for this model are tensile cracking and compression crushing. The damaged concrete response is characterized by two independent damage variables DamageT and DamageC, which are assumed to be dependent on plastic strain and temperature. They range from 0 for undamaged material to 1 for completely damaged material. In the case of our work, no dependence on temperature is considered. The constitutive parameters of the concrete class B50 damaged plasticity model were used (Table 1).

Table 1 -Constitutive parameters of the concrete class B50 damaged plasticity model [START_REF] Jankowiak | Identification of parameters of concrete damage plasticity constitutive model[END_REF] We are interested only in the first two modes of bending in this direction afterwards. The complete SHM chain was performed using the accelerations of the structure in both states: the initial and the final state.

Frequencies and mode shapes identification

The SSI algorithm was implemented using the response of the structure to a white noise.

The eigen-frequencies of the first modes have been successfully identified. Results obtained by the finite element analysis and the Stochastic Subspace Identification method are very close (Table 2). The identification errors, not exceeding 1%, testify to the robustness of this AMO technique. These eigen-frequencies correspond to the 1 st and 2 nd bending modes in the ݕ Ԧ direction (Figure 5).

The FDD algorithm was implemented using SSI results and the same response of the structure to a white noise. Comparison of the first two estimated mode shapes with corresponding FE model ones is shown in Figure 6. Good agreement between the estimated mode shapes by FDD and the numerical ones can be observed. Fig. 6 -Table 1 Result of mode shape identification using FDD algorithm

Damage detection

Using the eigen-frequency method, the damage was detected thanks to a reduction of 2.63% in the first bending mode and a reduction of 1.93% in the second one (Figure 7). Unfortunately, in real life situations, such values do not allow to define the state of the structure. Damage is detected with confidence only when the shift of frequencies is 5% or more. Shifts lower than 5% can be explained by hygrothermal effects [START_REF] Alvandi | Contribution à l'utilisation pratique de l'évaluation dynamique pour la détection d'endommagements dans les ponts[END_REF]. The damages are not detected using the MAC criterion applied to the first mode shapes, since the values of MAC remain relatively close to 1 without any relevant indication of a defect (Figure 8a). Using Abaqus's results, the damages are detected in the 11 th mode, indicated by a value 0.7 diagonally, which satisfies the detection condition (Figure 8b).

Damage localization

For the localization level, significant variations in the mode shape curvatures have been noted. These variations were located between node 6 and 8 in the first mode (Figure 9a) and around the node 4, 6 and 8 in the second mode (Figure 9b). Variations were also observed using the CDF between node 6 and 8 (Figure 10).

CONCLUSION

In this article, a complete chain of SHM of civil engineering structures was set up on a numerical model. The chosen model is the Rivière-Aux-Mulets bridge. It was damaged by the seismic signal of L'Aquila earthquake. The response of the healthy and the damaged structure, to a white noise excitation, was recorded. From its output only response, eigenfrequencies and modal deformations were identified by using Operational Modal Analysis techniques. These dynamic characteristics were used to define the state of the bridge. The study carried out allows us to highlight the following conclusions:

• As it has been reported in the literature, the Stochastic Subspace Identification technique and the Frequency Domain Decomposition technique proved to be very robust.

• The eigen-frequencies method has been able to detect the damage, but the reduction of the frequencies is weak. Experimentally, these variations can be caused by hygrothermal effects, without the structure being damaged.

• The damages were detected using the MAC method in the 11 th mode. This higher order mode is difficult to identify experimentally.

• In this test case, the flexibility curvature method showed the most accurate damage localization results.

1 - 7 ) 2 -

 172 Computing the flexibility variation matrix: ܨ∆ ൌ ܨ ௨ െ ܨ ௗ (Computing the maximum absolute value: ϒ ఫ തതത ൌ ݔܽ݉ ܨ∆| | (Ndambi, 2002) (8) 3-Computing the flexibility curvature:

Fig. 1 -

 1 Fig. 1 -Rivière-Aux-Mulets bridge: (a) Street view; (b) Numerical model and location of the 11 sensors.

Fig. 2 -

 2 Fig. 2 -Vertical acceleration recorded at station AQV for 2009 L'Aquila earthquake, 2.5 mile from the earthquake epicenter (www.strongmotioncenter.org )

Fig. 4 -

 4 Fig. 4 -Flowchart explaining the different steps of the numerical studies allowing this work

Fig. 5 -

 5 Fig. 5 -The mode shapes of vibration: (a) first bending mode 1.523 [Hz], (b) second bending mode 2.574 [Hz]

Fig. 7 -

 7 Fig. 7 -Shifts of the eigen-frequencies after the seismic event

Fig. 8 -

 8 Fig. 8 -Modal Assurance Criterion: (a) two modes; (b) eleven modes

Fig. 9 -

 9 Fig. 9 -Mode Shape Curvature Method: (a) 1 st bending mode, (b) 2 nd bending mode

Fig. 10 -Fig. 11 -

 1011 Fig. 10 -Curvature Damage Factor

  Step 1: Computing the Power Spectral Density matrix :ሻݓ݆ሺݕݕܩ Gyyሺjwሻ ൌ ൦ PSD ଵଵ ሺjwሻ CSD ଵଶ ሺjwሻ CSD ଵଷ ሺjwሻ CSD ଵସ ሺjwሻ CSD ଶଵ ሺjwሻ PSD ଶଶ ሺjwሻ CSD ଶଷ ሺjwሻ CSD ଶସ ሺjwሻ CSD ଷଵ ሺjwሻ CSD ଷଶ ሺjwሻ PSD ଷଷ ሺjwሻ CSD ଷସ ሺjwሻ CSD ସଵ ሺjwሻ CSD ସଶ ሺjwሻ CSD ସଷ ሺjwሻ PSD ସସ ሺjwሻ ൪

Table 2 -

 2 Result of eigen-frequencies identification using SSI algorithm

			Before the damage			After the damaging event
	Mode	Abaqus	SSI-COV	Error (%)	Abaqus	SSI-COV	Error (%)
	1	1.52	1.52	0	1.49	1.48	0.5
	2	2.57	2.59	-0.62	2.56	2.54	0.7

ACKNOWLEDGMENTS

The study received financial support from the National Association of Research and Technology (ANRT). The authors are very thankful for this support.

Concrete compression hardening

Concrete compression damage

Concrete tension stiffening

Concrete tension Damage

After the seismic event, the model is found to be substantially damaged at the measuring nodes 3, 6 and 8 (Figure 3). The complete experimental SHM chain was carried out numerically. Meaning, the sensors were modelled by model-related nodes, and the response of the structure was recorded. In the first place, the structure was excited by a white noise in its undamaged state and its acceleration was recorded. In the second place, the structure was damaged by a seismic signal, excited by a white noise and its acceleration was recorded. Three types of analysis were used on Abaqus: Dynamic explicit to introduce the seismic excitation, Modal dynamics to record the structure's response and Frequency for modal analysis and to allow damage to appear. The different steps, mentioned above, are summarized in Figure 4.