Axel Charpentier
email: axel.charpentier@polymtl.ca

Nora Boulahia

Frédéric Cuppens
email: frederic.cuppens@polymtl.ca

Reda Yaich
email: reda.yaich@irt-systemx.fr

Deep Reinforcement Learning-Based Defense Strategy Selection

Keywords: Security and privacy → Network security, Systems security, • Computing methodologies → Machine learning, Modeling and simulation Moving Target Defense, Deception,Deep Reinforcement Learning

Deception and Moving Target Defense techniques are two types of approaches that aim to increase the cost of the attacks by providing false information or uncertainty to the attacker's perception . Given the growing number of these strategies and the fact that they are not all effective against the same types of attacks, it is essential to know how to select the best one to use depending on the environment and the attacker. We therefore propose a model of attacker/defender confrontation in a computer system that takes into account the asymmetry of the players' perceptions. To simulate attacks on our model, a basic attacker scenario based on the main phases of the Cyber Kill Chain is proposed. Analytically determining an optimal solution is difficult due to the model's complexity. Moreover, because of the large number of possible states in the model, Deep Q-Learning algorithm is used to train a defensive agent to choose the best defensive strategy according to the observed attacker's actions.

INTRODUCTION

It is widely acknowledged that in cybersecurity, there is an asymmetry between the attacker and the defender. Indeed, the attacker Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ARES 2022, August 23-26, 2022, Vienna, Austria © 2022 Association for Computing Machinery. ACM ISBN 978-1-4503-9670-7/22/08. . . $15.00 https://doi.org/10.1145/3538969.3543789 chooses when and how they will try to penetrate or compromise a network or a machine. The attacker can therefore collect information about their target and come back later to compromise it, while it is very difficult for the defender to assess the threats they face.

To reduce this advantage of the attacker, several approaches have been proposed. The first existed long before computer systems and is at the basis of the warfare strategy. It is called deception. It consists in preventing the attacker from obtaining real information about a system by providing them false information in order to make their reconnaissance phase inefficient and thus reduce the probability of successful attacks. This can be done by several means such as perturbation, obfuscation or the use of honeypots [START_REF] Peter | A practical guide to honeypots[END_REF]. Today, deception strategies have developed a lot. Some surveys list and classify existing deception techniques and strategies [START_REF] Fraunholz | Demystifying deception technology: A survey[END_REF][START_REF] Han | Deception techniques in computer security: A research perspective[END_REF][START_REF] Pawlick | A game-theoretic taxonomy and survey of defensive deception for cybersecurity and privacy[END_REF].

Another approach to reduce the attacker's advantage is Moving Target Defense (MTD). The objective of MTD is to eliminate the attacker's time advantage due to the static nature of network infrastructures. This consists of changing the attack surface as well as the exploration surface to make any attack more difficult. This approach is sometimes considered as a deception strategy. However, while the objective of deception strategies is to provide false information to the attacker, the objective of MTD strategies is to prevent the use of previously obtained information, i.e., to make the information previously obtained by the attacker no longer valid. MTD strategies have been the subject of many research articles. Previous surveys have examined existing MTD strategies, each with its own criteria for analysis and classification [START_REF] Cho | Toward proactive, adaptive defense: A survey on moving target defense[END_REF][START_REF] Sengupta | A survey of moving target defenses for network security[END_REF][START_REF] Zheng | A survey on the moving target defense strategies: An architectural perspective[END_REF].

There are many deception and MTD techniques in the literature. However, each of them is not effective against the same types of attacks and permanently maintaining a deception strategy has a cost. It is therefore necessary to be able to choose the most appropriate strategy according to the environment and actions of the attacker.

To address this issue, models for the interactions between the attacker and the defence in a cyber environment have been proposed. A large part of these models is based on game theory (See Section 2). This enables to obtain some theoretical results such as mixedstrategy or pure-strategy Nash equilibrium of the game. However, in general, these models are too basic and do not allow to represent well the complexity of an attacker/defender confrontation, in particular by considering the perception of the different players and the multiplicity of possible states for a computer system.

However, Reinforcement Learning (RL) and more precisely Deep Reinforcement Learning (DRL) has emerged to enable the training of agents in complex environments. This allows for agents to obtain approximations to optimal solutions in models that a formal resolution can't obtain because of their complexity. Moreover, model-free RL algorithms allow to obtain solutions without using the transition probabilities associated with a Markov Decision Process (MDP). DRL also makes it possible to find solutions in models where the number of states or the space of actions can be very large.

This article proposes solutions to the issues outlined above. We summarize our contributions of this work as follows:

• An attacker-defender model considering the asymmetry of the players' perception and the difference in kind between MTD and deception strategies. • An attack scenario integrating Cyber Kill Chain (CKC) stages and based on the attacker's perception. • Use Deep Q-Learning in the model to optimize the choice of defense strategy against an attacker • Experiments to measure the performance of the defense strategy in the model The remainder of the paper is organized as follows. In Section 2, we discuss related works about modeling a confrontation between defender and attacker in a computer system along with the strategy selection of each player. In Section 3, our attacker/defender model considering the perception of each player is presented. We describe how the confrontation game works. The attacker's scenario is then introduced in Section 4. In Section 5, we discuss the use of a DRL algorithm to train agents on our model to find an optimal defender strategy. In Section 6, we provide the data required for the experiments, the results obtained and an analysis of them.

RELATED WORKS 2.1 Game Modeling

To model deception, the perception of the different players must be considered. A common approach is to use game theory to model the interactions between players. There are several types of games to achieve this. We present the most common types of games and some of their uses in the context of protecting a computer system using deception.

A first type of game mentioned in the literature is the Stackelberg Game. This is a type of game in which one player is the leader, often the defender, and the other player is the follower, often the attacker. The leader chooses their actions first, which is observed by the second player who plays next. Clark et al. [START_REF] Clark | Deceptive routing in relay networks[END_REF] use Stackelberg Games to study a jamming defense. They model the interactions between a defender and a jammer in a two-stage game. The defender will generate a false traffic flow. The defender will seek to maximize throughput while minimizing delay and the attacker will choose the fraction of each flow to jam. The existence of a pure-strategy Stackelberg equilibria is shown by the authors. Clark et al. [START_REF] Clark | A gametheoretic approach to IP address randomization in decoy-based cyber defense[END_REF] propose to use a Stackelberg Game to analyze the interactions between an attacker and a network with decoy nodes. The attacker seeks to identify real nodes by looking at the response times of the nodes and the protocols used. The defender chooses when to randomize the IP addresses of devices in the network. This game admits a unique threshold-based Stackelberg equilibrium. Feng et al. [START_REF] Feng | A stackelberg game and markov modeling of moving target defense[END_REF] model defender-attacker interactions by combining a Stackelberg game with an MDP. At the beginning of each round, the defender chooses an MTD strategy for several periods. The attacker chooses a state to attack. An algorithm is designed to choose the best strategy under worst-case. Sengupta and Kambhampati [START_REF] Sengupta | Multi-agent reinforcement learning in bayesian stackelberg markov games for adaptive moving target defense[END_REF] propose a Bayesian Stackelberg Game to model the interactions between an attacker who can be of different types and a system using MTD. A Q-Learning algorithm is used to find an optimal solution.

A second type of game often used to model the interaction between an attacker and a defender in a computer system is Signaling Game. This is a class of two-player games of incomplete information. A first player performs an action and transmits information to the other player with a certain cost which will be higher if the information is false. The second player does not know the nature of the information they received but chooses an action based on it. Pawlick and Zhu [START_REF] Pawlick | Deception by design: evidence-based signaling games for network defense[END_REF] use Signaling Games to analyze the effectiveness of HoneyPots deployments in a computer network. In particular, they show that sometimes the usefulness of the defender can increase when the attacker can detect the luring. La et al. [START_REF] Duy | Deceptive attack and defense game in honeypot-enabled networks for the internet of things[END_REF] propose the use of a Signaling Game to model the interactions between a defender and an attacker in a honeypot-enabled network. This time, the attacker plays first by choosing among several types of attacks while the defender plays second and can use HoneyPots to trap the attacker. Çeker et al. [START_REF] Çeker | Deception-based game theoretical approach to mitigate DoS attacks[END_REF] use Signaling Games to model the interactions between an attacker and a defender whose goal is to protect a server from Denial-of-Service (DoS) attacks while providing a service to legitimate users. The defender can disguise a normal system as a HoneyPot and vice versa. Rahman et al. [START_REF] Mohammad Ashiqur Rahman | A game-theoretic approach for deceiving remote operating system fingerprinting[END_REF] propose a mechanism to limit Remote Operating System Fingerprinting. They use a Signaling Game to model the interactions between the fingerprinter and its target.

Stochastic games are a type of multi-state game where the game is played as a sequence of states. In each of state, each player will choose an action which will have an impact on the new state of the system. A few articles use this type of game to study the impact of a defensive deception strategy on the perception of the attacker in different environments [START_REF] Ahmed H Anwar | Honeypot allocation over attack graphs in cyber deception games[END_REF][START_REF] Horák | Manipulating adversary's belief: A dynamic game approach to deception by design for proactive network security[END_REF]. Anwar et al. [START_REF] Ahmed H Anwar | Honeypot allocation over attack graphs in cyber deception games[END_REF] study the optimal placement of honeypots in an attack graph in order to slow down or prevent an attack. They are interested in the trade-off between security cost and deception reward for the defender. Horák et al. [START_REF] Horák | Manipulating adversary's belief: A dynamic game approach to deception by design for proactive network security[END_REF] propose the analysis of an active deception against an adversary that seeks to infiltrate a computer network to exfiltrate data or cause damage. They use a one-sided partially observable Stochastic Game to study the impact of deception on the attacker's perception. Here, it is assumed that the defender is able to detect the attacker's progress while the attacker lacks information about the system and is therefore vulnerable to deception.

Strategy Selection

Once the game is modeled, the choice of player actions must be optimized. In game theory, some tools are available for this purpose like Mixed Strategy Nash Equilibrium (MSNE). This is an equilibrium in which no player can improve their expected utility if they are the only one to choose another strategy because each player chooses their optimal strategy assuming that the opponent does the same. These equilibria match with the optimal solutions when considering the decisions of other player. This requires a good knowledge of the opponent's strategies and their outcomes. Hu et al. [START_REF] Hu | SOCMTD: selecting optimal countermeasure for moving target defense using dynamic game[END_REF] study the selection of the best countermeasure to maximize defense payoff. This approach uses game theory and more precisely signal game theory. The perfect Bayesian equilibrium is calculated and an algorithm is proposed for the selection of the optimal defense strategy. Experiments in a small environment are done to show the efficiency of the proposed approach.

Some papers propose to find the Nash equilibrium to optimize the choice of a MTD strategy [START_REF] Lei | Optimal strategy selection for moving target defense based on Markov game[END_REF][START_REF] Zhang | Strategy Selection for Moving Target Defense in Incomplete Information Game[END_REF]. [START_REF] Zhang | Strategy Selection for Moving Target Defense in Incomplete Information Game[END_REF] study the selection of the MTD strategy of a defender in a web application environment. The confrontation between the attacker and the defender is done through an incomplete information game. The Nash-Q Learning algorithm is used to find this optimal strategy and its performance is compared to other algorithms such as the Minimax-Q learning algorithm and the Naive-Q learning algorithm in this environment. The results of the experiments show the efficiency of the Nash-Q learning algorithm compared to the others. L. Cheng et al. [START_REF] Lei | Optimal strategy selection for moving target defense based on Markov game[END_REF] use a Markov Game to propose a model considering the multiphase and multistage nature of the confrontation.

Tan et al. [START_REF] Tan | Optimal strategy selection approach to moving target defense based on Markov robust game[END_REF] make a parallel between MTD transformation and change of the attack surface and the exploration surface. They propose a MTD model based on a Markov robust game model. Robust Games can reduce the dependence on prior knowledge for the optimal strategy selection present in other models. As shown by L. Cheng et al. [START_REF] Lei | Optimal strategy selection for moving target defense based on Markov game[END_REF], the use of such a model allows to take into account the multi-stage and multi-state nature. The existence of a robust equilibrium and an optimal solution for this model is proven under certain conditions. An algorithm is proposed to find these and simulations are done to show the efficiency of this approach.

Some articles propose models that are not strictly based on game theory. As MTD strategies are often effective against one type of attack, H. Zhang et al. [START_REF] Zhang | Efficient strategy selection for moving target defense under multiple attacks[END_REF] study the impact of using MTD strategies with one or multiple mutant elements against several attacks. An algorithm based on the Genetic Algorithm is proposed to find the most efficient combination of MTD strategies. It is shown that even in an environment with limited resources it is still important to use multiple mutant elements.

However, as a model becomes more complex, finding equilibria becomes increasingly and computationally intractable. Equilibria as Nash equilibria do not consider past behavior of players which often lead to predict future behavior. Moreover, the fact that the optimal solution found is often a mixed strategy makes its practical use difficult. DRL proposes to obtain an approximate pure strategy of the best strategy while limiting the computational cost. It also allows to find an approximation of the optimal solution in models that are not based on game theory [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF].

Chai et al. [START_REF] Chai | DQ-MOTAG: deep reinforcement learning-based moving target defense against DDoS attacks[END_REF] propose to use DRL to improve MOTAG [START_REF] Jia | Motag: Moving target defense against internet denial of service attacks[END_REF], a MTD mechanism to counter Distributed DoS (DDoS) attacks. MOTAG uses proxies to transmit traffic between legitimate users and protected servers. By using these proxies, it is able to isolate the external attacker from the innocent clients while shuffling the client/proxy assignments. DQ-MOTAG provides a self-adaptive shuffle period adjustment ability based on reinforcement learning. Experiments show the efficiency of the method used.

T. Eghtesad et al. [START_REF] Eghtesad | Adversarial deep reinforcement learning based adaptive moving target defense[END_REF] propose to find an optimal MTD strategy in the model proposed by Prakash and Wellman [START_REF] Prakash | Empirical game-theoretic analysis for moving target defense[END_REF]. For this purpose, a two-player general-sum game between the adversary and the defender is created. In this model, the attacker and the defender oppose each other for the control of servers, the mechanism of server compromise is simplified. A compact representation of the memory for each player is proposed and enables to act better in the partially observable environment. This paper succeeds in solving this game using a multi-agent RL framework based on the double oracle algorithm which allows to find mixed-strategy Nash equilibrium in games.

S. Wang et al. [START_REF] Wang | An intelligent deployment policy for deception resources based on reinforcement learning[END_REF] study the deployment policy for deception resources and especially the position of these resources. A model is proposed to represent the attacker/defender confrontation and the attacker's strategy is provided. A Threat Penetration Graph (TPG) is used to preselect the locations of deception resources. A Q-learning algorithm is provided to find the optimal deployment policy. A real-world network environment is used to demonstrate the effectiveness of the proposed method.

The problem with games such as signaling games is that they are generated with two players who each has two possible actions. These are games with incomplete information taking into account the asymmetry of the players' perceptions but do not support multiple sources of deception as there could be in a computer system. Moreover, this type of game is not adapted to model a multi-state and multi-stage situation. Other Bayesian games take into account multistate and multistage characteristics but to our knowledge, in the literature, there is no game modeling that takes into account the different perceptions of the players and that allows to take into account both the effects of deception and the effects of MTD strategies on a computer system. Another difficulty is to find an optimal solution in a cyber environment model where the number of states can be very large. The use of Deep Reinforcement Learning will help to address this problem.

CONFRONTATION MODEL

In this section, we present our model of a computer system with several types of potential vulnerabilities. A single machine is modeled but we could model a network of machines in a similar way. We will come back to this in the Section 6.4. Our modeling takes into account the perception of the players and the actions allowed for each of them are real-world actions, so the attacker can launch scans and attacks on different types of vulnerabilities while the defender can deploy Deception or MTD strategies. Due to their different characteristics, these strategies will have a different effect on the environment. MTD strategies will affect the attacker's prior knowledge, while deception strategies will provide false information to the attacker's scans. Deception strategies are deployed over several time-step but are not deployed permanently because maintaining a deception strategy has a cost.

In Section 3.1, we present the main components of the model. We define the different states of the game in Section 3.2 and the observations of the two players in Section 3.3. The game process is presented in Section 3.4 and the reward system in Section 3.5. These sections are useful to understand how DRL is used in this context (see Section 5).

Environment

In this section, we define our attacker/defender model using player perception. For this, we consider a machine, an attacker and a defender. The attacker seeks to compromise the machine while the defender seeks to defend it by using MTD and deception strategies. The machine will be modeled by a vector of vulnerabilities V and by a compromise score C. For each available vulnerability i, if it is present on the machine then

V [i] = 1 otherwise V [i] = 0.
The compromise index C is between 0 and 1. If the machine is not compromised then C is 0. Its score can then evolve to 1 meaning that the machine is fully compromised.

Attacker.

A first player of the model is the attacker whose goal is to compromise the machine. For this, they have several actions available: a GlobalScan, Scans or Attacks. For each vulnerability in V , present or not on the machine, a scan and an attack are available. To each of these actions is associated a DI score indicating the information damage, a DC score indicating the compromise damage and a Cost indicating the cost of this action. The compromise damage of Scans is zero while the information damage is higher for Scan actions than for Attack actions. The greater the information damage for an action against a certain vulnerability, the faster it will be for the attacker to acquire the information needed to exploit it with an attack. In contrast, if a vulnerability is not present on the machine, a large DI score for an action will make it faster to remove that vulnerability from potential vulnerabilities. In addition to these available actions, there is a Global Scan that scans all vulnerabilities at once. In return, the information obtained on each vulnerability by a GlobalScan is lower than for a singular Scan.

Other variables are associated with the attacker and allow measuring the attacker's progress in compromising the machine. We define two vectors pV and perceivedpV containing for each vulnerability v in V , respectively, a score measuring the amount of real information the attacker has about this vulnerability and a score measuring the amount of information the attacker thinks they have about this vulnerability. If perceivedpV [v] = 0.5, the attacker has no information about the vulnerability v. The closer perceivedpV [v] is to 1, the more information the attacker has about the vulnerability v and the more they think the machine has it. The closer perceivedpV [v] is to 0, the more information the attacker has about the vulnerability v and the more they think the machine has not it. perceivedpV and pV are not necessarily equal. If the attacker is deceived, then these two vectors may not be equal. Since pV represents a real amount of information, if the machine is vulnerable to a vulnerability v then pV [v] will be between 0.5 and 1 and if it is not then pV [v] will be between 0 and 0.5. The variable Ca measures the state of compromise of the machine perceived by the attacker. An assumption of our model will be C = C a . This means that the deception will not be about the status of compromise but will focus on the information needed for attacks. Another variable will be the phase of the CKC in which the attacker is located. This one can take three values (0,1 or 2) because we will consider only 3 phases of the CKC to make the model simpler: Reconnaissance, Intrusion and Pivilege Escalation/Exploitation. This parameter will be updated at each step according to the values of pV ,I a and C a .

It is considered that some attacks will only be available at certain phases of the CKC.

Finally, two other variables will measure the attacker's overall knowledge of the machine. The first one is System Info Perception I a (See Equation 1) which measures the overall amount of information perceived by the attacker about the observable vulnerabilities of the machine. This means that in CKC 0 and 1, we do not consider vulnerabilities that can only be used in CKC 2. The second is Useful System Info Perception Iu a (See Equation 2) which measures the amount of useful information for attacks perceived by the attacker on the observable vulnerabilities of the machine.

I a = 2 K v ∈V max(0; 0.5 -|V [v] -pV [v]|)) (1)
Iu a = 2 K v ∈V max(0; perceivedpV [v] -0.5) ( 2
)
where K is the number of available attack strategies considered in the model.

Defender.

The second player will be the defender. Their goal is to prevent the machine from being compromised while minimizing the cost to achieve this goal. Among its possible actions, there are MTD strategies and deception strategies. The defender can also choose not to use any actions. MTD strategies will influence the knowledge already acquired by the attacker. Indeed, if the attacker has collected information about the system and the defender successfully uses a MTD strategy then some of the information acquired by the attacker will become incorrect.

On the other hand, deception strategies do not modify the validity of the information already obtained by the attacker. They provide the attacker with erroneous information. If the attacker acquires information about the system in the steps following the use of a deception strategy, this information may be erroneous and thus deceive the attacker.

Each of the possible strategies is associated with a cost that considers the difficulty to implement it and the software or hardware resources needed to do so.

Model parameters.

A table E f f defines for each action of the defender its effectiveness against an attacker's strategy. This value includes both the ability to prevent an attack and the ability to prevent information about the system from being obtained. We will come back to this in Section 6.1.

In our model, the defender does not necessarily detect all attacks. It detects them with the probability P Det ect ion which is a parameter to be specified in the model. To simplify, it's the same for all attacks. In addition, when an attacker takes an action, the defender can detect which potential vulnerability has been targeted but is not able to distinguish a scan from an attack. This will have an effect on the defender's observations. Deception strategies do not necessarily achieve their objectives. There is a probability of deception success (See Equation 3) depending on the attacker's information about the machine and the phase of the CKC:

P Decept ion = exp(-λ Decept ion • (CKC + 1) • (I a + 0.3)) (3)
Moreover, the attacks are not necessarily a success. The probability of success of an attack targeting a vulnerability on the machine against a defense strategy (See Equation 4) depends on the attacker's knowledge of the targeted vulnerability and the effectiveness of the defensive strategy against the attack.

P At t ack (A, D) = V [A] • exp(- λ At t ack • (E f f (A, D) + 0.3) max(0.01; (pV [A] -0.5) • 2) ) (4)
where A represents an attacker's action and D represents a defender's action. If the vulnerability is absent from the machine then the probability of success of the attack is zero. λ Decept ion and λ At t ack are parameters that will calibrate the above probabilities to make them more consistent with the real environment we want to model. Similarly, the value of certain offsets in these probabilities could also be adjusted to better reflect the values of a real situation.

States of the game

Our model uses a discrete time scale. The game is a succession of stages. At each stage, the game is characterized by its state. In a given time step t, the state of the game is :

s t = ⟨V , C, pV , CKC⟩ (5)
Each of these parameters has been defined previously.

Observations

For each of the players, the attacker and the defender, we will define the observations. These are the information held by each player that can be used at each time step of the game to choose a strategy. The state of the game for the attacker is defined by the tuple O a :

O a = ⟨perceivedpV , C a ⟩ (6)
This provides to the attacker both information about the vulnerabilities of the machine and about the state of compromise of the machine.

On the other hand, the defender has information about the different vulnerabilities that have been targeted by the attacker in the past if the attacks have been detected. They also have information about the defense strategies used in the past. The state of the machine for the defender is defined by the tuple O d :

O d = ⟨last_attacks, last_de f enses, times_since_last_attack, times_since_last_de f ense, attack_counts⟩ (7)
where

• last_attacks is a vector that contains for the last k time steps, if detected, the vulnerabilities targeted by the attacker in One-Hot encoding i.e. each element of the vector is 1 if the corresponding attack has been detected and 0 otherwise. • last_de f enses is a vector that contains for the last k time steps, the strategies used by the defender in One-Hot encoding. • times_since_last_attack is a vector containing for each vulnerability the number of time steps elapsed since the last action (Scan or Attack) detected from the attacker on it.

• times_since_last_de f ense is a vector containing for each defense strategy the number of time steps elapsed since its last use. • attack_counts is a vector containing for each vulnerability the total number of detected actions of the attacker on it.

k is a parameter to be defined of the model. It represents the number of time steps considered in the last_attacks and last_de f enses vectors of the observations.

Course of a game step

In this section, we will explain how the game proceeds. For that, we will introduce the Information Gathering function (See Algorithm 1). It corresponds to the unfolding of an information gathering strategy of the attacker and its impact on the model. It will modify the attacker's information held on the vulnerabilities by considering the decoy strategies set up by the defender. This function is called at different moments of the game as for example when the attacker performs a scan, a global scan or an attack that fails. Indeed, it is considered that even when an attack fails, the attacker still gets information about the targeted vulnerability.

This Information Gathering function takes as parameter the vulnerability targeted by the attacker AND µ a parameter quantifying the potential impact of the information gathering on the vulnerability information. Indeed, the larger µ is, the more pV and perceivedpV will be modified by the information gathering. In the algorithms,

V [A A ] is short for V [v]
where v is the vulnerability targeted by the attack or scan A A . It is the same for pV [A A ] and perceivedpV

[A A ].
Algorithm 2 provides the pseudocode of the unfolding of a game timestep given the actions chosen by the defender and by the attacker. To summarize its operation, the first part is to check if the strategy of the defense is a MTD strategy. If this is the case, then pV evolves in this way. Then the nature of the attacker's action is checked. If it is an individual scan, then the Information Gathering function is used with a rather high µ, i.e. a strong potential impact of the information gathering. If it is a Global Scan, then we do the same but for all the available vulnerabilities and with lower µ. If not, we look to see if it is an attack. If it is and the attack is successful, then we update V , pV , perceivedpV and C. If not, we use the Information Gathering function with a low µ.

This process will repeat itself to form an episode. An episode is a sequence of states, actions and rewards that ends in a final state. In our model, an episode ends if the machine is compromised or if the number of time steps exceeds 100 because we consider that the attacker will be dissuaded or discouraged from attacking and will stop the attack.

Reward

To optimize the defender's strategy selection at each step, we need to define a utility function U D that specifies the defender's objective. Its objective is to make the compromise of the machine as long as possible and thus to minimize the information held by the attacker on it as well as the level of compromise of it. U D is defined as follows:

U D = w • (1 -I A ) + (1 -w) • (1 -C) (8)
the more scans will be promoted. The larger w3 is, the more compromise damage will be promoted in the utility and therefore the more attacks will be promoted. The larger w1 is, the more the attacker will try to fill their lack of information about the global system. In this case, it is rather the Global Scan that will be promoted. The cost of each attacker strategy also has its influence. The reward given to the attacker at time t is given by:

r t A = U A -Cost[A A ] (10)
where U A is the Defender Utility at time t and Cost[A A ] is the cost of the attacker action used at time t. The attacker will choose at each step the action that maximizes the reward they can get.

STRATEGY SELECTION OPTIMIZATION

In this section, we will present the approach used to find an optimal policy. A policy is a function that associates with each state of the game the action that the agent will take in this state. Because of the complexity of the game, the large number of possible states and the stochastic nature of the rewards and transitions between states, it is difficult to find an optimal policy analytically. That is why we decide to use Deep Q-Learning algorithm [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF].

Firstly, we define the discounted return at time t for the defender:

G t = r t +1 D + γr t +2 D + • • • + γ T -1 r T D (11)
where we consider the episodes as finished, T is the length of the episode and γ is a discount factor that balances the weights between future and current rewards.

A policy is a function that takes as input a state S and returns the probability of using each action in this state. We also define the Q-function Q π (s, a) which returns the value of taking action a in state s under policy pi:

Q π (s, a) = E π [G t |s t = s, a t = a] (12)
= E π [r t +1 D + γG t +1 |s t = s, a t = a] (13)
The optimal state-action value fonction is:

Q * (s, a) = max π Q π (s, a) (14)
= E[r t +1 D + γmax a ′Q * (s ′ , a ′ )|s, a, s t +1 = s ′ ] (15)
This equation is called Bellman Optimality Equation [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF].

CKC Phases Prerequisites Weights

Reconnaissance Ø 2: Attacker utility according to the type of attacks

w1 = 0.2 w2 = 0.6 w3 = 0.2 Intrusion Iu a > 0.3 w1 = 0.2 OR ANY w2 = 0.4 perceivedpV [i] > 0.8 w3 = 0.4 Exploitation C > 0.3 w1 = 0.4 OR w2 = 0.1 Privilege Escalation w3 = 0.5

Strategy Attacker Utility U

A (A A ) Scan w1 • (0.5 -|perceivedpV [A A ] -0.5|)+ (w2 • DI [A A ] + w3 • DC[A A ]) • perceivedpV [A A ] Attack (w2 • DI [A A ] + w3 • DC[A A ]) • perceivedpV [A A ] Global A i is Scan U (A i )/2 Scan Table
In our context, a state s corresponds to the defender's perception of the state of the game, i.e. its observations O d .

Temporal Difference Learning offers an iterative process that updates the Q-values for each state-action pair based on the Bellman Optimality Equation. This process converges to the optimal Qfunction.

Q(s, a) = (1 -α) • Q(s, a) + α(r t +1 D + γmax a ′Q (s t +1 , a ′ )) (16)
where α is a learning rate.

In classic Reinforcement Learning, a table is used to store the mapping between the pair state, action and their corresponding Q-value. In Deep Reinforcement Learning, a neural network is used for this. The input of the neural network is the state, and the output is the estimated Q-value of each action in this state.

We use the Deep Q-Learning algorithm (DQN) of Mnih et al. [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF] with a replay buffer and a target network to train our model. This algorithm uses many hyperparameters that must be defined before learning. They are in the Table 3

EXPERIMENTS 6.1 Data for Experiments

In Sections 3 and 4, we describe our model and a basic scenario for the attacker. Several parameters are required for the experiments. First, concerning the machine, we need to define the vulnerabilities considered in the model. These are not Common Vulnerablities and Exposures (CVEs), they are rather families of vulnerabilities. Then, for each of them, we have to define the damage in Information and in value of Compromission as well as their cost in the case of a scan or an attack. These values have been defined arbitrarily based on information from Mitre CAPEC [START_REF]Common Attack Pattern Enumeration and Classification (CAPEC)[END_REF] and Mitre CVE [START_REF]Common Vulnerabilities and Exposures (CVE)[END_REF]. Table 4 shows the data used in the experiments concerning the vulnerabilities and the attack strategies. We have selected a set of known vulnerability families that can be separated into two categories: those accessible from outside the machine and those requiring local access on the machine. For simplicity, we consider that the costs of the scans are the same. The same goes for the cost of the attacks.

Implementation

The experiments were done on an Intel Core i7-9750H CPU with an NVIDIA GeForce RTX 2070 graphics card.

Our model has been implemented in an OpenAi Gym environment [START_REF] Brockman | Openai gym[END_REF]. Stable Baselines3 is a python library providing the implementation of Reinforcement Learning algorithms [START_REF] Raffin | Stable-Baselines3: Reliable Reinforcement Learning Implementations[END_REF]. We use it to train our agents on the model.

We sought to optimize the hyperparameters used to train our agents. For this, we trained the model with many different parameters. The model parameters and the hyperparameters finally used for the tests are located in the Table 3.

Results

We trained agents in different environments with different parameters. We have measured the influence of the parameters on the performance of our trained agents. Shown in Figure 1 are the training curves of different defenders for different values of λ at t ack . We can thus compare the influence of the probability of attack success with the performance of the agents. Our model includes a significant uncertainty, so the reward has a large variance. In Figure 2 we compare the performance of our DQN-trained agent with the performance of agents using only one defensive strategy or randomly choosing the defensive strategy. We evaluate each strategy over 1000 episodes. We compare the average episode reward for each strategy but also the average episode length.

The agent trained with the DQN performs better than the other "naive" strategies in terms of both episode reward and episode length. This means that our agent is able to use its observations to deduce a defense strategy to use. Moreover, we can observe the influence of strategy costs in the reward because for some strategies the episode length is higher than for others but the corresponding reward is lower. For example, IP Random Strategy has a higher reward than Honey Service but a shorter episode length.

We compared the performances obtained by an agent trained with the DQN in different environments. We varied the probability of detection of attacks P Det ect ion , λ at t ack and λ decept ion . The lower the probability of detection of attacks, the lower the rewards for the defender. This makes sense because if the defender does not observe the attacks, they cannot choose an appropriate strategy. λ at t ack , which inversely influences the probability of success of attacks, has a strong impact on the performance of our agent. λ decept ion inversely impacts the probability of deception success. The larger the value for λ decept ion , the lower the probability of deception success.

The problem with the DQN algorithm is that the resulting policy is not necessarily optimal. Indeed, it may be a local optimum. This explains some of the results in Figure 3, for example, for P Det ect ion = 0.8 and λ at t ack = 0.7, the performances of the agent for λ decept ion = 0.5 are lower than for λ decept ion = 0.7 which is anomalous according to the explanations of the previous paragraph.

Discussion

The proposed model is multi-state and multi-stage. It allows to consider a large number of possible states for the system. Moreover, the representation of the vulnerabilities in the V vector allows to consider different types of machines such as servers or workstations which will have different types of vulnerabilities. Futhermore, thanks to our representation of the defender's observations, we are able to use the attacker's past behavior to better predict these future actions and thus choose an optimal strategy. Moreover, our modeling takes into account the difference in nature between MTD strategies and Deception strategies. Thus they act differently on the system. Deception and MTD strategies should be seen as complementary. Experiments show that to counteract a variety of potential attacks, it is necessary to have access to a variety of defensive strategies. The experiments also show the importance of good coordination of these strategies and the need to limit the deployment of certain strategies to critical moments because their cost to the system can be very high.

Regardless of the parameters used, we manage to train an agent that performs better than the unitary strategies against our attacker following our scenario detailed in Section 4. The advantages of this scenario are that it is based on the perception of the attacker and it separates remote attacks and local attacks such as privilege escalation.

The realism of the model could be improved by using data from experiments. To do this, we could implement several defensive strategies on a system with known vulnerabilities in order to measure the ability of these strategies to prevent or slow down their exploitation by an attacker. In particular, as the model is sequential, it would be interesting to integrate the time of each action for both the attacker and the defender in the calculation of their cost. Moreover, the effectiveness of defensive strategies against attacks should not be static but depend on the attacker's progress in their attacks. Indeed, for example, the IP Random defense strategy will not be as efficient against an attacker located outside the network or against an attacker who already has access to the internal network.

In this paper, the model seeks to represent the confrontation of a defender and an attacker around a single machine. However, to better represent reality, it would be necessary to extend this model to a network of machines. This would multiply the number of possible actions for each player, the number of states of the model and the number of entry points for the attacker. It would then be necessary to consider the conflicts that can exist in the deployment of defensive strategies on several machines. This would allow the simulation of more realistic attack scenarios. It would then be interesting to know if the DQN algorithm is still efficient in a model of this type. This is ongoing research work that will be the subject of a future article.

CONCLUSION

Deception and MTD strategies are two types of strategies that consist in bringing false information or uncertainty to the opponent's perception in order to increase the cost of the attacks. In this paper, we proposed a model of an attacker/defender confrontation in a computer system considering the asymmetry of perceptions and the impact of the two players' strategies on them. We then proposed an attacker scenario based on the CKC and using their perception. Thanks to this, we performed simulations and trained with the DQN algorithm a defensive agent. It is able to choose the most adapted defensive strategies to prevent the compromise of the machine by using the observed past actions. This simulation framework could be used to optimize the use of MTD and Deception strategies in real context by using data from experiments for model parameters.

Future works would then be to develop an emulation environment allowing the deployment of different MTD and deception strategies. This would provide data such as the cost or effectiveness of the latter. It could also allow for the training of defending DQN agent directly on the emulation environment. Another part of the work would be to improve the attacker scenarios. To do this, we could try to train an attacking DQN agent based on its observation.

Figure 1 :

 1 Figure 1: Learning curve of different agents in different environments with different λ at t ack and λ Decept ion = 0.5, P Det ect ion = 1

  Figure 3 contains the comparison of the rewards obtained with a DQN agent for different values of these parameters. It contains one heatmap Table 6: Efficiency of the defensive strategies against the attacker actions per P Det ect ion value. Each heatmap shows the rewards obtained for 3 different values of λ at t ack and λ decept ion .

Figure 2 :

 2 Figure 2: Comparison of the performance of different defense strategies in different environments with λ at t ack = 0.5 and λ Decept ion = 0.5, P Det ect ion = 1

Figure 3 :

 3 Figure 3: Comparison of the reward of the defender using a DQN agent in environments with different values of detection probability P Det ect ion , λ at t ack and λ decept ion

Table 1 :

 1 Prerequisites and weights of the utility function for each CKC phase.

Table 3 :

 3 . List of model parameters and DQN hyperparameters

	Parameter	Value
	Weight in Defender Utility	0.3
	k for Defender Observations	3
	Exploration Fraction	0.1
	Target Update Interval	5000
	Neural Network Layers	[256x256]
	Replay Buffer Size	1000000
	Activation Function	Tanh
	Train Frequence	One episode
	Batch Size	32
	Learning Rate	0.0005
	Discounted Factor	0.99

Table 4 :

 4 List of considered vulnerabilitiesConcerning the defender, we consider eight possible defense strategies: four MTD and four deception strategies. They can be found in the Table5with the cost of each strategy.

			required
	Vulnerability	Type CKC phase Cost DI DC
	Identity	Scan	0	0.2	0.3	0
	Spoofing	Attack	1	0.3 0.15 0.4
	Traffic	Scan	0	0.2	0.2	0
	Injection	Attack	1	0.3	0.1 0.4
	Brute Force	Scan Attack	0 1	0.2 0.3	0.2 0.1 0.4 0
	Command	Scan	0	0.2	0.4	0
	Injection	Attack	1	0.3	0.2 0.4
	Code	Scan	0	0.2	0.3	0
	Injection	Attack	1	0.3 0.15 0.4
	Privilege	Scan	2	0.2	0.2	0
	Abuse	Attack	2	0.3	0.1 0.6
	Authentication	Scan	2	0.2	0.1	0
	Bypass	Attack	2	0.3 0.05 0.6
	Privilege	Scan	2	0.2	0.2	0
	Escalation	Attack	2	0.3	0.1 0.6
	Strategy
	Number		Strategy		Type	Cost
	0		IP Random		MTD	0.1
	1		Port Random		MTD	0.2
	2	Rekeying Keys Random	MTD	0.1
	3		Language Random	MTD	0.3
	4		Honey Service		Deception 0.4
	5	Honey Credentials/Accounts Deception 0.3
	6		Honey Process		Deception 0.2
	7		Honey Files/Logs	Deception 0.2

Table 5 :

 5 Considered defense strategiesAnother essential element in the experiments is the table of efficiencies of the defensive strategies against the actions of the attacker mentioned in the Section 3.1.3 and shown in the Table6. These are chosen arbitrarily based on our knowledge of the different strategies, but one way to improve the model could be to use data from real experiments.

w is a numeric parameter to give more weight to the information or to the compromise in the defender's utility function.

The cost of each strategy also has its influence. The reward given to the defender at time t is given by:

where U D is the Defender Utility at time t and Cost[D A ] is the cost of the defender action used at time t. The defender seeks to maximize the total reward over an entire episode.

ATTACKER SCENARIO

Now that we have presented our Attacker/Defender model, we will present our attacker scenario. Indeed, we will train our defender to react to the attacker's actions to make the machine's compromise as long as possible. We therefore need an attack scenario for the attacker. This is based on a simplified version of the Cyber Kill Chain. Only three steps of the CKC are considered, the same as those used for the model (See Section 3.1.1): Reconnaissance, Intrusion and Exploitation or Privilege Escalation.

To compromise the machine, the attacker will have to go through each of these phases. Actions available to the attacker will depend on the phase in which they are. Indeed, for example, if the attacker is in the third phase of the CKC, then it means that they have infiltrated the machine and therefore there may be new potential Algorithm 2 Model -Run one step

Run Information Gathering with A A , and µ end if end if vulnerabilities reachable only from inside the machine. Moving from one phase to another of the CKC requires certain prerequisites which for this example scenario can be found in the Table 1.

Attacker Strategy Selection

At each step, the attacker will have to choose an action among those available. For this purpose, we use a very common function in game theory reffered to as utility function. This function associates with each available action a utility representing the satisfaction of the player to choose this action. The attacker will then choose the action that maximizes this function.

In our case, the satisfaction depends on the objective of the attacker and thus on the phase of the CKC in which they are. Indeed, for example, in the recognition phase, it is the collection of information and therefore the scans that are promoted while in the intrusion phase it is the attacks that are promoted. To this end, some parameters of the utility function will depend on the phase of the CKC in which the attacker is (see Table 1)

The utility function is not the same depending on the type of action of the attacker (See Table 2). Each parameter influences the utility function in a different way. Indeed, the larger w2 is, the more information damage will be promoted in the utility and therefore


Raphaël De Matos
email: rdematos@bavans.faurecia.com

Emmanuel Foltête

Noureddine Bouhaddi

Catherine Guyard

Total Calalytic Converter Modelling for Durability Improvement

Keywords: Exhaust gaz purification, dynamic modeling, non-linear damping, mechanical ageing, fiability

From the simple drain of flow to the gas cleaning, exhaust systems of automotive vehicles become complex modules playing a decisive role in the control of pollutant emissions and on acoustic comfort. In terms of depollution, the key elements are the catalysts and other systems of gas purification such as particle filters, NOx or SOx traps, etc. Catalyst is a subset in which chemical reactions are initiated with the aim of maintaining the pollutant emissions below certain rates fixed by standards. Evolutions of emission control standards deal with service life of catalytic converters. Mechanical durability should not restrict the lifetime of catalysts. This aspect has led us to develop a prediction tool, with the aim of preselecting design solutions and reducing testing time. To reach such requirements, a total catalytic converter modeling is needed. Many efforts have been made on both numerical modelling and experimental identification. Advanced measurement techniques such as 1D pulsed ESPI have been used. A non-linear damping identification approach based on the wavelet transform of time responses has been developed and validated.

Introduction

The need for strongly reducing air pollution generated by vehicles led to the elaboration of antipollution regulations. Those gave rise to well-known exhaust systems components: the exhaust catalysts.

These ones generally consist of the following under-components (FIG. 1):

-a ceramic substrate, location of chemical reactions.

-a metallic box.

-an holding mat which maintains the ceramic substrate within the steel shell. Such a device is submitted to different kind of mechanical sollicitations: on the one hand, all the vibrations generated by the engine or due to the road, as well as shocks, are transmitted to the steel shell. On the other hand, high thermal variations may reduce the service life of catalysts. As a consequence, designers must take into account all these factors to minimise their influence on durability.

Mat is a fibrous material which plays several significant roles:

-First of all it maintains the substrate, by compensating for geometrical defects and protects it from possible blows and impacts. -Secondly, it offers a thermal insulation of the metallic shell.

-Lastly, it guarantees that the gas flow crosses the substrate in its quasi-integrality (rate higher than 99%).

For all these reasons, mats constitute essential elements of catalysts. Their ageing directly determine the service life of the catalytic function. The current state of knowledge shows that failures noted on exhaust catalysts are mainly due to the fact that the mat does not ensure its maintenance property anymore.

The experimental characterization as well as the modelling of mats are essential to improve the durability of the catalytic function. This paper does not deal with ceramic substrates as they are considered, thanks to measurements, as rigid body components.

Shell characterization using speckle interferometry

Electronic Speckle Pattern Interferometry (ESPI) is a recent optical technique which allows the displacement field of a harmonically vibrating structure to be measured. Based on the holographic interferometry principle, it provides a contactless, full field and instantaneous measurement by using a powerful pulsed laser and a specific CCD camera, along one two or three directions. In the past few years, a technique has been developped in order to extract quantitative data for model updating from this kind of dynamic measurements [START_REF] Foltête | Quantitative dynamical measurements for model updating using electronic speckle interferometry[END_REF].

In order to perform a model updating under the best conditions as possible, the huge measured data (about one million of pixels) are reduced to the image points which correspond to the FE nodes. By using the video image, the relation between the 3D coordinates and 2D coordinates in the image, as well as the measurement direction (called sensitivity direction) are identified for each FE node.

The extraction of the measured data from the displacement image is made up of two operations: the data condensation on the finite elements (FE) nodes which are visible from the observation point and the computation of the sensitivity directions. A measured value is then associated to each visible node.

In order to perform efficient and rapid measurements, avoiding a large amount of non pertinent data, we first proceed to a classical accelerometric measurement leading to the knowledge of the modal frequencies and damping. Then ESPI measurements are performed on resonant frequencies [START_REF] Lepage | High spatial resolution vibration analysis using hybrid method : Accelerometers -double pulse espi[END_REF]: some mode shapes are presented in FIG. 2.

Solving the problem of multiple eigenvalues

As most of catalytic converters are cylindrical, we must take care about multiple eigenvalues. Let's write Y (a) , the matrix constituted by the N calculated eigenvectors. The matrix Y (a) is divided in sub-spaces: the first sub-space includes the rigid body modes in the case of a free-free boundary conditions study, then each sub-space is constituted by a pair of eigenvalues (same modal shape, but not with the same principal direction) (Equation 1).

Y (a) = . . . SE 1 SE 2 SE 3 . . . SE P (1)
This division is operated regarding eigenfrequencies: if the difference between two eigenfrequencies is below a predefined tolerance, we consider that they are associated to a pair of eigenvectors. Now, we dispose of two sets of eigenvectors:

-the eigenvectors extracted from measurements: Y (e) , -the analytical eigenvectors: Y (a) . For each eigenvector extracted from measurements y

(e)
ν , a combination of coefficients c i ν is calculated by applying the least square method on the Equation 2 below (minimizing ε i ):

SE i Y (a) c i ν = y (e) ν + ε i (2)
The set of coefficients Finally a normalization with respect to the mass is operated:

Ỹ (a) T [M ] Ỹ (a) = [`I `]

Comparison between experimental and analytical results

The experimental mounting consists of an empty pipe suspended with free-free boudary conditions. The excitation force is generated by a shaker: acceleration and force are measured at the excitation point. ESPI measurements are performed under harmonic excitation at the resonant frequencies.

Because of multiple eigenvalues, accelerometric measurements do not give enough information for model updating. The MAC matrix between experimental and analytical modes shows that each experimental mode could be represented at least by two analytical modes (FIG. 3-(a)).

On the other hand, we can illustrate the improvment provided by ESPI measurements, as well as the sub-space division for multiple eigenvalues (FIG. 3-(b)(c)). As said before, mats play a decisive role in the durability of catalytic converters. Moreover, mats are not well know in dynamics, because of their structure complexity.

In the following developpement, we present an original method to extract modal parameters of mats [START_REF] Mallat | A Wavelet Tour of Signal Processing[END_REF][4].

Experimental setup

Mats are constituted of ceramic fibers. An organic binder ensures a good cohesion between these fibers in order to facilitate the manipulation. When submitted to heat, this binder flows away: on vehicule, this occurs during the first kilometers. After that and for the whole lifetime of the vehicule, mats hold on ceramic substrates without any binder.

For this reason, we have choosen to work on binder free mats, that is to say with mats that can not be manipulated safely. The experimental characterization bench has been designed in order to be able to study the dynamic response of the 1D oscillator constituted with such a material.

A thermal aged mat, considered as isotropic, is packaged in working like conditions. A harmonic force is applied to the 1D oscillator and suddenly stopped. By observing the dynamic free response we are able to extract a large information field by using the wavelet transform.

Data processing

Despite of the short time of measurement (due to high damping ratio), the wavelet transform gives good results as shown on FIG. 4-(a). Frequency and damping ratio are represented vs. displacement: the behavior shown is viscoelastic. Since the experimental bench can be assimilated to a single-degree-of-freedom system, it can be idealized to a system shown in FIG. 5-(a). Therefore, we can use the concept of viscous damping to represent the damping mechanism of viscoelastic material (FIG. 5-(b)). Mathematically, for By considering the real and imaginary parts of the above equations, it yields to:

✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✂ ✁✂
✄ ✁✄ ✄ ✁✄ ✄ ✁✄ ☎ ✁☎ ☎ ✁☎ ☎ ✁☎ K B F(t) M x(t) ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✂ ✁✂
K R = M ω 2 0 K I = 2ζM ω 2 0 ( 3
)
where ζ is the damping factor of the system. By expressing the displacement field as a function of the geometric parameters, the Lamé coefficient µ is known. As a consequence, the complex Young's Modulus can be computed FIG. 4-(b).

Conclusion

ESPI is a recent measurement technology wich provides a high spatial resolution. The correlation between the model and the real structure has been improved by using ESPI measurements. In addition, we have shown how to solve the problem of multiple eigenvalues: as a consequence, we would be able to perform model updating on cylindrical entities.

Concerning mats, measurements have underlined a non-linear response. The wavelet transform method allows us to extract eigen-frequency and damping ratio of mats as a function of the amplitude. Moreover, it provides a quick testing method: if classical testing method were used, we would have applied the Fourier transform technique on dozens of measurements at different amplitudes. This long and tedious step is favorably replaced by a single measurement followed by a single data processing.

The extraction of the complex stiffness is done as the characterization bench can be considered as a single-degree-of-freedom system. As a consequence, the Young modulus of the material is computed.

This will lead to a dynamic representation of a whole catalytic converter, at ambient temperature. The next step will consist in the evaluation of a wide range type of mats, at different temperatures.

FIG. 1 -

 1 FIG. 1 -Constitutive elements of a catalytic converter

FIG. 2 -

 2 FIG. 2 -Mode shapes extracted from ESPI measurements

  c i ν leads to a transformation matrix [T ]. The last operation consists in calculating the matrix Ỹ (a) = Y (a) [T ] where Ỹ (a) is a new modal matrix representing the same sub-spaces as Y (a) but respecting the principal directions of the experimental eigenvectors Y (e) .

3 -

 3 FIG. 3 -MAC matrix for a pipe, considering accelerometric and holographic measurements

4 -

 4 FIG. 4 -Instantaneous Resonant frequency, damping factor and Young's modulus vs. displacement

FFIG. 5 -

 5 FIG. 5 -Viscous vs. viscoelastic systems

  ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂

  ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂ ✁✂

	✄ ✁✄ ✄ ✁✄ ✄ ✁✄ ☎ ✁☎ ☎ ✁☎ ☎ ✁☎
	✆ ✁✆ ✁✆ ✁✆ ✁✆ ✁✆ ✁✆ ✁✆ ✆ ✁✆ ✁✆ ✁✆ ✁✆ ✁✆ ✁✆ ✁✆ ✝ ✁✝ ✁✝ ✁✝ ✁✝ ✁✝ ✁✝ ✁✝ ✝ ✁✝ ✁✝ ✁✝ ✁✝ ✁✝ ✁✝ ✁✝

Acknowledgments

This work was sponsored in part by the French Agency for Environment and Energy Management (ADEME). Agence de l'Environnement et de la Maîtrise de l'Energie (ADEME) Sophia Antipolis -06560 Valbonne, France