Sarra Mehamel

Samia Bouzefrane

Soumya Banerjee

Mehammed Daoui

Valentina E Balas

Soumya Banarjee

Modified reinforcement learning based-caching system for mobile edge computing

Keywords: Caching, Reinforcement Learning, Fuzzy Logic, Mobile Edge Computing

come

Introduction

In the era of data flood, storing popular contents at the edge is a promising technique to meet user's demands while alleviating the overcrowding on the backhaul. The idea that characterizes Mobile Edge Computing (MEC) is to push the computation/storage resources to the network edges that are in the vicinity of the user equipment (UE) [START_REF] Wang | In-edge AI: Intelligentizing mobile edge computing, caching and communication by federated learning[END_REF]. In the other hand, the introduction of MEC paradigm and the development of advanced applications have led to datatraffic growth as MEC services are relying on big data with different types and of significant amounts. Specifically, mobile edge computing servers (MECS) are equipped with computation, analytic and storage capabilities which deal with the significant amount of content requests leading to improve the quality-of-service (QoS) of the network and the quality-of-experience (QoE) to the end user. Local caches are one of the MEC equipment that helps to reduce the request traffic and to improve the response time. In particular, because data requests can be processed by MEC nodes that are close to the user, edge caching will reduce network traffic and speed up the data retrieval. This allows edge caching to relax the need of continuous connectivity by decoupling the producer and the receiver. However, edge caching has been widely investigated due to the requirements in term of information freshness. Caching is considered 1 a critical bottleneck for the development of MEC systems, since edge caches are located at the edge of the networks and physically closer to the end user. With limited storage, edge caches use different strategies of placement and replacement, which are in general designed to allow spreading only popular contents around the network. In addition, MEC paradigm presents an exclusive opportunity to implement edge caching and to design caching placement and replacement strategies. Therefore, caching popular contents at MECS is considered as a cost effective solution [START_REF] Luo | Energy-efficient caching for mobile edge computing in 5g networks[END_REF] due to the benefits of avoiding network congestion and alleviating the backhaul links. Fulfilling the requirement of both network and the end user is a key point of MEC that consists in a dynamic computational offloading that might cause congestion in the network; which decreases QoS and raises the decision making and the optimization problem on the whole system. Our problem statement is how to jointly encompass the problem of stochastic dynamics of the network and enhance the decision of caching. Several pioneer works about caching in mobile edge computing have been proposed and achieved with quite a good results but these approaches suffer from the following issues:

• Not enough Inputs: They consider only popularity as a key information factor while there are several factors that can effect on the caching decision.

• Strategy combination: They consider either placement or replacement strategies and not the whole caching system.

• Dynamic conditions: Dynamics of the environment and the caching system are not well addressed.

• Tentative isolation: Most of caching strategies do not consider the long term effect of the current caching decision.

Other proposed solutions are optimal but they suffer from a lack of intelligence. In order to fill this gap, We focus in this paper on the design of a novel intelligent caching strategy that tailors mobile edge computing servers among learning methods. As one of learning methods, we propose the use of reinforcement learning (RL) that enables agents to deal with decision making problems by learning through interactions with the environment [START_REF] Zhu | Deep Reinforcement Learning for Mobile Edge Caching: Review, New Features, and Open Issues[END_REF]. Reinforcement learning model has some limitations notwithstanding its efficacy in many fields since scaling is a big challenge [START_REF] Botvinick | Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective[END_REF], [START_REF] Niv | Dialogues on prediction errors[END_REF], [START_REF] Daw | Chapter 16 -Advanced Reinforcement Learning[END_REF]. In fact, RL model does not cope well with areas where there is a large space of possible actions or environment states. Hence, many RL models have been used for simplified learning cases [START_REF] Suomala | Modified Reinforcement Learning Infrastructure[END_REF].

In this context, we define a modified reinforcement learning (mRL) based caching system for MEC, which considers the content features namely frequency, cost and size as well as the device feature like mobility and storage capability. This aims to solve the problem of caching decision making in a realistic way. The main contribution of this paper is to bring a solution to the caching problems in a scenario where caching units are stochastically distributed through MEC servers with a limited backhaul and storage capacity. In particular, we build on a caching system model and define its performance metrics (cache hit ratio and stability). In addition, we define cache storage size, user mobility and content popularity distribution. By coupling the caching decision problem with reinforcement learning layer while relying on recent results from [START_REF] Mehamel | Energy-efficient hardware caching decision using Fuzzy Logic in Mobile Edge Computing[END_REF], we show that a certain hit ratio can be achieved by increasing the total storage size while the number of MECs is fixed. To the best of our knowledge, our work differs from the previous works in terms of studying a new system model with both placement and replacement aspects of edge caching. The remaining content of this paper is organized as follows: background and review are presented in section 2. The modified RL algorithm and its solution process are given in Section 3. Problem formulation and model development are provided in Section 4. Evaluation and discussion of results are given in section 5. Finally, concluding remarks including future directions are given in Section 6.

3 Background and related work

Caching in MEC systems

Mobile Edge Computing (MEC) is a decentralized computing concept in which computing resources and application services can be distributed along the communication path from the point storing data to Base Stations (BSs) in wireless networks [START_REF] Safavat | Recent advances in mobile edge computing and content caching[END_REF] Mobile-edge Computing has been also mentioned in ETSI (European Telecommunications Standards Institute) white paper as an environment that offers applications and content providers cloud-computing capabilities at the edge of the mobile network. This environment is characterized by high bandwidth, low latency and a real-time access to radio network information [START_REF] Hu | Mobile Edge Computing: A Key Technology Towards 5G. ETSI Whitepaper[END_REF]. In particular, generic-computing platforms are used by mobile edge computing servers to be implemented directly at the base stations, which enables the deployment of caching and context-aware services in the vicinity of the mobile users [START_REF] Luo | Energy-Efficient Caching for Mobile Edge Computing in 5G Networks[END_REF]. Consequently, MECS became a unique opportunity to implement and perform edge caching. To describe the scenario of mobile edge caching, we first should discuss the questions of where to cache, what to cache (popular content) and how to cache (caching policies).

Caching improves data availability during disconnection because even when a connection is unavailable, data can still be accessed through the on-board cache, thus allowing mobile clients to continue operating during disconnection [START_REF] Kwong Yuen | Supporting user mobility through cache relocationMobile Information Systems[END_REF].

In general, edge caching means that popular contents can be cached in edge nodes such as macro base stations, small base stations or even user equipment. As shown in figure 1, caching a content which is close to the end users can effectively reduce the redundant data traffic and greatly improve the QoE of users [START_REF] Bastug | Living on the Edge: The Role of Proactive Caching in 5G Wireless Networks[END_REF].

Concerning what to cache, the range of contents is widely increasing including videos, audio files and Internet-of-Things data, which makes the number of reachable contents over the internet extremely huge. Not all of the available contents should be cached, regarding to the fact of limited storage space of edge nodes. In literature, to decide what to cache, content's popularity is one of the prime concern. It represents the probability of requesting contents by users or the frequency of demanding the contents over specific time period. This feature should be taken into consideration as a main factor. Most of the current works that deal with mobile edge caching consider that content popularity follows a static Zipf distribution [START_REF] Leconte | Placing dynamic content in caches with small population[END_REF]. But since the user preferences change with time and the user groups associated with edge nodes are varied, considering only content popularity is not enough as a parameter to infer the caching decision. Appropriate contents are selected by edge caches to be stored using caching policies. The caching policies decide to obtain different objectives such as traffic offloading, quality of experience (QoE), energy consumption and so on in order to maximize the cache hit ratio. Caching policies can be divided into categories depending on their characteristics:

• Depending on cache coordination:

Coordinated: caches must exchange information to be efficient, that is, to have a good estimation of where to place the content and to avoid caching the same content in too many caches. Coordinating cache is represented by a distributed session feature that allows multiple instances of a session to broadcast content changes among each other so that each cache is updated. Uncoordinated: in uncoordinated scheme, each cache is working in an individual way.

• Depending on cache size:

Homogeneous: all caches in base stations have the same size. Heterogeneous: each cache has a different size.

• Depending on the cooperation between caches

Cooperative: caches cooperate with each other by establishing a cache state that allows other caches to know the different states like in [START_REF] James | Cooperative proxy caching for wireless base stations[END_REF]. Non-cooperative: caching decisions are made independently with no need to advertise the information of cache state.

• Depending on where the content is cached:

On path: caching only the contents caught along the downloading path. Off path: caching the content caught outside the downloading path.

In general, conventional caching policies are divided into two main phases: placement phase and replacement phase:

• The placement phase which is the process that decides whether we should cache and how and when we can cache the contents.

• The replacement phase which is the process that decides which data to drop if there is no free storage space.

To make best usage of edge caching, researchers have prosperously used classical web caching algorithms either for placement or replacement phases relying on popularity as a main factor. For placement, they adapt: leave copy everywhere (LCE) which is the default caching mechanism in most caches designs used to minimize the upstream bandwidth demand and downstream latency [START_REF] Zhang | A Survey of Caching Mechanisms in Information-Centric Networking[END_REF]. In LCE, the popular contents are cached at every cache along the path and it is considered as a homogeneous and non-cooperative caching mechanism. The methods leave copy down (LCD ) and move copy down (MCD) realize the heterogeneous caching in a cooperative way in order to reduce redundancy [START_REF] Zhang | A Survey of Caching Mechanisms in Information-Centric Networking[END_REF] [START_REF] Arif | Cache Replacement Positions in Information-Centric Network[END_REF]. Probabilistic mechanisms have been widely used, like the policy referred to Prob(p). It was used as a benchmark scheme in the literature [32] [33] [34]. This mechanism sets to every content a probability p in advance and does not cache the content for a probability equal to (1-p). It is a heterogeneous and non-cooperative strategy. We can also mention Prob-cache Prob-cache is another solution identified as an on-path, heterogeneous and cooperative probabilistic caching mechanism. It adds additional fields to each request and content Time Since Inception (TSI) and Time Since Birth (TSB) [START_REF] Psaras | Probabilistic in-network caching for information-centric networks[END_REF] making the cached content visible to neighboring caches, which enhances the cooperation. Jason Min et al. proposed an explicit cache cooperation named intra-domain cache cooperation. Specifically, it maintains two cache summary tables that record the information of the contents currently cached by the content router, and exchange it with other caches [START_REF] Wang | Intra-as cooperative caching for content-centric networks[END_REF].

For edge caching, classical web caching was widely used . However, several works have proposed new placement strategies for the edge, such as in [START_REF] Zhang | Caching and prefetching content at the edge in the mobilityfirst future internet architecture[END_REF], where they present the edge buffering as a caching and pre-fetching strategy, pointing out the insufficiency of strategies that rely on past history of the device. Instead, they propose a prediction model based on the aggregated network-level statistics. Unlike blind popularity decisions, authors in [START_REF] Mahmood | Mobility-aware edge caching for connected cars[END_REF] proposed a mobility-aware probabilistic (MAP) placing scheme which caches contents at edge servers where the vehicles are connected considering the vehicular trajectory predictions and the time required to serve a content. S. Zhang et al. [START_REF] Zhang | Cooperative Edge Caching in User-Centric Clustered Mobile Networks[END_REF] propose a cooperative edge caching in large scale, where the optimization is carried on the content placement and the cluster size using different features such as traffic distribution, channel quality, stochastic information of network topology and file popularity. Table 1 summarizes some of classical placement strategies.

The replacement phase is the phase that decides which content to evict from the cache. It is categorized into: recency, frequency based such as the least frequently used (LFU) [START_REF] Ibrahim Abdullahi | Cache skip approach for Information-Centric Network[END_REF] and the least recently used (LRU) [START_REF] Podlipnig | A survey of Web cache replacement strategies[END_REF], and semantic like First In First Out (FIFO) that always replaces the oldest contents in the cache [START_REF] Osman | A Comparison of Cache Replacement Algorithms for Video Services[END_REF]. Recently, an agebased caching replacement was developed in [START_REF] Leconte | Placing dynamic content in caches with small population[END_REF]. The previous caching mechanism has been successfully adopted in classical web caching, but it may not meet the performance in the mobile edge caching due to the ignorance of other features and by the use only of popularity. Thus, it would be unable to take advantage of specific characteristics of both the environments and the end user like mobile network topology uncertainty, user mobility, limited storage and cost. In order to overcome these limitations and cope with the future internet usage, the works described in [START_REF] Mehamel | Energy-efficient hardware caching decision using Fuzzy Logic in Mobile Edge Computing[END_REF] and [START_REF] Mehamel | New caching system under uncertainty for Mobile Edge Computing[END_REF] adjust the varied content properties and its influencing factors using fuzzy control caching system (placement and replacement) for edge servers, which can select the more and the less priority content to be cached or evicted respectively.

Fuzzy logic

Fuzzy logic is an extension of Boolean logic dealing with the concept of partial truth which denotes the extent to which a proposition is true. While classical logic holds that everything can be expressed in binary terms (0 or 1, black or white, yes or no), fuzzy logic replaces Boolean truth values with a degree of truth. The degree of truth is often employed to capture the imprecise modes of reasoning that play an essential role in the human ability to make decisions in an environment of uncertainty and imprecision [START_REF] Hassan | Cache replacement engine: A fuzzy logic approach[END_REF].

A Fuzzy Inference System consists of an input phase, a processing phase and an output phase. In the input phase, the inputs are mapped to an appropriate membership function with specific values. The processing stage consists in performing each appropriate rule and in generating a corresponding result. It then combines the results. Finally, the output phase converts the combined result back into a specific output value. The membership function of a fuzzy set represented by divided ranges defines how each value in the input space is mapped to a membership degree. The inference system is based on a set of IF-THEN statements representing logic rules, where the IF part is called the "antecedent" and the THEN part is called the "consequent". The fuzzy inference system (FIS) that we built, in our work, consist of four inputs:

1. Mobility that refers to the distance between the mobile user and the nearest base station that contains the cache.

2. Frequency that represents the popularity or how often the contents are requested in a specific period of time interval.

3. The size of the cache that is determined thanks to cache occupancy.

4. The cost of retrieval that reflects the cost of the bandwidth required by a MECS to extract the content as requested by the end user.

Appropriate rules are applied during the processing phase that generates the corresponding results; and finally, producing a quantifiable result with the assignment of the corresponding membership degree. The aim of using fuzzy logic is to take into consideration the environment and the mobile user factors that affect caching performance to make the caching policy context-aware.

Reinforcement learning

Efficient mobile edge caching policy needs not only to be context-aware, but also to be intelligent to grasp both the environment and the user behavior in order to take optimal or appropriate caching decisions over time. As one of the intelligent learning methods, the reinforcement learning (RL) enables agents to deal with decision making problems by learning through interactions with the environment [3] [28]. As described in [START_REF] Zhu | Deep Reinforcement Learning for Mobile Edge Caching: Review, New Features, and Open Issues[END_REF], the basic components of reinforcement learning enrolment (see figure 2) are the following:

• In the reinforcement learning model, an agent learns through its perpetual interaction with the environment. At each time t and in a state space S, the agent observes a state S t related to its environment, and from an action space A the agent selects an action a t according to a policy π = P(a t |s t ) which is a probability of choosing action a t from state s t .

• The agent earns a reward r t and moves to a new state S t+1 , in respect to the model or to the dynamics of the environment, for reward function R(s, a) and state transition probability P(S t+1 | S t , a t ). The reward is defined as a return R t = ∞ k = 0 γ k r t+k where γ ∈ [0, 1] is a discount factor.

• The agent aims at finding an optimal policy, π * , to achieve from all the states the maximum expected return. The state-value function

V π(s) = E[R t | s t = s] and the action-value function Q π (s, a) = E[R t | s t = s, a t = a]
can measure how π is good. V π (s) represents the expected return from state s and for policy π while Q π(s,a) represents the expected return when selecting action a in state s and following π.

Reinforcement learning based caching policies have been studied in many works, since it is capable of learning the environment and making online decision through observations. Thus, it is a potential solution to decision problems under dynamic environment, such as the caching policy design problem in [START_REF] Chen | Reinforcement Learning Policy for Adaptive Edge Caching in Heterogeneous Vehicular Network[END_REF]. Specifically, a Q-learning based caching scheme was developed in [START_REF] Sadeghi | Optimal and scalable caching for 5G using reinforcement learning of space-time popularities[END_REF] modeling content popularity as Markov processes. while in [START_REF] Amar Abane | A lightweight forwarding strategy for Named Data Networking in low-end IoT[END_REF]the authors applied Q-learning to manage named data networking caching. A more realistic approach have been proposed in [START_REF] Jiang | Multi-Agent Reinforcement Learning for Efficient Content Caching in Mobile D2D Networks[END_REF] where the authors design a content caching strategies in mobile D2D networks using multi-agent reinforcement learning without assuming the knowledge of content popularity distribution.

A policy gradient reinforcement learning based caching scheme was introduced in [START_REF] Somuyiwa | A reinforcement-learning ¨approach to proactive caching in wireless networks[END_REF] by considering the Poisson shot noise popularity dynamics. Using both service cost and popularity, a dual-decomposition based Q-learning approach was presented in [START_REF] Sadeghi | Reinforcement learning for adaptive caching with dynamic storage pricing[END_REF]. Particularly, related studies on edge caching, such as [START_REF] Sadeghi | Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-Time Popularities[END_REF] [27], have shown that reinforcement learning is effectively labor to be effective in joint resource management.

The caching policies adopted by the RL are trained with observations, depending basically on a reward function resulting from its actions. This reward covers a wide range of factors that can influence the performance of the caching system. The caching policy can also be configured by using RL. In this paper, an emphasize is given to combine fuzzy logic and RL. In turn, the policy of caching can be made more adaptive with the proposed design of reward function while proposing the modifications. The highlight of the combination is to prepare an optimal caching decision. The optimal caching is indicative towards multi-objective optimization of cache decision influencing several parameters (mobility, cost, size and frequency).

4 Our proposal: a modified Reinforcement Learning (mRL) over the edge caching system

Our aim, in this paper, is to propose a modified reinforcement learning (mRL) by combining RL and Fuzzy logic as an embodiment of the idea of exploiting previously acquired knowledge and capabilities from others in order to speed up learning in RL. For example, the knowledge about the content distribution, the mobile user properties and the network conditions can be used by the caching agent. The combination of knowledge methods and RL has significant advantages like constructing more efficient caching policies. On one hand, transferring prior knowledge can be used to help training RL agents, thus making convergence to the optimal caching decision easier. On the other hand, using policies learned by other relative networks with RL agents will improve the efficiency and the robustness of the current RL algorithm.

In this section, we focus on the scenario of caching contents in edge nodes, as depicted in figure 3. There are C contents, denoted as c i = 1, ....C : c ∈ C, we note that all mobile users in the system may request R requests. The content popularity is defined as fr, which is the frequency distribution of content requests from all users. S is the content size denoted as: s i = 1, ....S :s ∈ S, the user mobility is modeled by the distance between the user and the nearest base station and finally the cost of retrieving the requested content. The contents distribution is described by MZipf distribution function. For each request, the RL agent in the edge node can make a decision to cache or not to cache according to the decision of fuzzy rules, and if yes, the agent determines which local content shall be replaced if there is no storage space. We assume that content distribution, user mobility, cost and average arrival time of the requests are assigned to each content during t time period. The most important part is how to define a reward function because it affects directly the performance of the algorithm. In order to design a suitable reward function, the key ideais to assign high reward values to content caching actions in order enhance the cache hit ratios. While the loss is the amount of misses, the gain is the amount of cache hits to be enhanced by the newly cached content.

We illustrate an example of applying mRL to the mobile edge caching, where one MEC server is considered and the storage space of the edge cache is initialized to be enough for caching half of the available contents in the network. The MEC server can serve all the requests directly, according to the arrival time. Initially, the caching policy caches locally the contents according to the priority accomplished by the fuzzy system and to the cache size availability. Otherwise, the cache replaces the content less prior by the highest one. Our aim is to find the optimal caching decision by maximizing the cache hit ratio, that is, the number of contents answered by the edge cache. This problem can be solved based on mRL which requires training an agent for representing the policy, and an appropriate reward function that describes how the agent ought to behave. In other words, reward functions have normative contents that provide what the agents should accomplish.

The detailed features are shown in figures 4, 5 and explained below : The environment is modelled thanks to a stochastic finite state where the inputs are the actions sent from the agent (i.e., fuzzy rules and user requests) and where the outputs are the observations and rewards sent to the agent:

• State transition function P(S t+1 | S t , a t ) • Observation (output) function P(O t | s t , a t ) • Reward function R t = ∞ k=0 γ k r t+k .
It is noted that the agent observations depend on his actions, which means that perception is an active process. The agent is a stochastic finite state machine having as inputs: observations and rewards received from the environment, and as outputs : actions sent to the environment.

• State transition function:

S t = f(S t-1 O t , R t , a t )
• Policy/output function: π = P(a t |s t ) The agent's goal is to find the optimal policy and the state function so that to maximize the expected sum of discounted rewards.

Mathematical model

In case of caching decision with fuzzy policy dynamically, the learning rate should be modified. However, the modification of learning rate may alter the conventional method of stochastic gradient descent (SGD) [START_REF] Ch | Stochastic Gradient Descent Escapes Saddle Points Efficiently[END_REF] considerably. We represent the weight adjustment of mRL as follows: new weight = (existing weight -learning rate ×gradient), i.e, :

θ = θ -α δ δ θ ζ θ (1)
with ζ is the tendency of SGD. if α is too small, the gradient descent can be slow. If α is too large, the gradient descent can overshoot the minimum. It may fail to converge or even to diverge. We define a cost function for a neural network. The goal is to minimize this cost function. Hence, it becomes here an optimization problem using SGD. Mathematically, if the cost function (or loss function) is L(w,b), then the goal is to minimize L. For a convex optimization problem like this, we use the derivation of the loss function ∆L. Thus,

w = w -η∆L (2)
while η is the learning rate.

To consider this principle, we initiate certain modifications of η in the reinforcement learning, which will be a considerable modification over SGD. Precisely, we will modify the following parameter for mRL with respect to SGD: momentum : That will help normal SGD by adding a function of the direction of the previous step to a current step. The function is usually a range of [0.1]. Hence, the proposed model of RL can make use of the following configuration changes of learning:

• The learning rate will select new learning values at the end of each epoch/session combining statistical optimization and rejection strategy.

We provide further the following modification :

1. Input search space S.

Find a mean function µ and variance function ρ

3. η m is Modified by optimizing α, a function therefore:

σ m = η∈S α(η; µ(η), ρ(η)) (3)
5 Problem formulation and model development

High level description of mRL

The reinforcement learning based algorithms have been successfully applied to various optimization problems in many domains. However, in order to obtain better solutions for specific optimization problems like our multi awareness caching system. The core of the modified RL algorithm used in this study is to generate a sub-environment based on fuzzy policy as following:

• Fuzzy policy weight : F t , η m .

• Input search space S, the number of initial search

• Update the value of toward an optimum value α *

We minimize the expected value of the objective in next dynamic instances which implies:

policy[ f (θ t ) = policy[ f (µ(θ t-1 ), α t )] (4)
We update the previous learning rate (α t-1 ) to the final value α t . Then, we perform the gradient descent:

σ p = ( f oµ(θ t , α t )/(ηα t ) (5)
We also assume an optimum value of the learning rate and that at each step the descent value will change towards all the sessions. However, we need to introduce a continuous function with the context of noise, hence:

α t = α t-1 -β ∆ θ f (θ t-1 ) + ∆ α µ(θ t-2 , α t-1 ) (6)
The rule can be interpreted as multiplication. This will be an additive adaptation of changing of λ .

Policy improvement

The caching policy can be defined as a set of dynamic artifacts [START_REF] Mannion | Policy invariance under reward transformations for multiobjective reinforcement learning[END_REF]. This phenomenon becomes obvious as the context of caching particularly should be dynamic depending on the condition of network traffic. Hence, the modification of RL also demands essential formal model to cope up with respect to dynamic policies.

Formal model of policy improvement

We define a finite state space S for caching policies including all the possible actions as the policy of caching is stochastic. Therefore, to measure the random changes of the state space of caching denoted by S c , we introduce a transaction kernel function. For simplicity, we assume a finite set of caching policies and we have as follows:

{K sc : τ → [0, ∞]} < ∞ (7)
where K sc is the kernel state space for caching.

In the dynamic context of caching, we may have more than one context. This additional context either can be combined or mixed to derive the improved policy. Let there are two such transitional finite kernels to make decision for caching from S c to τ. Therefore, to evaluate these two kernels, we need to multiply them. Hence, the final policy (as improved) can be formed:

K Q sc1 ⊗ K R sc2 (S Q , A) = τ K Q sc1 (S, dt). u K R sc2 (S R ,t)du × 1A(t, u) (8)
• If the search space is predefined from a stochastic caching mechanism from τ to u for all t ∈τ u ∈U.

• If the policy for both kernel K Q sc1 and K R sc2 can be combined by integration,the policy π chosen randomly for caching decision is known as greedy.

Therefore, if there is a caching context with all four parameters, then obviously we will select a policy that gives a maximum reward. Therefore, the estimated policy for caching decision to a particular network content is:

P E = N E Totalnumbero f trainingparameters (9)
In equation [START_REF] Leconte | Placing dynamic content in caches with small population[END_REF], N E is the number of time this policy has been used and the training parameters are out of our four parameters. Among these parameters, the one that has the highest gain at a particular instance allows us to find the number of iterations that the same policy can trigger; which will assist us in decision making.Hence, as a high level step, we formulate the following algorithms: Algorithm 1 Policy iteration scheme f r ← 0; P E ← 0.5; trails ←P E ×ntrails; for (int i = 0; i ≤ntrails; i++) trails ←P E ×ntrails; end for If ( f r ≤trails) detect greedy (); fr ++; else select A(); Algorithm 2 mRL Caching policy for edge node Input: Parameters of the system including dynamic conditions: Mb, Fr, CO and Cr initialization: initialize the network model: content and requests, fuzzy rule table Learning Loop Choose an action a t from an action space A following the policy based on fuzzy caching system π = P(a t |s t ) Measure the delay of downloading and cache hit ratio according to (A1) Apply greedy policy iteration scheme according to (A2) Update fuzzy rule table Generate the new caching policy EndLoop Cache content based on the optimal policy

Reward function

Recently, there has been novel technologies that replace prediction with a much more efficient way known as reward functions [START_REF] He | A Design of Reward Function in Multi-Target Trajectory Recovery with Deep Reinforcement Learning[END_REF]. Reward functions are used for reinforcement learning models and allow to obtain the final results as a conclusion instead of prediction. For decision making problems, prediction is not the only input as the other fundamental input is judgment.

For our caching system, mRL treats learning and content placement/replacement as a whole operation. The caching policy adopted by the mRL agent is trained with observations, based on a reward resulting from its actions that relate to the factors that affect caching performance such as mobility. Usually, this reward should be chosen in a way that covers a wide range of factors that can affect the performance such as offloaded traffic or QoE.

Finding the best reward function to reproduce a set of observations can be implemented by the maximum likelihood estimation, Bayesian, or information theoretic methods. The system reward represents the optimization objective. In our work, the objective is to maximize cache hit ratio. For our caching approach, we have all the variables that can go into known values. The time of retrieving is determined while distributing the contents and the users requests (including the associated parameters of each). In the scenario of mRL, when the system state s(t) and the system action a(t) are given, the priority can be determined. When the content requests of a typical user arrive, the system can acquire the knowledge from the fuzzy rules, whether the cache should preform the operation of placement or replacement and also satisfies the requests. We defined a network model where several MECs are deployed. At the network edge, the cache characterized by limited resources is connected to the cloud via the backhaul link. Let C = 1, 2, . . . , c, . . . ,C where C denotes the content provider which is located in the cloud. For simplicity, each cache can store X(X ≤ C) contents. Meanwhile, a time-slotted system is considered. Let N = 1, 2, . . . , n, . . . , N with N the users served by the MECS during time slot t. The distribution of user requests and contents is determined by the ZIPFs distribution function [START_REF] Gabaix | Zipf's Law for Cities: An Explanation[END_REF]. Let d c (t) denote the amount of the instantaneous user requests towards C contents during time t. Let I t (t) denote the cache indicator for the considered MECS during time slot t. Specifically, I c (t)=1 (with a corresponding priority) indicates that the cth content is cached during the time slot t and I c (t) = 0 otherwise. Correspondingly, the caching decisions are made according to the fuzzy caching system where the cache hit rate measures the caching performance. Hence, it is used as a reward function in the mRL -caching solution.

In the conventional reinforcement learning, an agent learns how to optimize its behaviors in uncertain environment by executing control polices experiencing the decision of rewards and improvising the policy based on the reward. Without satisfyingly strong reward function or decision, the learning may look very difficult for the present problem of caching the state space (combination of mobility, cost, frequency and cache size) that could be too extended. Therefore the time of retrieving information should be maximum and the learning would be difficult. This type of cases may shape the reward decision of caching as extremely sparse. We can approach three different reward decisions on caching itself as in the following:

1. A reward decision with the sparse value that is multiplied with certain values compared to the goal state. However, this kind of reward decision may also slow down the learning because the agent needs to achieve many actions before getting the decision of good caching or not effective caching 2. Reverse reward in case of collisions where the constraints of caching may suffer from the situation where keeping cost always lower cannot optimize the other values of constraints.

3. Zero reward for any other state (ideally, there may be some situations where no decision can be made satisfying the cost as lower).

The reverse reward can be a primary objective in terms of evaluation of the distance and the time of information to be propagated from an initial base station to a nearest base station. It has been observed that collision may happen not for the initial iteration, but when both the number of iterations are increased and either of the three parameters like Mb, Fr and size of the cache are made dynamic. In most of the cases, the designing of the reward function in terms of decision is amalgamated with the respect to the procedure of designing state space. For example, the cache is a time depending problem. Therefore, the distance covered to reach the nearest base station can make a good reward decision or not effective reward decision. To simplify, we only mentioned a generic reward function customized with the present problem:

• The state value function νπ(s) gives the long term value of state S when following the policy π • The action value function qπ(s, a) is the expected return starting form state s taking an action a as in the following:

νπ(s) = X a ∈ Aπ(a|s)qπ(s, a) (10)
The expected return of decision for caching will be dynamic for caching process. This is because the state of constraints except the cost will change from time to time. This relation is composed with state value function νπ(s) and the action value function qπ(s , a).

In the given expression below, A belongs to the all number of actions and X denotes the all number f states to achieve the based reward decision of caching.

Data and result analysis

In order to validate our work, a catalogue of N= 10 3 contents has been generated with a cache size of 600MB, where each content is associated with a random size. The frequency is calculated depending on the catalogue size and the number of requests by a user. The cache occupancy is defined as the proportion of the content size among the cache size.

With U number of users U = {u,. . . u N }: u ∈ U, the mobility is formed as the distance d = XY between the users in the points X ={x 1 , x 2 ....x U } and the base station in the point Y.

Mb = |Y -X U | (11)
The cost to retrieve each content is the time incurred to retrieve the content to the end user. This time is calculated according to the max bandwidth required to provide peak data rates of at least B = 0.2 Mbit/s, and the distance d= XY between the end user and the base station where the cache is located.

Cost = Mb B (12)
A resource from the catalogue content is an object that contains an ID, a size and a payload that refer to the type of the content we have (video, image, audio). For simplicity, it is set to a string value. Studies have shown that Zipfs law is an appropriate distribution model for the distribution of requests and contents over Internet such as YouTube videos or peer-to-peer file sharing systems. According to Zipfs law, most user requests target a small fraction of popular web contents. Zipfs distributions related to a catalogue of N contents with request probabilities Z(r) have been defined for the objects popularity ranks as in the following:

Z(r) = αr β with : r β < 0; β = Z(1) = 1/ N ∑ r=1 r β > 0 ( 13
)
where α is a normalization constant and β an adaptive shape parameter. R = 10 3 corresponds to the users requests with α set as : α = 0.75.

Based on this distribution law, several characteristics curves can be generated (see appendix for the snap of the data). The characteristics of the curves are defined as following

• Choosing an optimization function to balance the four optimal parameters like frequency, mobility, size and cost. However, the cost as well as the mobility always have to be kept as minimum as possible.

• Deviation of cache size and time of caching is expected.

• The optimization of objective function to balance mobility and cost with the respect to the others generates another section of optimization

• It is also expected that there will be reference mean and an actual mean between the max and the min values of bandwidth. Hence, setting histogram plots could be worthy to demonstrate these variations.

The next subsection describes the analogy behind the data and it demonstrates also the four given components as a holistic performance of the model followed by an optimization characteristic curve.

Results Discussion

We should mention that unbalanced data have been used for two reasons. First, the scenario is a multi-objective [START_REF] Marler | Survey of multi-objective optimization methods for engineering[END_REF]. Second, using unbalanced data requires methods to solve the problems of optimization under nonlinear constraints of inequalities. Hence, we used Karush-Kuhn-Tucker conditions (KKT) which is an optimization problem with intervalvalued objective function [44] [45]. For general characteristics, the deviation of cache size may occur with respect to time of caching (see figure 6). In this figure, there are two segments of curves. The one in red color demonstrates that the cache size is 0.01 unit and the time of caching is only variable (in ms), then the slop of the curve becomes linear and deviates more toward a sustain value. After 3 ms, there is no significant change in deviation of cache size. However, in the same figure, the upper segment shows cache size between [0.02-0.03] units. If it differs, then the slop of the curve can only be flat after 25 to 30 ms time. This describes that the deviation of cache size while keeping the time of caching fixed can be a significant characteristic to the performance of this model. In the histogram plot of figure 7, frequency is a range of 10 3 samples. However, these large samples of ranges of frequency may not serve the RL model as part of the given data shown in appendix. It also shows that the axis x of the plot of histogram can be sustained and becomes non significant after 20000 values of Fr. Therefore, there is a requirement to bifurcate the maximum and the minimum range of frequency by using the actual mean value as a reference as shown in the plot. This will help to generate the data proposed for this model accordingly.

After formulating the data generation and investigating the tendency of cache size with time, the given performance has been focused on a multi objective optimization problem. Here, the four parameters, used to analyze the trusted caching decision, can vary and significantly impure the results. For example in figure 8, it is shown x axes containing an iteration count and y axes containing an optimization function. In this case, the optimization function can vary with cost and mobility, although two variables are kept as minimum with respect to the remaining two variables. Finally, this will produce at least two optimization functions with respect to cost and mobility. Therefore, the data part should be unbalanced to yield an optimal solution of caching. It is shown in the figure that the training data in this model has been occurred from fuzzy model and therefore there may be substantial possibilities to present data in a mixed cluster of any of these four variables. These features provide that there is a necessity of final clustering on the unbalanced data for optimal caching decision. The figure has demonstrated three dots where clearly the middle part shows dense clusters of any of the four variables line in this region. However, the left and right sides of those dots are clearly isolated from these clusters, which means that the middle dense cluster can be suitable for a good caching decision whereas the other may not. These all dots represent the combination of any of the four parameters without any balance ratio. This necessitates the requirement KKT optimization in the analysis part of these data. We followed a simple linear regression model according to Figure 9. In this figure, it is a three dimensional plot to show the relation between cost and frequency and it is shown in the below part of the three dimension plot that regression becomes dense. However, the upper part becomes scattered and we observe that the generated data have not been balanced enough to produce a good outlier decision for caching. Outlier means that each time the data and the iterations of caching session should be proportional. Data is the combination of four parameters as it is unbalanced. Therefore, the relationship between cost and frequency may not provide any significant decision for caching. Additionally, it can be noted that for bandwidth or frequency, we have referred the histogram plot for reference mean and actual mean of the maximum and minimum frequency. Therefore, we require more optimal clustering for Figure 10. Here, we observe that there are three axes in the given plot where Fr, CR, and CO of the content are plotted based on the sample data generated (see appendix). In the given figure, there are three sections based on the distribution of the optimization function:

• The dense blue dots occupy the Fr and the CR region of the plot. This entropy will distribute certain scattered blue dot from CR toward the mapping region of the cost. As it is mentioned, the cost should be kept as minimum. Therefore, very few blue dots are available in the cost region satisfying the variable Fr and CR according to the generated data.

• This plot is a broader outlier analysis of clustering. It describes the plot regions into different visible sections. However, due to the absence of real effective cost data, it is not possible to find more blue dots which may represent appropriate cost of caching with respect to the other parameters.

• The modified RL thus can distribute a map value for state -action and reward clustered for appropriate decision of caching.

In both figures:

• 3D distribution of input data has been shown using scatter diagram. Additionally, these following values are analyzed :

-Means, Median, Std/Variance all attributes, data distribution and histograms.

-Co-variance of pair of attributes.

Pairwise linear regressions.

• Fr = a + b ×Cr.

• Fr = a + b ×Co.

• Fr = a + b × Mb.

• Cr = a + b ×Co.

• Scatter diagram is showing the distribution of the data in all directions. There is no such biased or prominent grouping of data. From the graphical and tabular views, it can be easily found that some models are working with certain errors. This is also to emphasize that apparently it seems to formulate a perfect optimization function based on 4 parameters. However, in real time, it may not be suitable too.

Conclusion

In this paper, we have proposed and demonstrated a realistic decision scheme for caching scenario by using reinforcement learning as a machine learning component. Hence, to incorporate RL in the mobile edge caching system, we proposed certain elementary modifications. This is because following the principles of RL, the state, policies and actions or rewards are continuous variables inter-playing with each other. To consolidate this objective, the present content of the paper has been initiated with fuzzy logic module [START_REF] Mehamel | New caching system under uncertainty for Mobile Edge Computing[END_REF] followed by the inclusion of RL. This is because in normal fuzzy system, the values of membership becoming static in certain cases, may not follow the continuous function. Therefore, the formulation of a hybrid model with fuzzy and RL is required. However, the modification were expected to customize mobile-edge parameters (frequency, mobility, size and cost) while the respective policies should be kept dynamic. These dynamic properties of the proposal can support the different caching scenarios at different intervals depending on the demand and the hit ratio in real time. Certain interesting observations are made from the given proposal. It is demonstrated in the paper that to keep a minimum cost for the whole system and to vary different other parameters except mobility, a well balanced machine learning based optimization function is formulated. The data snap and the analogy have also been described in appendix of the paper for further reference. Considering the demand of the huge bandwidth and big data, the dynamic and intelligent component could foster machine learning and mobile edge caching systems more contemporary.

Finally, we have presented a decision scheme for caching scenario using a modified reinforcement learning model. Our future work will investigate the use of real datasets generated from real information centric networking scenarios. [START_REF] Salsano | Transport-Layer Issues in Information Centric Networks[END_REF] Homogeneous, non-cooperative, on-path caching the content at each node that it traverse along the downloading path what causes caching redundancy. Leave Copy Everywhere (LCD) [START_REF] Laoutaris | The LCD interconnection of LRU caches and its analysis[END_REF] Heterogeneous, cooperative onpath caching the content one hop closer to the user or to the next hop down, it caching only popular content and prevent the unpopular Move Copy Down(MCD) [START_REF] Laoutaris | The LCD interconnection of LRU caches and its analysis[END_REF] Heterogeneous,cooperative, onpath caching all popular content, the policy move the copy of the requested content with a hit to its underlying cache in the path and deletes it, it may suffer from caching pollution on the local replica in case of multiple requests Probabilistic Cache (ProbCache) [START_REF] Chen | Caching in Information-Centric Networking: From a content delivery path perspective[END_REF] Heterogeneous,cooperative, onpath Contents should be cached closer to their destination with higher probability in order to leave caching space Intra-AS Cooperative Caching [START_REF] Wang | Intra-as cooperative caching for content-centric networks[END_REF] Homogeneous, cooperative,offpath allows different caches to serve each other's request leading to solve the limited storage space problem and eliminate redundancy. Least Unified Value(LUV) [START_REF] Wang | Intra-as cooperative caching for content-centric networks[END_REF] Homogeneous, cooperative, onpath, Each cached content is assigned a Least Unified Value(LUV) with cache distance from the content source and cache the content later according to this value WAVE [START_REF] Wong | My cache or yours? Making storage more exclusive[END_REF] Homogeneous, non cooperative, on-path

The contents are cached based on their popularity and counts the access frequency of requests. Opportunistic Caching (OC) [START_REF] Mardham | Opportunistic distributed caching for mission-oriented delay-tolerant networks[END_REF] Homogeneous, non cooperative, off-path Probability based caching policy that use the distance factor beside the popularity factors to cache a content. Congestion-Aware Caching (CAC) [START_REF] Badov | Congestion-aware caching and search in informationcentric networks[END_REF] Homogeneous, non cooperative, on-path it aims to decrease the download time by exploiting the available cache capacity, it is based on two factors the download time and the content popularity Progressive caching policy (PCP) [START_REF] Wang | Progressive caching in CCN[END_REF] Heterogeneous, cooperative, onpath, it is a combination of some existing caching policies that are based on the position of a cache in the network : intermediate or edge , PCP avoid caching unpopular contents and shares the characteristics of both LCD and probabilistic caching The users proximity from the base station FR(% request number) CR (seconds) CO(% of Migabytes) MB ( client IP modeled by distance) 0,7 1 0,01666667 1 1 0,8 0,01666667 0,5 1 0,85 0,01666667 0,1 0,8 1 0,91666667 1 1 1 0,05 0,7 0,6 0,5 0,56666667 0,8 0,55 0,9 0,01666667 1 1 1 0,35 1 0 0,4 0,35 0,8 0,5 1 0,13333333 0,7 1 1 0,03333333 0,45 0,4 0,5 0,05 1 1 0,75 0,05 0,6 0,65 0,6 0,01666667 0,3 0,6 1 0,05 0,3 1 0,45 0,05 0,7 0,4 0 0,13333333 0 1 0,4 0,21666667 1 0,8 0,5 0,01666667 0,2 0,55 1 0,01666667 0,1 0,3 0,7 0,03333333 0,4 0 1 0,05 0,3 1 1 0,03333333 0,1 0,6 1 0,01666667 0,2 0,5 0,9 0,01666667 0,1 1 0,6 0,13333333 1 1 0,9 0,05 0,1 1 1 0,56666667 0,6 0,9 0,1 0,01666667 0,4

  Cr = a + b × Mb and Co = a + b × Mb • Multiple linear regressions (Fr = a + b ×Cr + c ×Co, Fr = a + b ×Cr+ c × MB,Co = a + b × Fr + c × Mb, Fr = a + b ×Cr + c ×Co + d × Mb) • Multiple quadratic regressions Fr = c 0 + c 1 × Cr + c 2 × Co + c 3 × Mb + c 4 × Cr × Cr + c 5 × Co × Co + c 6 × Mb × Mb + c 7 ×Cr ×Co c 8 ×Cr × Mb + c 9 ×Co × MbTo identify possible clustering, results are shown using graphical representations. For all regressions, we have used Least Square optimization techniques for the learning of model constants, i.e., regression coefficients.

Figure 1 :

 1 Figure 1: Edge caching architecture

Figure 2 :

 2 Figure 2: Reinforcement learning model

Figure 3 :Figure 4 :

 34 Figure 3: scenario of caching using reinfocement learning

Figure 5 : cahing Agnet process 23 Figure 6 :Figure 9 :

 52369 Figure 5: cahing Agnet process

Table 1 :

 1 Summary of some existing classical placement mechanisms Placement strategy Type Description Leave Copy Everywhere (LCE)

Table 2 :

 2 List of mathematical symbols

	Symbol	description
	S t	State
	A t	Action
	π	Policy
	R t	Reward function
	V	state-value function
	Q	action-value function
	θ	weight
	η	learning rate
	σ	Gradient descent
	µ	Mean value
	ρ	Variance function
	F t	Fuzzy policy
	K	Kernal space
		Table 3: Notation
	Notation	Description
	S i	cache size
	Id i	Id or Index of content
	N	Number of contents
	R	Number of Requests
	C	content provider
	U	Number of users
	Fr	Frequency of demand of each content in time period
	Cr	The time incurred to retrieve the content
	Co	Occupancy of content in the cache
	Mb

Appendix A

Part of the data :


Fabrice Silva
email: silva@lma.cnrs-mrs.fr

Thomas Hélie

Victor Wetzel

Energy

Keywords:

come

Introduction

The physics of voice has been fruitful in producing a wide variety of models, from full-featured numerical ones (mostly based on FEM for tissues and FVM for airflow) to reduced-order models focusing on the predominant phenomenon. A large body of work in the second category relies on the description of the glottal aerodynamics from the late 1950, based on Bernoulli theorem, thus ignoring rapid or large-amplitude motions of the vocal folds. This is justified on experiments in rigid static larynx-like ducts. However, those simplified models contain an intrinsic paradox where the tissues are driven by the power received by the flow, whose description assumes it does not provide power to tissues.

The main objective of the current communication is to propose a minimal model of the full vocal apparatus that restores a consistent description of the power exchange between the glottal airflow and the vocal folds.

Methods

The model is formulated using the port-Hamiltonian systems (pHS) theory, a framework emphasizing the separation between the intrinsic behavior of components and their interconnection through ports (see Ref. [START_REF] Van Der | Port-Hamiltonian Systems Theory: an Introductory Overview[END_REF] for an introduction). The detailed description of the proposed model is given in Ref. [START_REF] Hélie | 3 rd conf[END_REF]. Its main characteristics is that it includes the simplest kinematics of an incompressible potential flow of inviscid air in the glottis that ensures the continuity of the normal velocity on the surface of the vocal folds (assumed planar for sake of simplicity, see Fig. 1)

Fig. 1: Example of the velocity field during the glottal opening (on left) and representation of the volume flow division (on right).

The capabilities of the proposed model are explored by means of time-domain simulations (notably using the passive guaranteed numerical scheme derived from the pHS theory) and numerical continuation (giving access to bifurcation and limit cycles, i.e., phonation thresholds and steady-state oscillating regimes).

Results

Several cases were explored numerically to investigate the effect of the pre-phonatory configuration and of the asymmetry of the vocal folds.

Discussion

Accounting for the transverse component of the glottal flow leads to a more consistent modeling of the fluid-structure interaction that drives phonation. The results obtained with the proposed model already evidence interesting outcomes such as the contact stresses that apply to the superficial layers of tissues during glottal closure, the production of raucous sounds (quasi-periodic regimes) or the existence of a second possible limit cycle for asymmetric folds that can lead to intermittency and thus to pathological voice.

Increasing the complexity of the model will remain easy thanks to the modularity provided by the pHS framework.

Fig. 2 :Fig. 3 :

 23 Fig. 2: Example of time-domain simulation in a strongly asymmetric configuration (here, right fold stiffer than left fold)

Acknowledgements

We acknowledge the support of the INFIDHEM ANF-DFG project (ANR-16-CE92-0028).


Xueyu Li

Kunpeng Cui

Tao Lin Sun

Lingpu Meng

Chengtao Yu

Liangbin Li

Costantino Creton

Takayuki Kurokawa

Jian Ping Gong

Mesoscale bicontinuous networks in self-healing hydrogels delay fatigue fracture

Load-bearing biological tissues, such as muscles, are highly fatigueresistant, but how the exquisite hierarchical structures of biological tissues contribute to their excellent fatigue resistance is not well understood. In this work, we study antifatigue properties of soft materials with hierarchical structures using polyampholyte hydrogels (PA gels) as a simple model system. PA gels are tough and selfhealing, consisting of reversible ionic bonds at the 1-nm scale, a cross-linked polymer network at the 10-nm scale, and bicontinuous hard/soft phase networks at the 100-nm scale. We find that the polymer network at the 10-nm scale determines the threshold of energy release rate G 0 above which the crack grows, while the bicontinuous phase networks at the 100-nm scale significantly decelerate the crack advance until a transition G tran far above G 0 . In situ small-angle X-ray scattering analysis reveals that the hard phase network suppresses the crack advance to show decelerated fatigue fracture, and G tran corresponds to the rupture of the hard phase network. S uppression and deceleration of crack growth under cyclic loading are observed in load-bearing biological tissues, such as muscles and bones [START_REF] Taylor | Living with cracks: Damage and repair in human bone[END_REF][START_REF] Akkus | Cortical bone tissue resists fatigue fracture by deceleration and arrest of microcrack growth[END_REF]. Such antifatigue ability of biotissues, which is essential for their function, should be related to their exquisitely organized hierarchical structures at different scales. For example, skeletal muscles, having a hierarchical structure ranging from folded proteins through noncovalent bonds at the molecular scale to muscle fibers at the micrometer scale [START_REF] Edgerton | Muscle fibre type populations of human leg muscles[END_REF][START_REF] Fitts | Cellular mechanisms of muscle fatigue[END_REF], sustain repeated stretching and contraction over 1 billion cycles [START_REF] Huxley | Reflections on Muscle[END_REF][START_REF] Hunter | A comparison of muscle with artificial actuators[END_REF] and crack propagation is suppressed even to an injury [START_REF] Taylor | The fracture toughness of soft tissues[END_REF]. On the other hand, common synthetic soft materials are unstructured and usually exhibit poor fatigue resistance as a result of strain concentration at crack tip [START_REF] Mzabi | A critical local energy release rate criterion for fatigue fracture of elastomers[END_REF] or polymer network imperfection [START_REF] Bai | Fatigue of hydrogels[END_REF]. For example, synthetic hydrogels could be made tough and self-healing by introducing noncovalent bonds to dissipate energy [START_REF] Sun | Highly stretchable and tough hydrogels[END_REF][START_REF] Li | Stiff, strong, and tough hydrogels with good chemical stability[END_REF], but these tough hydrogels show poor fatigue resistance [START_REF] Bai | Fatigue fracture of self-recovery hydrogels[END_REF][START_REF] Zhang | Fracture toughness and fatigue threshold of tough hydrogels[END_REF]. Above a threshold of energy release rate G 0 , the crack grows rapidly with the fatigue cycle. It has been revealed that the threshold G 0 is only determined by the primary network structure of the materials, which depends on chemical cross-linking density [START_REF] Zhang | Fracture toughness and fatigue threshold of tough hydrogels[END_REF][START_REF] Zhang | Fatigue of double-network hydrogels[END_REF][START_REF] Zhou | The stiffness-threshold conflict in polymer networks and a resolution[END_REF][START_REF] Creton | 50th anniversary perspective: Networks and gels: Soft but dynamic and tough[END_REF]. The antifatigue properties of soft materials are found to be effectively improved by introducing additional structures to the primary network structure, such as crystals [START_REF] Lin | Muscle-like fatigue-resistant hydrogels by mechanical training[END_REF][START_REF] Lin | Anti-fatigue-fracture hydrogels[END_REF][START_REF] Rublon | Multiaxial deformation and strain-induced crystallization around a fatigue crack in natural rubber[END_REF], composites [START_REF] Li | Fatigue-Resistant elastomers[END_REF][START_REF] Xiang | Stretchable and fatigue-resistant materials[END_REF][START_REF] Wang | Stretchable materials of high toughness and low hysteresis[END_REF], ordered folded units [START_REF] Kruzic | Materials science. Predicting fatigue failures[END_REF][START_REF] Suhr | Fatigue resistance of aligned carbon nanotube arrays under cyclic compression[END_REF], and so on. These results suggest that hierarchical structures may contribute to reduce the strain/stress concentration at the crack tip and hence be beneficial for fatigue resistance in these kinds of soft materials. However, the mechanism by which well-designed hierarchical structures at microscopic level can mitigate or suppress crack advance remains poorly understood.

In this work we attempt to elucidate the role of hierarchical structures on the fatigue resistance of self-healing soft materials, focusing on the mechanism of suppression of crack advance and delayed antifatigue-fracture. We chose the recently developed self-healing polyampholyte hydrogels (PA gels) [START_REF] Sun | Physical hydrogels composed of polyampholytes demonstrate high toughness and viscoelasticity[END_REF][START_REF] Ihsan | Self-healing behaviors of tough polyampholyte hydrogels[END_REF][START_REF] Sun | Molecular structure of self-healing polyampholyte hydrogels analyzed from tensile behaviors[END_REF] as model materials. These PA gels, containing ionic bonds at the 1-nm scale, a cross-linked polymer network at the 10-nm scale, and isotropic bicontinuous hard/soft phase networks at the 100-nm scale (Fig. 1), exhibit a multiscale response to loads and dissipate large amounts of energy to show a very high toughness [START_REF] Cui | Multiscale energy dissipation mechanism in tough and self-healing hydrogels[END_REF]. Such hierarchical structures make the PA gels good candidates as a simple model system to study the effect of hierarchical structures on the fatigue behavior of self-healing materials containing abundant noncovalent bonds. By combining fatigue measurement and time-resolved synchrotron radiation small-angle X-ray scattering (SAXS) we discovered that, upon fatigue cycling, the bicontinuous phase networks in PA gels form a transient oriented structure to induce a pronounced crack blunting and crack deceleration effect, resulting in an extremely slow mode of crack growth above the energy release rate threshold G 0 . This mode is maintained until the energy release rate exceeds a transition G tran that is several times higher than G 0 . Above G tran , the stress-carrying hard phase network ruptures, resulting in an abrupt jump of crack growth to a fast mode. Thus, the 100-nmscale bicontinuous hard/soft phase networks substantially decelerate the fatigue fracture and remarkably enhance the fatigue resistance of the tough hydrogels.

Results and Discussion

Tuning Hierarchical Structures in Self-Healing Hydrogels. PA gels used in this study were obtained from a chemically cross-linked copolymer consisting of the oppositely charged monomers sodium p-styrenesulfonate (NaSS) and methyl chloride quaternized N,N-dimethylamino ethylacrylate (DMAEA-Q) (SI Appendix, Fig. S1). The opposite charges on the polymer chains form ionic bonds when the small counter ions (Na + and Cl -) are removed from the PA gels by dialysis of the samples in pure water. The formation of these ionic bonds induces local aggregation of the polymers, and as a result the gels form a dense hard phase and sparse soft phase interconnected at ∼100-nm scale [START_REF] Cui | Multiscale energy dissipation mechanism in tough and self-healing hydrogels[END_REF]. Such a structure acts as bicontinuous hard/soft phase networks. The primary network structure in the 10-nm scale depends on the formulation of monomer and cross-linker concentrations in precursor solution, which controls the cross-linking density of the network from the chemical cross-linker and from trapped polymer entanglement, while the contrast and the size of the hard/ soft phase networks depend on the primary network structure as well as the temperature at which the dialysis of the PA gels was performed. In this work, we adopted the samples with the same primary network structure but different phase network structure. For this purpose, the samples were synthesized from the same formulation but dialyzed at different temperatures (see SI Appendix for sample preparation). The samples are coded as PA-T dial , where T dial (5 °C, 30 °C, and 60 °C) stands for the dialysis temperature. The water content and thickness of these samples were ∼46 wt % and ∼1.64 mm, respectively, independent of T dial .

First, we show the structure and mechanical behavior of these PA gels. As shown in Fig. 2A, SAXS images of the samples show an isotropic ring and a peak in one-dimensional (1D) scattering profiles, attributed to the bicontinuous phase networks consisting of a soft network (with a low modulus) and a hard network (with a high modulus) [START_REF] Cui | Multiscale energy dissipation mechanism in tough and self-healing hydrogels[END_REF][START_REF] Cui | Effect of structure heterogeneity on mechanical performance of physical polyampholytes hydrogels[END_REF]. The intensity of the peak I m significantly increases with the increase of T dial . Furthermore, the average mesh size of the phase networks, related to the periodicity of the hard and soft phases (d 0 ), increases slightly with the increase of T dial (Fig. 2D). These results indicate that a higher dialysis temperature induces stronger phase separation, to form bicontinuous networks with a higher density contrast and slightly larger size. Such a temperature effect should be attributed to an enhanced hydrophobic interaction of the polymers at high temperature. The tensile behaviors of gels with different phase network structure exhibit little difference (Fig. 2B).

Deformation of Mesoscale Bicontinuous Phase Networks under

Tensile Test. The deformation of the phase networks of PA gels shows three regimes, as observed by in situ time-resolved SAXS during a uniaxial tensile test (SI Appendix, Fig. S3 and Fig. 2C). In regime I, the bicontinuous networks deform affinely, assuming incompressibility. That is, the mesh size of the phase networks in the parallel (d // ) and perpendicular (d ⊥ ) direction obeys d // /d 0 = λ and d ⊥ /d 0 = λ -1/2 , respectively, with the global elongation ratio λ. As previously demonstrated ( 28), the loading-unloading curves exhibit a fully recoverable hysteresis in the affine deformation regime. During loading, the ionic bonds break, and the polymer chains in a globule conformation are unfolded, contributing to energy dissipation responsible for toughening the materials. During unloading, the entropic elasticity drives the polymer chains back to their original globule conformation. After a certain waiting time, the temporary ionic bonds formed in the deformed state dissociate and the original ionic bonds in the undeformed state are reformed to show full recovery of structure and properties. In regime II, d ⊥ /d 0 decreases slowly and deviates from affine deformation. In this regime, the first loading sample softens but fully shrinks back to its original size after a sufficiently long waiting time, which indicates that the hard phase network ruptures, while the soft phase network stays intact, providing the entropic restoring force [START_REF] Cui | Multiscale energy dissipation mechanism in tough and self-healing hydrogels[END_REF]. In regime III, d ⊥ /d 0 increases slightly with λ, and the failure of the whole sample occurs soon thereafter, by rupture of the soft phase network. We observe that the maximum stretch ratio for the affine-like deformation of the phase network, λ affine , slightly increases with the dialysis temperature of the samples (Fig. 2D). This order is consistent with the change in the degree of phase contrast and mesh size d 0 of the two networks that both increase with dialysis temperature.

Suppressed Crack Advance and Delayed Fatigue Fracture. Then, we explored the cyclic fatigue behavior in uniaxial tension of these PA samples with differently tuned bicontinuous phase networks. The fatigue experiments were performed using prenotched samples in the pure shear geometry (Fig. 3C, Inset) in a humidity-controlled box (SI Appendix, Fig. S2A), following the method by Suo and coworkers [START_REF] Tang | Fatigue fracture of hydrogels[END_REF]. The cyclic fatigue behaviors are found to be sensitively depended on the relationship between the maximum stretch ratio λ max applied in the cyclic fatigue test and the maximum affine deformation ratio λ affine observed in the tensile test. As shown in Fig. 3 A-C for the PA-30 as a representative, we observe a striking difference for λ max = 2.94 (<λ affine ) and λ max = 3.19 (>λ affine ). For λ max = 2.94, a blunt crack tip forms progressively (Fig. 3A). The crack length c initially increases fast, but crack growth slows down soon to a much slower crack propagation rate with increasing cycle N. The average crack advance rate at steady state Δc/ΔN is extremely small (Δc/ΔN = 0.028 μm per cycle; red sphere in Fig. 3C). On the other hand, for λ max = 3.19, a relatively sharp crack tip is observed (Fig. 3B), and the crack length c always increases fast with the number of cycles N. The samples failed after only 800 cycles, showing a fast crack propagation rate Δc/ΔN (∼59 μm per cycle; red diamond in Fig. 3C), three orders of magnitude larger than that of the sample tested at λ max = 2.94.

We observed similar behaviors for PA-5 where the crack length c also initially increased fast, slowed down, and finally hardly increased at a representative elongation ratio λ max = 2.54 (<λ affine ), while it increased rapidly until fracture of the whole sample at λ max = 2.94 (>λ affine ) (SI Appendix, Fig. S4A). PA-60 also exhibits the same suppression of crack advance (crack blunting) and fast crack extension behavior at λ max = 2.94 (<λ affine ) and λ max = 3.84 (>λ affine ), respectively (SI Appendix, Fig. S4B).

The λ max dependences of the crack growth rate Δc/ΔN for the three materials with different T dial are shown in Fig. 3D. All three of the materials show no measurable crack growth over 40,000 cycles below a threshold value of λ max defined here as λ 0 . Above λ 0 , the crack started to grow slowly at steady state. A transition of Δc/ΔN from a slow crack growth mode to a fast crack growth mode occurred at a value of λ max that we define as λ tran . Specifically, the three materials show a similar threshold value λ 0 ∼2, and the crack advances very slowly (Δc/ΔN < 0.1 μm/cycle) at λ 0 < λ max < λ tran , as a result of crack blunting. At λ tran = 2.89 ± 0.07, 3.14 ± 0, and 3.44 ± 0.14 (SI Appendix, Fig. S5) for PA-5, 30, and 60, respectvely, a jump of crack growth rate Δc/ΔN to the fast mode (Δc/ΔN > 10 μm/cycle) occurs (Fig. 3D). The values of λ tran are almost coincident with the values of λ affine observed by the tensile test (Fig. 2D). This result suggests that λ tran corresponds to the limit of affine deformation ratio for the fatigue test, and that the high fatigue resistance is achieved when the hard phase network is intact.

We consider that the coincidence of λ tran and λ affine is due to two reasons. One is that the rupture of the hard phase network occurs at similar elongation ratio λ in the fatigue test (in 10 mm × 50 mm pure shear geometry) and in the tensile test (in 16-mm × 7.5-mm strip geometry). In fact, the stress-elongation curves of the uniaxial tensile tests in pure shear geometry and in strip geometry almost overlap with each other at small λ values, where the hard phase network is in the region of affine deformation (SI Appendix, Fig. S6). Another reason is that there is no strong stress concentration around the crack tip before the rupture of the hard phase network, which is seen in Fig. 3A.

The maximum applied energy release rate G was also calculated from the work of extension of the unnotched sample [START_REF] Bai | Fatigue of hydrogels[END_REF][START_REF] Tang | Fatigue fracture of hydrogels[END_REF][START_REF] Long | Fracture toughness of hydrogels: Measurement and interpretation[END_REF]. To obtain a meaningful elastic energy release rate, the hysteresis energy U hys (λ max ) that is dissipated during each cycle by the breaking/reforming of the ionic bonds during loading is excluded from the work of extension, and G was calculated from the equation G = W e ðλ max ÞH 0 , where H 0 is the initial distance between the two clamps and W e ðλ max Þ is the released elastic strain energy density upon unloading for the unnotched sample up to the stretch ratio λ max under the 3,000th cycle (at which the sample reached the steady state) (SI Appendix, Fig. S2 B andC). Even after 3,000 cycles the hysteresis loop is still very large (SI Appendix, Fig. S2D), indicating that breaking and reforming of the ionic bonds reaches a dynamic equilibrium for that applied deformation rate. Within the range of applied G, two characteristic crack growth modes are observed on a Δc/ΔN versus G plot: a slow crack growth mode with a linear relation Δc/ΔN = β(G -G 0 ) at G 0 < G < G tran (Fig. 3E) and a fast crack growth mode after a jump at G = G tran (Fig. 3F), where G 0 is the fatigue threshold, below which the crack does not grow, β is the slope of the linear response region, and G tran is the energy release rate at which the crack growth rate transits from slow to fast mode. G 0 and G tran correspond to λ 0 and λ tran, respectively. High G 0 and G tran as well as low β values mean high fatigue resistance. The parameters G 0 , β, and G tran extracted from Fig. 3 E and F are summarized in Table 1. The β is reduced and G tran is much increased for gels dialyzed at high temperature, consistent with a high value of λ affine and a delay in fatigue fracture. The G 0 for these three samples is almost the same (∼68 J/m 2 ). This is consistent with the reports by Suo and coworkers showing that the threshold G 0 is determined by the chemically cross-linked polymer network structure (9, 13, 15), which is the same for the three samples. Furthermore, G 0 has the same order of magnitude as the energy required to rupture the polymer strands across the fracture plane, Γ 0 , predicted by the Lake-Thomas model [START_REF] Lake | The strength of highly elastic materials[END_REF]. The G tran is more than twice higher than G 0 for PA-5, and it increases to about three times and 3.5 times for PA-30 and PA-60, respectively. Since G tran corresponds to the maximum affine elongation ratio λ affine of the phase networks, the abrupt increase of the crack growth rate at G tran should be related to the breaking of the hard phase network.

The Role of Bicontinuous Phase Networks in Antifatigue Fracture. To investigate the effect of the bicontinuous phase networks on the fatigue-resistance behavior, we performed in situ time-resolved SAXS experiments to observe the evolution of the structure of unnotched gels under cyclic tensile deformation with the strip geometry (SI Appendix, Fig. S3A). Fig. 4 A and C show the stretch ratio of the phase networks in the perpendicular direction d ⊥ /d 0 versus the maximum applied stretch ratio λ max at the first, third, 300th, and 1,000th cycles for λ max = 2.54 (<λ affine ) and λ max = 3.44 (>λ affine ), respectively. The PA-5 sample was adopted as an example. In the case of λ max < λ affine , the d ⊥ /d 0 always obeys affine deformation from the first to the 1,000th cycles, except at very small λ (Fig. 4A). The latter should be attributed to the residual strain at unloading state. For our consecutive cyclic tensile test at a strain rate 1 s -1 , the sample has not enough time to reach full recovery when the sample approaches full unloading (SI Appendix, Fig. S7). The result suggests that the deformation is reversible under cyclic loading-unloading except for very small λ. This result confirms that the hard network remains intact in cyclic loading. The representative two-dimensional (2D) SAXS patterns at different λ for the 1,000th cycle are shown in Fig. 4 A, Inset. The scattering intensities exhibit a strongly elongated spot in the perpendicular direction at large deformation (λ = 2.54), suggesting the formation of highly anisotropic structure of the phase networks during the cyclic loading. It is also confirmed that no strain-induced crystallization occurs during the cyclic loading. The highly anisotropic hard strands of the intact phase network may carry, rather homogeneously, most of the stress during loading, as suggested by other oriented hard components in composite materials [START_REF] Lin | Muscle-like fatigue-resistant hydrogels by mechanical training[END_REF][START_REF] Li | Fatigue-Resistant elastomers[END_REF][START_REF] Xiang | Stretchable and fatigue-resistant materials[END_REF], preventing the stress concentration necessary for crack growth as schemed in Fig. 4B. This is similar to the effect of strain-induced crystallization during fatigue test in elastomers, where the hard crystal reinforces crack tip [START_REF] Rublon | Multiaxial deformation and strain-induced crystallization around a fatigue crack in natural rubber[END_REF][START_REF] Rublon | In situ synchrotron wide-angle x-ray diffraction investigation of fatigue cracks in natural rubber[END_REF][START_REF] Mars | A literature survey on fatigue analysis approaches for rubber[END_REF]. This explains why crack blunting occurs ahead of the crack tip at λ max < λ affine during the fatigue test in this work.

On the contrary, when cyclic loads are performed up to λ max > λ affine (Fig. 4C), d ⊥ /d 0 becomes increasingly larger with increasing number of cycles, deviating from the value predicted by the affine deformation. This suggests that the hard strands gradually rupture with increasing number of fatigue cycles, and the force-carrying ability of the hard strands decreases (SI Appendix, Fig. S8). Meanwhile, due to the fracture of hard strands during cyclic loading, the recovery and reassociation ability of ionic bonds with their original pairs of the virgin sample decreases, resulting in a decrease of d ⊥ /d 0 at small λ with cycles, as well as a more anisotropic pattern at λ = 1 after cyclic loading at λ max = 3.44 (>λ affine ) compared with at λ max = 2.54 (<λ affine ) (Fig. 4 C, Inset), and a much smaller hysteresis loop U hys at λ max = 3.14 (>λ affine ) than at λ max = 2.34 (<λ affine ) (SI Appendix, Fig. S2D). Once the hard strands fracture, the soft strands with sparse polymer density may carry most of the load, leading to a fast crack propagation under fatigue test as schemed in Fig. 4D, similar to a single network system [START_REF] Tang | Fatigue fracture of hydrogels[END_REF][START_REF] Zhang | Fatigue fracture of nearly elastic hydrogels[END_REF]. That explains why we see a crack velocity jump when the maximum cyclic λ max exceeds λ affine (Fig. 3D).

To further verify the role of the hard strands for the suppression of the fast fatigue crack growth, we turned off the hard strands by preloading the unnotched sample at λ pre > λ affine and then performed the fatigue tests immediately at λ max < λ affine (Fig. 5A). The sample with ruptured hard strands exhibited the fast crack growth mode even at λ max < λ affine (Fig. 5B). We also preloaded the sample at λ pre < λ affine , a stretch at which the hard strands are not damaged. This preconditioned sample showed even slightly better fatigue resistance than the pristine sample without preconditioning. These results clearly show that the presence of hard strands suppresses the fast crack growth in fatigue fracture.

Once the hard strands rupture, the PA gels with intact soft strands act as a single network gel. If we treat the dependence of Δc/ΔN with G in the fast crack propagation regime (G > G tran ) as a linear relation, the slope β 2 is 3.1 × 10 -6 ∼ 4.3 × 10 -6 m 3 /(J•cycle) (SI Appendix, Fig. S9), close to the β value observed for single networks of PAAm gel (Table 1). The slope β 2 slightly increases with dialysis temperature of the samples, which could be associated with the increasing degree of phase contrast of the samples prepared at high dialysis temperature (Fig. 2A). A higher phase contrast indicates a higher concentration contrast and a higher modulus ratio between the hard phase network and the soft phase network. The PA-60, having the relatively weaker soft phase network among the three samples, exhibits the largest exponent β 2 . This result indicates that the crack grows faster with a weaker soft phase network when the hard phase network has ruptured. Such a mechanism leads to notch sensitivity of the PA gels. The prenotched sample ruptures at much smaller elongation ratio than the unnotched sample when being compared at the same pure shear geometry, due to rupture of the hard phase network (SI Appendix, Fig. S10).

Thanks to the role played by hard phase network in PA gels, the fatigue fracture slowed down dramatically above the threshold G 0 . The crack advance rate Δc/ΔN is less than 0.1 μm per cycle and β is as low as 10 -10 m 3 /(J•cycle) below G tran . To see if this fatigue fracture behavior of PA gels is unique to the selfhealing PA gels with mesoscale structure, we summarized the reported fatigue fracture behaviors of various hydrogels (9, 12-14, 35) (SI Appendix, Fig. S11 and Table 1). We see that the single-network PAAm gels and the double-network PAMPS/ PAAm gels, free of physical bonds, only exhibit a single mode increase of Δc/ΔN above the threshold G 0 , although in the double-network gels the crack growth is significantly suppressed, resulting in a very large G 0 . On the other hand, the tough and self-healing gels with physical bonds (PAAm-PVA and Ca-alginate/ PAAm) also show multimode-like fatigue behaviors above the threshold G 0 , though not as prominently as the PA gels. Especially in the case of Ca-alginate/PAAm gels with ionic cross-links, a relatively clear slow to fast crack growth transition is observed at G tran = 84 J/m 2 . These results suggest that these gels containing physical bonds may also contain mesoscale structure. In fact, phase separation was observed in PAAm-PVA gels [START_REF] Li | Stiff, strong, and tough hydrogels with good chemical stability[END_REF][START_REF] Mishra | On the mechanical strength of biocompatible semi-IPNs of polyvinyl alcohol and polyacrylamide[END_REF]. We also see that the β of PA gels and Ca-alginate/PAAm gels with ionic bonds is orders of magnitude smaller than the PAAm-PVA hydrogels without ionic bonds, which suggests that strong physical bonds play a role to suppress the crack growth in the slow mode regime. These results clearly show that the interplay between different scales of the gels substantially delays the fatigue crack growth.

Conclusion

Most amorphous hydrogels exhibit extremely poor fatigue resistance. Above the fatigue threshold G 0 , which can be predicted by the classic Lake-Thomas theory, the fatigue crack grows rapidly to cause sample failure. Thus, the fatigue threshold G 0 has been thought of as the most critical parameter for hydrogels under cyclic loading. In this work, we discovered that the selfhealing PA hydrogels with a hierarchical structure exhibit high fatigue resistance due to the existence of multimode fatigue behaviors. The 10-nm-scale polymer network structure determines the fatigue threshold G 0 for the onset of crack growth, while the phase networks at the 100-nm scale, having ionic bonds at the 1-nm scale as building blocks, decelerate the advancing of fatigue crack dramatically until reaching a transition G tran . Above G tran , which is directly related to the rupture of the hard phase network, fast growth of the fatigue crack occurs. The synergistic effect between the structures at different length scales results in high fatigue resistance. As G 0 and G tran are related to the energy release rates at rupture of the primary network and the hard phase network, respectively, increasing the network density and rupture elongation ratios of these networks at different length scales should enhance the fatigue resistance of the materials. Furthermore, increasing the hierarchy number should induce high-order G transitions with large values. The role of hierarchical structures on the delay of fatigue crack advancing elucidated in this work should guide the designing of antifatigue soft materials. It also provides clues to understanding the fatigue process of biological tissues, which have a more sophisticated hierarchical structure [START_REF] Meyers | Structural biological materials: Critical mechanics-materials connections[END_REF][START_REF] Reznikov | Fractal-like hierarchical organization of bone begins at the nanoscale[END_REF][START_REF] Atala | Engineering complex tissues[END_REF].

Materials and Methods

Materials. The raw materials cationic monomer DMAEA-Q, anionic monomer NaSS, chemical cross-linker N,N-methylenebis(acrylamide) (MBAA), and ultraviolet initiator α-ketoglutaric acid (α-keto) were purchased from Wako Pure Chemical Industries, Ltd and used as received.

Preparation of PA Gels. PA gels were synthesized by a one-step random copolymerization of precursor aqueous solution containing DMAEA-Q, NaSS, MBAA, and α-keto according to previous work [START_REF] Sun | Physical hydrogels composed of polyampholytes demonstrate high toughness and viscoelasticity[END_REF][START_REF] Ihsan | Self-healing behaviors of tough polyampholyte hydrogels[END_REF][START_REF] Ihsan | A phase diagram of neutral polyampholyte-from solution to tough hydrogel[END_REF]. The total monomer concentration is C m = 2.0 M, and both MBAA and α-keto content are 0.1 mol % (relative to C m ). All details associated with sample preparations, measurement of water content, tensile tests, fatigue tests, and SAXS measurements are available in SI Appendix. Data Availability. All data are included in the main text and SI Appendix.

  fatigue resistance | hierarchical structure | phase network | crack growth | affine deformation

Fig. 1 .

 1 Fig. 1. Illustration for hierarchical structures in PA gels. PA gels are composed of hierarchical structures containing ionic bonds (sacrificial bond) at the 1-nm scale (indicated by blue arrow), cross-linked polymer network with mesh size at the 10-nm scale (indicated by red arrow), and bicontinuous hard/soft phase networks with mesh size at d 0 ∼100-nm scale (indicated by black arrow).

Fig. 2 .

 2 Fig. 2. Structure and tensile behavior of PA gels used in this study. (A) The 1D SAXS profiles of undeformed gels dialyzed at various temperatures T dial , coded as PA-T dial . (Inset) The 2D SAXS images. (B) Tensile behaviors of the gels. (C) Phase network deformation ratio in parallel (d // /d 0 ) and perpendicular (d ⊥ /d 0 ) directions versus global elongation ratio λ. The orange lines stand for the affine deformation of the phase networks. The maximum affine deformation λ affine for each sample is indicated by arrows. (Inset) For clarity, the data of PA-30 are shown, and three deformation regimes are marked. (D) Summary of SAXS peak intensity I m , average phase network mesh size d 0 , maximum affine elongation ratio of phase networks λ affine in tensile test, and critical elongation ratio of fatigue test for fast crack growth, λ tran (extracted from Fig. 3D). a.u., arbitrary unit.

Fig. 3 .

 3 Fig. 3. Delayed fatigue fracture of PA gels. (A and B) Shape of crack tip for fatigue test under λ max = 2.94 (<λ affine ) at 30,000 cycles (A) and λ max = 3.19 (>λ affine ) at 100 cycles (B) for PA-30 as a representative. (C) Crack propagation length c as a function of cycle N at elongation ratio λ max = 2.94 and 3.19 for PA-30. (Inset) The geometry of the sample applied in the fatigue test (width L 0 = 50 mm, height H 0 =10 mm, initial notch length c 0 = 10 mm). (D) Δc/ΔN as a function of λ max for gels dialyzed at various temperatures T dial . A transition for crack propagation from slow to fast mode is observed universally at λ tran ∼ λ affine for all of the samples. The comparison of λ tran and λ affine is shown in Fig. 2D. (E) The enlarged slow mode regime for Δc/ΔN as a function of energy release rate G. A linear relation Δc/ΔN = β(G -G 0 ) at G 0 < G < G tran , is observed (as schemed in Inset). The G 0 , β, and G tran are shown in Table 1. (F) The Δc/ΔN as functions of G in log-log plot. Δc/ΔN jumps from a low plateau (<0.1 μm per cycle) to a high value (>10 μm per cycle) at G = G tran . The dashed lines are guides for the eyes. The average crack advance velocity Δc/ΔN is estimated from the slope at steady state. The Δc/ΔN values at the boundary of the transition regime in D, E, and F are obtained from the statistical average values of the high probability mode at the corresponding λ max (see SI Appendix, Fig. S5).

Fig. 4 .

 4 Fig. 4. Structure evolution of PA gels under cyclic tensile deformation. (A and C) Microscopic deformation under cyclic loading at λ max = 2.54 (<λ affine ) (A) and λ max = 3.44 (>λ affine ) (C) for unnotched PA-5 gel. (Inset) The SAXS patterns (the stretching is along horizontal direction) at representative λ under the 1,000th cycle. The orange solid line in each figure is for affine deformation. The dashed lines are guides for the eyes. The black arrow in each figure is the maximum λ to show affine deformation λ affine . (B and D) Illustration of bicontinuous phase network deformation and rupture at λ max of fatigue test for λ max < λ affine (B) and λ max > λ affine (D), respectively. The red starburst pattern in D is the rupture of hard phase network.

Fig. 5 .

 5 Fig. 5. Turning off the hard phase network and its effect on fatigue behavior. (A) The experimental protocol to turn off the hard phase network: The unnotched PA-5 sample is first preloaded at λ pre = 3.44 (>λ affine ) for 1,000 cycles, then a notch is made and fatigue test is performed at λ max = 2.54 (<λ affine ). For comparison, a sample with λ pre = 2.74 (<λ affine ) is also studied. The hollow arrows indicate cyclic loading. The single-line arrow indicates that fatigue test for notched sample is performed after the cyclic preloading. (B) Fatigue experimental results for samples experienced different preloading: blue, λ pre = 3.44 (>λ affine ) with ruptured hard phase network; red, λ pre = 2.74 (<λ affine ) with undamaged hard phase network; and black, pristine sample without preloading.

Table 1 .

 1 Comparison of fatigue parameters of PA gels with various hydrogels The Young's modulus E is tested under the same condition for the fatigue test.

	Sample	E*, MPa	G 0 (λ 0 ), J/m 2	β, m 3 /(J•cycle)	G tran , J/m 2
	PA-60	0.14	71.1 (1.94)	4.1 × 10 -10	236.2 ± 15.0
	PA-30	0.14	69.8 (1.94)	6.1 × 10 -10	207.6 ± 0
	PA-5	0.14	67.3 (2.04)	7.3 × 10 -10	165 ± 12.7
	Alginate/PAAm (13, 41)	∼0.1	35 (1.9)	(1.3 ∼ 2.3) × 10 -9	84
	PAAm-PVA (12)	∼0.03	9.5 (1.4)	5.3 × 10 -7	40
	PAMPS/PAAm DN (14)	∼0.3	200 ∼ 418 (∼1.5)	(1.3 ∼ 4.5) × 10 -9	-
	PAAm (30, 35)	∼0.04	4.3 ∼ 64.5 (∼1.3)	(0.3 ∼ 3.4) × 10 -7	-

*

ACKNOWLEDGMENTS. This work was supported by Japan Society for the Promotion of Science KAKENHI grants JP17H06144 and JP19K23617; and by ImPACT Program of the Council for Science, Technology and Innovation of the Cabinet Office, Government of Japan. The SAXS experiments were performed at the Chinese National Centre for Protein Science Shanghai BL19U2 beam line at Shanghai Synchrotron Radiation Facility. The Institute for Chemical Reaction Design and Discovery was established by World Premier International Research Initiative, Ministry of Education, Culture, Sports, Science and Technology, Japan.