Nathan Grinsztajn
email: nathan.grinsztajn@inria.fr

Toby Johnstone
email: toby.johnstone@aqemia.com

Johan Ferret

Philippe Preux

Deep Reinforcement Learning Workshop at NeurIPS 2022 BETTER STATE EXPLORATION USING ACTION SE-QUENCE EQUIVALENCE

Keywords:

Incorporating prior knowledge in reinforcement learning algorithms is mainly an open question. Even when insights about the environment dynamics are available, reinforcement learning is traditionally used in a tabula rasa setting and must explore and learn everything from scratch. In this paper, we consider the problem of exploiting priors about action sequence equivalence: that is, when different sequences of actions produce the same effect. We propose a new local exploration strategy calibrated to minimize collisions and maximize new state visitations. We show that this strategy can be computed at little cost, by solving a convex optimization problem. By replacing the usual ϵ-greedy strategy in a DQN, we demonstrate its potential in several environments with various dynamic structures.

INTRODUCTION

Despite the rapidly improving performance of Reinforcement Learning (RL) agents on a variety of tasks [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF], they remain largely sample-inefficient learners compared to humans [START_REF] Toromanoff | Is deep reinforcement learning really superhuman on atari? leveling the playing field[END_REF]. Contributing to this is the vast amount of prior knowledge humans bring to the table before their first interaction with a new task, including an understanding of physics, semantics, and affordances [START_REF] Dubey | Investigating human priors for playing video games[END_REF].

The considerable quantity of data necessary to train agents is becoming more problematic as RL is applied to ever more challenging and complex tasks. Much research aims at tackling this issue, for example through transfer learning [START_REF] Rusu | Progressive neural networks[END_REF], meta learning, and hierarchical learning, where agents are encouraged to use what they learn in one environment to solve a new task more quickly. Other approaches attempt to use the structure of Markov Decision Processes (MDP) to accelerate learning without resorting to pretraining. [START_REF] Mahajan | Symmetry learning for function approximation in reinforcement learning[END_REF] and [START_REF] Biza | Online abstraction with mdp homomorphisms for deep learning[END_REF] learn simpler representations of MDPs that exhibit symmetrical structure, while van der Pol et al. (2020) show that environment invariances can be hard-coded into equivariant neural networks.

A fundamental challenge standing in the way of improved sample efficiency is exploration. We consider a situation where the exact transition function of a Markov Decision Process is unknown, but some knowledge of its local dynamics is available under the form of a prior expectation that given sequences of actions have identical results. This way of encoding prior knowledge is sufficiently flexible to describe many useful environment structures, particularly when actions correspond to agent movement. For example, in a gridworld (called RotationGrid hereafter) where the agent can move forward (↑) and rotate 90 • to the left (↶) or to the right (↷), the latter two actions are the inverse of each other, in that performing one undoes the effect of the other. During exploration, to encourage the visitation of not yet seen states, it is natural to simply ban sequences of actions that revert to previously visited states, following the reasoning of Tabu search [START_REF] Glover | Future paths for integer programming and links to artificial intelligence[END_REF]. We observe Figure 1: Illustration of EASEE on RotationGrid environment. The input is information about the dynamics of the environment known in advance under the form of action sequence equivalences (Λ denotes the empty action sequence). This is used to construct a representation of all the unique states that can be visited in 3 steps. The probabilities of sampling each action are then determined to explore as uniformly as possible. The probabilities of visiting each unique state are displayed on the right. further that ↷↷ and ↶↶ both lead to the same state (represented as state 4 in Figure 1). If actions were uniformly sampled, the chances of visiting this state would be much higher than any of the others. Based on these observations, we introduce a new method taking advantage of Equivalent Action SEquences for Exploration (EASEE), an overview of which can be found in Figure 1. EASEE looks ahead several steps and calculates action sampling probabilities to explore as uniformly as possible new states conditionally on the action sequence equivalences given to it. It constructs a partial MDP which corresponds to a local representation of the true MDP around the current state. We then formulate the problem of determining the best distribution over action sequences as a linearly constrained convex optimization problem. Solving this optimization problem is computationally inexpensive and can be done once and for all before learning begins, providing a principled and tractable exploration policy that takes into account environment structure. This policy can easily be injected into existing reinforcement learning algorithms as a substitute for ϵ-greedy exploration.

Our contribution is threefold. First, we formally introduce the notion of equivalent action sequences, a novel type of structure in Markov Decision Processes. Then, we show that priors on this type of structure can easily be exploited during offline exploration by solving a convex optimization problem. Finally, we provide experimental insights and show that incorporating EASEE into a DQN [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] improves agent performance in several environments with various structures.

Overview We assume that we have sets of equivalent action-sequences for the environment. Equivalent action sequences are sequences that lead to the same state. These sequences are used to build a DAG that models where the agent will end up after any sequence of actions of length d. Because some sequences are equivalent, several parent nodes may share a child node. A naive exploration scheme like ϵ-greedy would waste resources by over exploring such child nodes. Instead, we leverage this information using the DAG constructed above; our method executes an exploratory action that maximizes the entropy of the future visited states.

RELATED WORK

Improved Exploration The problem of ensuring that agents see sufficiently diverse states has received a lot of attention from the RL community. Many methods rely on intrinsic rewards [START_REF] Schmidhuber | A possibility for implementing curiosity and boredom in model-building neural controllers[END_REF][START_REF] Chentanez | Intrinsically motivated reinforcement learning[END_REF][START_REF] Şimşek | An intrinsic reward mechanism for efficient exploration[END_REF][START_REF] Lopes | Exploration in modelbased reinforcement learning by empirically estimating learning progress[END_REF][START_REF] Marc | Unifying count-based exploration and intrinsic motivation[END_REF][START_REF] Ostrovski | Count-based exploration with neural density models[END_REF][START_REF] Pathak | Curiosity-driven exploration by self-supervised prediction[END_REF] to entice agents to unseen or misunderstood areas. In the tabular setting, these take the form of count-based exploration bonuses which guide the agent toward poorly visited states (e.g. [START_REF] Strehl | An analysis of model-based interval estimation for markov decision processes[END_REF]). Scaling this method requires the use of function approximators [START_REF] Burda | Exploration by random network distillation[END_REF][START_REF] Puigdomènech Badia | Never give up: Learning directed exploration strategies[END_REF][START_REF] Flet-Berliac | Adversarially guided actor-critic[END_REF]. Unlike EASEE, these methods necessitate the computation of non-stationary and vanishing novelty estimates, which require careful tuning to balance learning stability and exploration incentives. Moreover, because these bonuses are learned, and do not allow for the use of prior structure knowledge, they constitute an orthogonal approach to ours. In Gupta et al. (2018) exploration strategies are learned from prior experience. Unlike EASEE this requires meta-training over a distribution of tasks.

Redundancies in Trajectories

The idea that different trajectories can overlap and induce redundancies in state visitation is used in [START_REF] Leurent | Monte-carlo graph search: the value of merging similar states[END_REF] and [START_REF] Czech | Monte-carlo graph search for alphazero[END_REF] in the case of Monte-Carlo tree search. However, they require a generative model, and propose a new Bellman operator to update node values according to newly uncovered transitions rather than modifying exploration. Closer to our work, [START_REF] Caselles-Dupré | On the sensory commutativity of action sequences for embodied agents[END_REF] study structure in action sequences, but restrict themselves to commutative properties. [START_REF] Grinsztajn | There is no turning back: A self-supervised approach for reversibility-aware reinforcement learning[END_REF] quantifies the probability of cycling back to a previously visited state, motivated by the analysis of reversible actions. Tabu search [START_REF] Glover | Future paths for integer programming and links to artificial intelligence[END_REF]) is a meta-heuristic which uses knowledge of the past to escape local optima. It is popular for combinatorial optimization [START_REF] Hertz | The tabu search metaheuristic: How we used it[END_REF]. Like our approach, it relies on a local structure: actions which are known to cancel out recent moves are deemed tabu, and are forbidden for a short period of time. This prevents cycling around already found solutions, and thus encourages exploration. In [START_REF] Abramson | Tabu search exploration for on-policy reinforcement learning[END_REF], tabu search is combined with reinforcement learning, using action priors. However, their method cannot make use of more complex action-sequence structure.

Maximum State-Visitation Entropy Our goal to explore as uniformly as possible every nearby state can be seen as a local version of the Maximum State-Visitation Entropy problem (MSVE) [START_REF] De Farias | The linear programming approach to approximate dynamic programming[END_REF][START_REF] Hazan | Provably efficient maximum entropy exploration[END_REF][START_REF] Lee | Efficient exploration via state marginal matching[END_REF][START_REF] Daniel Guo | Geometric entropic exploration[END_REF]. MSVE formulates exploration as a policy optimization problem whose solution maximizes the entropy of the distribution of visited states. Although some of these works [START_REF] Hazan | Provably efficient maximum entropy exploration[END_REF][START_REF] Lee | Efficient exploration via state marginal matching[END_REF][START_REF] Daniel Guo | Geometric entropic exploration[END_REF] can make use of priors about state similarities, they learn a global policy and cannot exploit structure in action sequences.

Action Space Structure The idea of exploiting structure in action spaces is not new. Large discrete action spaces may be embedded in continuous action spaces either by leveraging prior information [START_REF] Dulac-Arnold | Deep reinforcement learning in large discrete action spaces[END_REF] or learning representations [START_REF] Chandak | Learning action representations for reinforcement learning[END_REF]. [START_REF] Tavakoli | Action branching architectures for deep reinforcement learning[END_REF] manage high-dimensional action spaces by assuming a degree of independence between each dimension. [START_REF] Farquhar | Growing action spaces[END_REF] introduce a curriculum of progressively growing action spaces to accelerate learning. These methods aim to improve the generalization of policies to unseen actions in large action spaces rather than enhancing exploration. Leveraging previous trajectories to extract prior knowledge, [START_REF] Tennenholtz | The natural language of actions[END_REF] provide an understanding of actions through their context in demonstrations.

FORMALISM

EQUIVALENCE OVER ACTION SEQUENCES

We consider a Markov Decision Process (MDP) defined as a 5-tuple M = (S, A, T, R, γ), with S the set of states, A the action set, T the transition function, R the reward function and the discount factor γ. The set of actions is assumed to be finite |A| < ∞. We restrict ourselves to deterministic MDPs. A possible extension to MDPs with stochastic dynamics is discussed in Appendix A.6.

In the following, the notations are borrowed from formal language theory. Sequences of actions are analogous to strings over the set of symbols A (possible actions). The set of all possible sequences of actions is denoted A ⋆ = ∞ k=0 A k where A k is the set of all sequences of length k and A 0 contains as single element the empty sequence Λ. We use . for the concatenation operator, such that for

v 1 ∈ A h1 , v 2 ∈ A h2 , v 1 .v 2 ∈ A h1+h2 .
The transition function T : S × A → S gives the next state s ′ when action a is taken in state s: T (s, a) = s ′ . We recursively extend this operator to action sequences T : S × A ⋆ → S such that, ∀s ∈ S, ∀a ∈ A, ∀w ∈ A ⋆ :

T (s, Λ) = s T (s, w.a) = T (T (s, w), a)
Intuitively, this operator gives the new state of the MDP after a sequence of actions is performed from state s.

Definition 1 (Equivalent sequences). We say that two action sequences a 1 . . . a n and a

′ 1 . . . a ′ m ∈ A ⋆ are equivalent at state s ∈ S if T (s, a 1 . . . a n ) = T (s, a ′ 1 . . . a ′ m ) (1)
Two sequences of actions are equivalent over M if they are equivalent at state s for all s in S. This is written:

a 1 . . . a n ∼ M a ′ 1 . . . a ′ m (2)
This means that we consider two sequences of actions to be equivalent when following one or the other will always lead to the same state. When the considered MDP M is unambiguous, we simplify the notation by writing ∼ instead of ∼ M .

We argue that some priors about the environments can be easily encoded as a small set of action sequence equivalences. For example, we may know that going left then right is the same thing as going right then left, that rotating two times to the left is the same thing as rotating two times to the right, or that opening a door twice is the same thing as opening the door once. All these priors can be encoded as a set of equivalences: Definition 2 (Equivalence set). Given a MDP M and several equivalent sequence pairs

v 1 ∼ w 1 , v 2 ∼ w 2 , . . . , v n ∼ w n , we say that Ω = {{v 1 , w 1 }, {v 2 , w 2 }, . . . {v n , w n }} is an equivalence set over M. Formally, Ω is a set of pairs of elements of A ⋆ , such that Ω ⊂ (A ⋆ ) 2 . By abuse of notation, we write v ∼ w ∈ Ω if {v, w} ∈ Ω.
Intuitively, it is clear that action sequence equivalences can be combined to form new, longer equivalences. For example, knowing that going left then right is the same thing as going right then left, we can deduce that going two times left then two times right is the same thing as going two times right then two times left. In the same fashion, if opening a door twice produces the same effect as opening it once, opening three times the door does the same. We formalize these notions in what follows. First, we note that equivalent sequences can be concatenated. Proposition 1. If we have two pairs of equivalent sequences over M, i.e. w 1 , w 2 , w 3 ,

w 4 ∈ A ⋆ such that w 1 ∼ w 2 w 3 ∼ w 4
then the concatenation of the sequences are also equivalent sequences:

w 1 • w 3 ∼ w 2 • w 4
The proof is given in Appendix A.1. We are now going to define formally the fact that the equivalence of two sequences can be deduced from an equivalence set Ω. We first consider the previous example where an action a has the effect of opening a door, such that a.a ∼ a. We can then write a.a.a ∼ (a.a).a ∼ (a).a ∼ a.a ∼ a by applying two times the equivalence a.a ∼ a and rearranging the parentheses. More generally and intuitively, the equivalence of two action sequences v and w can be deduced from Ω, which we denote v ∼ Ω w, if v can be changed into w iteratively, chaining equivalences of Ω.

More formally, we write v ∼ 1 Ω w if v can be changed to w in one steps, meaning:

∃u 1 , u 2 , v 1 , w 1 ∈ A ⋆ such that    v = u 1 .v 1 .u 2 w = u 1 .w 1 .u 2 v 1 ∼ w 1 ∈ Ω (3)
For n ≥ 2, we say that v can be changed into w in n steps if there is a sequence v

1 , . . . , v n ∈ A ⋆ such that v ∼ 1 Ω v 1 ∼ 1 Ω • • • ∼ 1 Ω v n = w.
Finally, we say that v ∼ Ω w if there is n ∈ N such that v can be changed into w in n steps. The relation ∼ Ω is thus a formal way of extending equivalences from a fixed equivalence set Ω, and at first glance not connected with ∼, which deals with the equivalences of the MDP dynamics. We now show a connection between the two notions.

Theorem 1. Given an equivalence set Ω, ∼ Ω is an equivalence relationship. Furthermore, for

v, w ∈ A ⋆ , v ∼ Ω w ⇒ v ∼ w.
The proof is given in Appendix A.2. Given this relation between ∼ and ∼ Ω , we will simplify the notation in what follows by writing ∼ instead of ∼ Ω when the equivalence set considered is unambiguous. As ∼ Ω is an equivalence relationship, it provides a partition over action sequences: two action sequences in the same set lead to the same final state from any given state.

LOCAL-DYNAMICS GRAPH

We leverage the equivalences defined above to determine a model of the MDP up to a few timesteps. As traditionally done in Monte-Carlo Tree Search [START_REF] Coulom | Efficient selectivity and backup operators in monte-carlo tree search[END_REF], an MDP (S, A, T, R, γ) with deterministic dynamics can be locally unrolled to produce a tree, where a node of depth h represents a sequence of actions v ∈ A h , and the edges represent transitions between such sequences. The root of the tree corresponds to the empty action sequence Λ. Here we adopt the same formalism, except that equivalent sequences will point to the same node.

Given a tree T of depth d ∈ N corresponding to a partial unrolling of sequences in A ⋆ , and an equivalence set Ω, we call local-dynamics graph of depth d under equivalence Ω the graph G = (V, E) corresponding to the tree T where nodes are quotiented with the equivalence relation ∼ Ω . Intuitively, it means that nodes corresponding to equivalent action sequences are merged. In this case, the resulting graph is not necessarily a tree. In the following, unless the distinction is necessary, we identify action sequences with their equivalence classes.

The graph G gives rise to a new, smaller MDP resulting from M: the state space V is the set of action sequences smaller than d quotiented by the equivalence relation ∼ Ω , the action space A is untouched. Given a node n corresponding to a sequence w ∈ A ⋆ , and an action a ∈ A, T (n, a) is the node representing the sequence w.a ∈ A ⋆ . Nodes representing sequences of length exactly d are final states. The initial state v 0 is the empty sequence Λ. This MDP represents the local dynamics induced by ∼ Ω from a given root state. We detail in the next section how to construct such graphs in practice, and how to use these sub-MDPs for a better exploration.

EQUIVALENT ACTION SEQUENCES FOR EXPLORATION (EASEE)

4.1 FROM EQUIVALENT ACTIONS TO LOCAL-DYNAMICS GRAPH Producing the local-dynamics graph involves considering all possible action sequences and merging those that are equivalent. Figure 2 illustrates the construction of a local-dynamics graph, given A = {a 1 , a 2 } and Ω = {a 1 a 1 ∼ Λ, a 2 a 1 ∼ a 1 a 2 }. Starting from the root node 0 (first step), we iteratively expand the graph by unrolling the nodes at the edges of the graph. Steps 2 and 3 create nodes 1 and 2 corresponding to action sequences a 1 and a 2 respectively. In a tree, the expansion of a node corresponding to a sequence w ∈ A h with the action a ∈ A always leads to the creation of a new leaf that results from the sequence of actions w.a ∈ A h+1 . However, in a local-dynamics graph the node representing w.a might already be present, in which case we add an edge from w without creating a new node. As a final construction step, we prune edges which go backward in the local-dynamics graph, like (1, 0) in Fig. 2, such that the resulting graph is a DAG. This is motivated by the fact that we are interested in finding a good exploration policy: an action which takes us back to a previously visited state should be ignored.

From a practical point of view, the graph construction algorithm takes as input the action set A, the sequence equivalence set Ω, and the desired depth d, and outputs a DAG. Informally, it starts from a graph G = (V, E) reduced to a root state {0} and iteratively expands G until a distance d to the root is reached. We store in each node every action sequence which allows to reach it from any parent nodes. When expanding a node n with an action a ∈ A, we check every sequence w stored in n if w.a appears in Ω, and if a node corresponding to an equivalent sequence of w.a is already in V . If it is the case, we simply add an edge from n to this node, otherwise we create a new node representing w.a. We provide a more detailed implementation of this algorithm in Appendix A. Despite this exponential theoretical complexity, the goal is to use this algorithm locally, thus for small depths. In practice we found that local-dynamics graphs could be computed within a few seconds on a standard laptop.

Ω = {a 1 a 1 ∼ Λ, a 2 a 1 ∼ a 1 a 2 }

FROM LOCAL-DYNAMICS GRAPH TO LOCAL EXPLORATION POLICY

Once the local-dynamics graph (V, E) has been constructed, our goal is to find a good local exploration policy in the resulting MDP as defined in Section 3.2. We recall that its set of states is V , and its actions dynamics are given by the edges E. Ideally, we would want to find a policy π such that all nodes in the local-dynamics graph are visited equally often.

Given a policy π, a state v ∈ V and an action a ∈ A, we denote p π,t (v) and p π,t (v, a) the t-steps state distribution and state-action distribution respectively. Formally, p π,t (v) = P π (v t = v) and p π,t (v, a) = P π (v t = v, a t = a).

Ideally we would like each t-step state distribution to be uniform. However, depending on the exact local-dynamics graph this may or may not be possible (see Figure 1 for an example where obtaining a uniform distribution is impossible). Instead, following the principle of maximum entropy [START_REF] Jaynes | Information theory and statistical mechanics[END_REF], we frame the objective of balancing the state distribution at step t as maximizing H([p π,t (v 0 ), p π,t (v 1 ), . . . , p π,t (v |V |-1 )] = H(p π,t ), where H is the Shannon entropy. For a local-dynamics graph of depth d ∈ N, we define our global objective as maximizing

J(π) = J(p π,1 , . . . , p π,d ) = 1 d d t=1 H(p π,t ).
Other global objectives are possible, for example optimizing entropy over only the final states, or some other weighted mixture. In practice, over simple experiments, we observed that changes in the entropy mixture hardly induced any variation in the computed policies and agent behavior.

Informally, our objective can be understood as maximizing state diversity locally, for every timestep smaller than d. For environments where additional priors about state interests are available, one could adapt the quantity J to compute the entropy on a subset of the most interesting states, therefore biasing exploration toward promising areas.

We consider K, the set of joint distributions (p 0 , p 1 , . . . p d ) which verifies the following properties:

• ∀t ≤ d, p t (v, a) ≥ 0 • ∀v ∈ V, a∈A p 0 (v, a) = p 0 (v) = 1 v0 (v) • ∀t < d, a∈A p t+1 (v, a) = v ′ ∈V,a∈A p t (v ′ , a)P(v | v ′ , a)
We denote D(A) the set of distributions over A. From any (p 0 , p 1 , . . . , p d ) ∈ K, it is possible to find a time-dependent policy π : V × {0, . . . , d} → D(A) such that p 0 = p π,0 , p 1 = p π,1 , . . . , p d = p π,d , and for any policy π we have (p π,0 , p π,1 , . . . , p π,t ) ∈ K (Puterman, 2014).

As the entropy H is concave, the function J is a concave function over K. Moreover, the constraints defining K are linear. Therefore, max (p1,...,p d )∈K J(p 1 , . . . , p n ) (4) can be solved efficiently using any convex solver. In our implementation, we use CVXPY [START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF][START_REF] Agrawal | A rewriting system for convex optimization problems[END_REF]. Once (p ⋆ 1 , . . . , p ⋆ d ) = arg max K J is computed, we can immediately calculate a time-dependent policy π ⋆ from such a distribution (Puterman, 2014) with:

π ⋆ t (v, a) = p ⋆ t (v, a) p ⋆ t (v) (5)
As the local-dynamics graph (V, E) is a DAG, the set of nodes V 0 , V 1 , . . . , V d which can be reached respectively at timesteps t = 0, t = 1, . . . , t = d are disjoint. Therefore any time-dependent policy defined on V can be framed as a stationary policy. Considering for example π ⋆ , we can write

π ⋆ (v, •) = π ⋆ 0 (v, •) if v ∈ V 0 , π ⋆ (v, •) = π ⋆ 1 (v, •) if v ∈ V 1 , . . . , and π ⋆ (v, •) = π ⋆ d (v, •) if v ∈ V d .

FROM LOCAL EXPLORATION TO GLOBAL POLICY

The optimal π ⋆ determined in the previous section can then be used to guide exploration. With an ϵ-greedy policy, each step has a probability ϵ of being an exploration step, where an action is sampled uniformly. Instead, we keep in memory the local-dynamics graph, and initialize the current state at v = Λ. Everytime an action a is performed, v is updated such that v ← v.a, and reinitialized to Λ after a sequence of length d. At each exploration step, instead of sampling a uniformly, EASEE samples a according to the distribution π ⋆ (v, •). Pseudocode for this process can be found in Appendix A.5.

RESULTS

For every experiments, additional details about environments and hyperparameters are given in Appendix B.

PURE EXPLORATION

To get a better understanding of EASEE, we consider two simple gridworld environments with different structures: CardinalGrid and RotationGrid. These environments are both 100 × 100 gridworlds, but with different action structures. In CardinalGrid, the agent can move one square in the four cardinal directions (→, ←, ↑, ↓), whereas in RotationGrid, the agent can move either forward one square (↑), or rotate 90 • on the spot to the left (↶) or to the right (↷). The agent starts in the middle of the grid and can explore for 100 timesteps, after which the environment is reset.

In CardinalGrid, we consider the 4 equivalence sets:

• {→←∼←→} (" → and ← commute ")

• {→←∼←→, ↑↓∼↓↑} ("all actions commute")

• {→←∼←→, ↑↓∼↓↑, →←∼ Λ} ("all actions commute and →←∼ Λ")

• {→←∼←→, ↑↓∼↓↑, →←∼ Λ, ↑↓∼ Λ} (" all actions commute and →←∼↑↓∼ Λ "), while in RotationGrid, we consider the three equivalence sets:

• {↷↶∼ Λ} • {↷↶∼ Λ, ↶↷∼ Λ} • {↷↶∼ Λ, ↶↷∼ Λ, ↷↷∼↶↶}.
Fig. 3 shows the benefits of exploiting the structure of the action space for exploration. Figures 3a,3b show the ratio of the number of unique states visited using EASEE over a standard uniform exploration policy. For both environments, a greater equivalence set leads to a more efficient exploration. In the environmentCardinalGrid for example, for a fixed depth of 6, adding the information that → and ← commute and that every actions commute allow to reach respectively 10% and 60% more states in 100 episodes. Furthermore, extra equivalences encoding that → is the inverse of ←, and ↑ the inverse of ↓ increase the number of new states encountered threefold. It can also be seen that deeper graphs provide better exploration, which is expected: using deeper graphs results in exploiting equivalence priors over longer action sequences.

Figures 3c,3d show the number of unique states visited with respect to the total number of episodes of exploration. We see that EASEE benefits exploration in all configurations considered: it allows the agent to visit more states within a single trajectory, and as well as across a thousand. It gives insight about the sample-efficiency gain which can be achieved using EASEE over a standard random policy. In the CardinalGrid setting, EASEE visits more unique states over 100 episodes than uniform exploration over 1000.

MINIGRID

The Minimalistic Gridworld Environment (MiniGrid) is a suite of environments that test diverse capabilities in RL agents [START_REF] Chevalier-Boisvert | Minimalistic gridworld environment for openai gym[END_REF]. We evaluated the influence of adding EASEE to Q-learning on the DoorKey task. The environment is a gridworld split into two rooms separated by a locked door. The agent must collect a key to get to the objective in the other room.The dynamics of the environment are those of RotationGrid with two extra actions: the agent may PICKUP the key when facing it and OPEN the door when carrying the key. The EASEE version of the Q-learning assumes the following action sequence equivalences:

↷↶ ∼ Λ ↶↷ ∼ Λ ↶↶ ∼↷↷ OPEN ∼ OPEN • OPEN PICKUP ∼ PICKUP • PICKUP
The reward over this training is presented in Figure 4a. Using a depth of 6, the EASEE augmented version outperforms classic Q-learning.

CATCHER

We test EASEE on a game of Catcher, where the agent must catch a ball falling vertically with a paddle that can move left and right. It receives a reward of +1 when the ball is caught and -1 when it is missed. The prior we incorporate into the exploration is that the actions commute i.e. ←→∼→←. For faster learning we restrict each episode to a single ball drop, with the agent starting in the middle of the environment.

We choose a depth of 30 for EASEE. This is also the length of a single episode. The mean reward over training is plotted in Figure 4b.

FREEWAY

We test our method on the Atari 2600 game Freeway [START_REF] Bellemare | The arcade learning environment: An evaluation platform for general agents[END_REF]. To illustrate EASEE's performance in a non-deterministic setting, we add stochasticity to the dynamics of the game using sticky actions [START_REF] Machado | Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents (extended abstract)[END_REF]. The agent has to cross a road with multiple lanes without getting hit by the cars, and receives a reward when it reaches safety on the other side. The action space is composed of 3 actions : moving forward of 1 lane (↑), moving backward of 1 lane (↓), and passing (-). As cars arrive randomly, it is not easy to find priors on action equivalences in this environment. Since passing and moving backwards can sometimes be useful to avoid cars we cannot forbid these actions. However, we have prior knowledge that performing these two actions does not lead to visiting new lanes. We restrict the use of these actions with Ω = {↓∼↓↓, -↓∼↓-}, which has the effect of removing every node which is reached by chaining two ↓ actions without moving forward, and compute the exploration policy on the remaining nodes. Results can be seen in Fig. 4c.

DISCUSSION

We assume that implementers of reinforcement learning agents can provide insights about the environment, despite not knowing its precise dynamics or optimal policy.

In this work, we argue that some of these insights can be efficiently represented using the notion of action-sequence equivalence, which we formalize. We propose a method to incorporate such priors in classic Q-learning algorithms and demonstrate empirically its ability to improve sample efficiency and performance. More precisely, our approach can be divided into two steps: first, the construction of a graph representing the local dynamics, and then the resolution of a convex optimization problem aiming to balance node visitation. We show that incorporating such prior knowledge can replace standard ϵ-greedy and improve at little cost RL algorithms, which traditionally start from a tabula rasa setting, learning everything from scratch.

We expect EASEE to be robust to slight errors in the action sequence equivalence set. It may be that the states at the end of two sequences are not exactly the same but very similar, or that an action-sequence equivalence is verified at all but a few states. In such cases, the exploration policy we determine is not optimal, but should be much closer to optimality than uniformly sampling actions.

We additionally experimented with EASEE on two other Atari games, where the improvements are less pronounced: the details are given in Appendix B.4, as well as possible explanations.

In an effort to help with reproducibility, we provide a light-weight implementation of the experiments discussed in sections 5.1 and 5.2. For Catcher and Atari environments, all details of the environments, as well as the preprocessing steps and the hyperparameters we chose are described in Appendix B.2 and Appendix B.3 respectively. Additionally, the pseudo-code for the proposed methods is given in Appendix A.3.

ETHICS STATEMENT

A direct application of this paper is to make reinforcement learning techniques in general, and deep Q networks in particular, more efficient at solving Markov decision processes by making use of prior knowledge about action sequence equivalences. On its own, we do not expect EASEE to have immediate societal risks. Moreover, approaches incorporating prior knowledge in reinforcement learning algorithms often increase sample efficiency at little cost, thereby leading to less expensive and more environmentally-friendly methods.

We organize the supplementary material as follows. In Appendix A we include the proofs of results from the main text, as well as additional details about the proposed algorithms, including pseudo-code. In Appendix B we detail our experimental procedure, including hyperparameters for all methods used.

A TECHNICAL ELEMENTS AND PROOFS

A.1 PROOF OF PROPOSITION 1 Proposition 1. If we have two pairs of equivalent sequences over M, i.e. w 1 , w 2 , w 3 , w

4 ∈ A ⋆ such that w 1 ∼ w 2 w 3 ∼ w 4
then the concatenation of the sequences are also equivalent sequences:

w 1 • w 3 ∼ w 2 • w 4
Proof. For any s ∈ S, we have T (s, w 1 ) = T (s, w 2 ) as w 1 ∼ w 2 . We apply the same property for w 3 and w 4 on the state T (s, w 1 ):

T (T (s, w 1 ), w 3 ) = T (T (s, w 2 ), w 4 ) T (s, w 1 .w 3 ) = T (s, w 2 .w 4 )
Therefore w 1 .w 3 ∼ w 2 .w 4 .

A.2 PROOF OF THEOREM 1

Theorem 1. Given an equivalence set Ω, ∼ Ω is an equivalence relationship. Furthermore, for v, w ∈ A ⋆ , v ∼ Ω w ⇒ v ∼ w.

∼ Ω is an equivalence relation Let u, v, w ∈ A ⋆ .

Proof.

• We immediately have

v ∼ 1 Ω v by choosing v 1 = Λ in equation 3, and therefore v ∼ Ω v, thus ∼ Ω is reflexive. • It is clear from its definition that ∼ 1 Ω is symmetric, as ∼ is symmetric. Then, we suppose that v ∼ Ω w. We have n ∈ N and v 1 , . . . , v n ∈ A ⋆ such that v ∼ 1 Ω v 1 ∼ 1 Ω • • • ∼ 1 Ω v n ∼ 1 Ω w, therefore w ∼ 1 Ω v n ∼ 1 Ω • • • ∼ 1 Ω v 1 ∼ 1 Ω v, thus w ∼ Ω v. Hence ∼ Ω is symmetric. • If u ∼ Ω v and v ∼ Ω w, we have n 1 , n 2 ∈ N, and u 1 , . . . , u n1 ∈ A ⋆ , v 1 , . . . , v n2 ∈ A ⋆ , such that u ∼ 1 Ω u 1 ∼ 1 Ω • • • ∼ 1 Ω u n1 ∼ 1 Ω v and v ∼ 1 Ω v 1 ∼ 1 Ω • • • ∼ 1 Ω v n2 ∼ 1 Ω w. It is then clear that u ∼ 1 Ω u 1 ∼ 1 Ω • • • ∼ 1 Ω u n1 ∼ 1 Ω v ∼ 1 Ω v 1 ∼ 1 Ω • • • ∼ 1 Ω v n2 ∼ 1 Ω w, and thus u ∼ Ω w. Therefore ∼ Ω is transitive.
The relation ∼ Ω is reflexive, symmetric and transitive. Therefore it is an equivalence relation.

∼ Ω implies ∼ Proof. Let v, w ∈ A ⋆ . From Proposition 1, we immediately get v ∼ 1 Ω w ⇒ v ∼ M w. Then we can prove by immediate induction that ∀n ∈ N, v 1 , . . . , v n ∈ A ⋆ , v ∼ 1 Ω v 1 ∼ 1 Ω • • • ∼ 1 Ω v n ∼ 1 Ω w ⇒ v ∼ w, from which we deduce v ∼ Ω w implies v ∼ w.
• Action Sequence equivalences: the difference with the deterministic case here is that given an action a ∈ A and a state s ∈ S, T (s, a) is not a state but a distribution over the set of states S. Therefore every equality considered in Section 3.1 has now to be understood not as an equality between two states but between two distributions. Other than this the formalism can be kept identical. Intuitively, two sequences of actions are equivalent if they lead to the same state distribution from any given state, i.e. if they produce the same effect everywhere.

• Local-Dynamics Graph: Here, the formalism can again be kept identical. A node in the local-dynamics graph will not represent a state anymore, but rather a distribution over S.

B.2 CATCHER

The paddle is 1 block wide. The environment is 60 blocks wide and 30 blocks high. The ball and the paddle both move at a rate of 1 block per timestep, so each episode lasts 30 timesteps.

We use the same architecture for the DQN with and without EASEE. Each observation is a 60 × 30 image. The feature extractor network is a CNN composed of 3 convolution layers with kernel size 3 followed by ReLU activation. In both cases, we update the online network every 4 timesteps, and the target network every 10 3 timesteps. We use a replay buffer of size 10 4 , and sample batches of size 32. We use the Adam optimizer with a learning rate of 10 -4 . Architecture and hyperparameters We use the same architecture and the exact same DQN parameters as in AppendixB.3. In all three environments, EASEE is used with a depth of 4.

Results

We can see the results on Fig. 7. We can see a slight gain for Boxing, and a marginal improvement for Carnival. This can come from various factors:

• When the number of action sequence equivalences considered is small compared to the number of actions, as it is the case for Carnival, the exploration policy computed with EASEE is very much like a uniform policy. It logically makes its performances converge toward those of a standard DQN.

• The action sequence equivalences considered here are only approximately true. In boxing, it is only approximately true that UPRIGHT ∼ UP.RIGHT for example. In Carnival, RIGHT and LEFT commute as long as the player is not at the edges of the screen, in which case RIGHT or LEFT could have no effect. In both cases, this induces a bias that may harm performance.

  3. Proposition 2. The complexity of this graph construction algorithm is upper bounded by O |A| 2d |Ω|d .

Figure 2 :

 2 Figure2: Example of iterative graph construction with Ω = {a 1 a 1 ∼ Λ, a 2 a 1 ∼ a 1 a 2 } and a maximum depth of 2. The 8 th construction step corresponds to the pruning of the edge (1, 0).

Figure 3 :

 3 Figure 3: (a, b): Ratio of the number of unique visited states during 100 episodes following EASEE over standard ϵ-greedy policy, for different equivalence sets and depths in the environments CardinalGrid and RotationGrid respectively. (c, d): Number of unique visited states according to the number of episodes for EASEE with a fixed depth of 4 compared to standard ϵ-greedy policy.

Figure 4 :

 4 Figure 4: Mean reward over training with 95% confidence intervals.

  on the DoorKey task. An illustration of the initial state is given in Fig. 5. The agent is represented by the red triangle. The yellow key is necessary to open the yellow door. The two room are respectively 12 × 17 and 4 × 17 grids. The agent has 3249 timesteps to reach the goal and receive a reward of 1 before the environment is reset.

Figure 5 :

 5 Figure 5: Example of initial state of DoorKey environment.

Figure 7 :

 7 Figure 7: Performances of DQN and DQN + EASEE on the Atari 2600 games Boxing, Carnival. A 95% confidence interval over 10 random seeds is shown.

  and a maximum depth of 2. The 8 th construction step corresponds to the pruning of the edge (1, 0).The proof is given in Appendix A.4. It is to be noted that this upper bound is in general far larger than the actual number of operations. Indeed, it supposes that the number of nodes in the graph is |A| d , although it can be much smaller thanks to the redundancies induced by Ω. A more precise formula is O |V ||A| d |Ω| , where |V | is the number of nodes in the final graph and depends on the structure of Ω.

•

  Local Exploration Policy: Solving directly the objective given in equation 4 would lead to maximize the diversity among state distributions encountered. As is, it would not necessarily lead to a better diversity among states, as two different distributions can have an almost similar support. Therefore, adapting EASEE to a stochastic setting would require encoding additional priors about the distributions represented by the nodes of the local-dynamics graph, which we leave for future work. If we suppose that the distributions encountered have disjoint supports, and that their entropy is the same, EASEE can be applied without modification.

Algorithm 1: Graph Construction Input Action set A; Input Equivalence set Ω; Input Maximum tree depth d; Initialize the graph G = (V, E) with V = {0} and V = ∅ ; Initialize the set of states to expand S = {0} ; Initialize the current tree depth l = 0; Initialize a dictionary E which stores partial sequences of Ω for each state of

A.3 GRAPH CONSTRUCTION ALGORITHM

We present in Algorithm 1 an overview of the graph construction algorithm. It takes as input the action set A, the sequence equivalence set Ω, and the desired depth d, and outputs a DAG. Informally, it starts from a graph G = (V, E) reduced to a root state {0} and iteratively expands G until a distance L to the root is reached. For a node n ∈ V we store in E(v) sequences which reach v, and are prefixes of sequences of Ω. When expanding a state v ∈ V using an action a ∈ A (Line. 12), we look at every partial sequence s ∈ E(v). If s.a is in Ω, it means that we have found a redundant sequence. If the equivalent sequence has already been computed, it means that a node u representing T (v, s.a) has previously been added in G. Otherwise, we add a new node u. In both case, we update the equivalences E(u) to account for the new ways of reaching u (Line. 18).

A.4 GRAPH CONSTRUCTION COMPLEXITY

As shown in Section 3.2, constructing the graph necessitates three intricate loops: The first one goes over every internal node n ∈ V , the second one loops over the set of actions A, and the last one loops over every partial sequence which allows to reach v from a parent node. Inside these three loops, one has to compare the partial sequence with every sequence of Ω. As sequence length in Ω can be bounded by d, the complexity cost inside the tree loops is bounded by O(|Ω|d). The total complexity is therefore lower than In this section we discuss the possibility of extending EASEE to the case of MDPs with stochastic transitions. EASEE relies on three components: the formalization of action sequence equivalences (Def. 1), the construction of a local-dynamics graph (Section 3.2), and the construction of a local exploration policy by solving a convex problem (Section 4.2). We now detail for each step the necessary changes to adapt EASEE to M = (S, A, T, R, γ), a MDP with stochastic dynamics.

We train for 3.10 5 timesteps. The exploration parameter ϵ is linearly annealed from 1 down to 0.05 over 20% of the training period. Other DQN hyperparameters were defaults in [START_REF] Raffin | Stable baselines3[END_REF]. Environment In Freeway, the agent has to cross a road with multiple lanes without getting hit by the cars. It only receives a reward when it safely reaches the other side of the road. An illustration is given in Fig. 6. The agent is represented by the yellow chicken.

B.3 FREEWAY

To add stochasticity to the dynamic of the environment, we use sticky actions as proposed in [START_REF] Machado | Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents (extended abstract)[END_REF], with a stickiness parameter of 0.25. More precisely, the environment has 0.25 probability of executing the previous action again instead of the current desired action. The frame is recast as a 84 × 84 × 3 image, and the number of frames to skip between each observation is set to 4. The reward is scaled to [-1, 1]. An observation corresponds to 4 stacked game frames.

Architecture and hyperparameters We use the same architecture for the DQN with and without EASEE. Input images first go through a convolutional neural network, with the same architecture as in [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]. We update the online network every 4 timesteps, and the target network every 10 3 timesteps. We use a replay buffer of size 10 5 , and sample batches of size 32. We use the Adam optimizer with a learning rate of 10 -4 .

We train for 10 7 timesteps. The exploration parameter ϵ is linearly annealed from 1 down to 0.01 over 10% of the training period, which are the default in [START_REF] Raffin | Rl baselines zoo[END_REF] for Atari games. Other DQN hyperparameters were defaults in [START_REF] Raffin | Stable baselines3[END_REF].

B.4 ADDITIONAL EXPERIMENTS

Environments We experimented EASEE on two other Atari environments, where the action sequence structures are less straight-forward. The three environments are preprocessed as explained in AppendixB.3.

• Boxing: This game shows a top-down view of two boxers. The player can move in all four directions, and punch his opponent (pressing the "FIRE" button).


Shun Yao

Robin Brahmi

Anaïs Bouschon

Jing Chen

Sami Halila

Supramolecular carbohydrate-based hydrogels from oxidative hydroxylation of amphiphilic -C-glycosylbarbiturates and -glucosidase-induced hydrogelation

HAL is

Introduction

Glycoamphiphiles are basically defined as a hydrophilic carbohydrate moiety covalently bound via a glycosidic linkage to a hydrophobic building block. In Nature, such amphiphilic species exist as glycolipids (glycosphingolipids, glycosylphosphatidylinositols, lipopolysaccharides, etc.) that are essential components of cell membranes and mediate diverse biological and pathological processes such as cell growth, fertility, immunity, metastasis, and viral and microbial invasion. 1,2 On the other hand, because of their self-assembling properties, their renewable raw materials, low toxicity and high biodegradability, glycoamphiphiles found many applications in a huge variety of fields, from agriculture to biomedical applications. 3,4 For instance, alkylpolyglucoside is used as a foaming agent, emulsifier, pharmaceutical granulating agent and cosmetic surfactant. [START_REF] Gutsche | Handbook of Detergents, Part F: Production[END_REF] Besides, glycoamphiphiles display advanced functions such as liquid crystals properties or self-assembly properties into various nanostructures ranging from micelles, vesicles and fibers in solution. [START_REF] Brito | [END_REF][7][8] A milestone was reached when we demonstrated that amphiphilic carbohydrate-based block copolymers could self-organized into periodic thin-films nanostructures of sub-10 nm resolution. [9][10][11] In addition, glycoamphiphiles or glyco-hydrogelators are known to form supramolecular hydrogels resulting from the three-dimensional network of interpretated self-assembled nanofibers. [12][13][14] Supramolecular hydrogels have gained tremendous attention in the development of soft materials with applications as drug releasing matrices, tissue engineering, removal of pollutants, sensors or template for nanostructured materials, etc… . [15][16][17] Glyco-hydrogelators are mainly achieved through a multi-step synthesis including protection/functionalization/deprotection reactions in toxic and unsustainable solvents that is non economical and ecofriendly strategy and remains a time-consuming and labor-intensive process. [12][13][14] Very recently, we tackled these problematics by reporting an efficient and versatile green synthetic access to glyco-hydrogelators through the Knoevenagel condensation in water of N,N'-substituted barbituric acids with protecting group-free carbohydrates. 18 The resulting sodium salt of N-monosubstituted -C-glycosylbarbiturates led to entangled nanostructured hydrogels either in strong acidic conditions or either by adding calcium ions at neutral pH (Fig. 1). Because of hydrogelation conditions that are inconsistent with biological applications, we investigated, once again, a sustainable chemical way to perform hydrogelation of -Cglycosylbarbiturates at neutral pH and without adding divalent metal ions. Starting from the observation that the exclusive presence of the keto tautomer form (KH) of barbiturates, that could be in equilibrium with the enol tautomer form (EH), is mandatory to promote hydrogelation, we decided, in this paper, to investigate the post-functionalization at the C-5 position of the barbituric moiety of the glycoamphiphiles, thus definitively blocking the KH form (Fig. 1). A survey of the literature indicated that reactions of i) bromination, 19 ii) alkylation, 20,21 iii) hydroxy methylation, 22 iv) malonyl peroxides-mediated oxidation, 23 and v) oxidative hydroxylation could be envisioned. Keeping in mind that green chemistry approaches should be favored, we took the route of oxidative hydroxylation 24 that occurs in water.

Therefore, herein, we describe the selective oxidative hydroxylation of a series of amphiphilic N-monosubstituted -Cglycosylbarbiturates as well as an investigation of their hydrogelation properties in pure water.

ARTICLE

Results and discussion

Our first attempts of selective oxidative hydroxylation at C-5 position of the barbiturate ring were done with model substrates, such as -C-glucosylbarbiturate 1 and N,N'-dimethyl -C-glucosylbarbiturate 1', that were easily synthesized and isolated from the aqueous reaction by precipitation into a mixture of EtOH/EtOAc (2:6; v/v) with 96 % and 98 % yields, respectively (Scheme 1). 25 A previous work from L'Oréal group reported that H2O2-mediated oxidative hydroxylation of 1' in water led in a one-pot procedure to -methylamide C-glucoside derivative 4' involving the release of NaOH in the mechanism pathway. 26 Scheme 1 H2O2-mediated oxidative hydroxylation of -C-glucosylbarbiturate 1 and 1'.

Moreover, a patent from the previous authors showed that the oxazolidine-2,4-dione intermediates 3 and 3' could be favoured by controlling the reaction time or by using oxone ® as oxidizing reagent during 16 h. 27 For the latter, by reducing the reaction time from 16 to 2 h, the C-5 hydroxyl derivative 2' have been obtained in 71% yield. And the compound 2 was not described. In this work, H2O2 was preferred because is a very mild oxidant, relatively cheap and produces only water as by-product, thus benefiting both the economic and green chemistry viewpoints. Since the basic conditions of H2O2-mediated oxidative hydroxylation led preferentially to rearrangements, we treated 1' with 2 equivalents of KH2PO4 1M in order to control the pH (between 7 and 8) during the reaction. In spite of these precautions, we rapidly obtained a mixture of oxazolidinone 2' and C-5 hydroxyl 3' derivatives according to mass spectrometry analysis (data not shown). However, the same experimental conditions applied to 1 led to 2 in a satisfactory yield of 66%. This result highlights that N-H groups kinetically disfavors the rearrangement pathway, thereby allowing control of the oxidative hydroxylation at the C-5 position of the barbituric acid.

As previously mentioned, we already demonstrated that MalB-3 and MalB-4 (Fig. 1), that are featured by a maltose disaccharide linked through a N-monosubstituted barbiturate to a C18 saturated aliphatic chain and C18 mono-unsaturated C18:1 (cis-9) chain, respectively, were able to hierarchically self-assemble into supramolecular hydrogels in strong acidic conditions or by adding calcium ions. 18 We found, with others, [28][29][30] that the hydrophobic-hydrophilic balance is a determinant parameter for exhibiting hydrogelation and since oxidative hydroxylation brings an additional hydroxyl function, we decided to screen a library of synthetic amphiphilic -C-glycosylbarbiturates GlcB(OH) and MalB(OH) towards their hydrogelation abilities (Scheme 2). Briefly, the synthesis of GlcB and MalB were performed through Knoevenagel condensation in water by reacting chemoselectively N,N'-alkyl barbituric acid derivatives into the reducing-end anomeric position of unprotected Glc and Mal, as already reported. 18 Next, H2O2-mediated oxidative hydroxylation of GlcB and MalB in water led successfully to GlcB(OH) and MalB(OH), respectively, with moderate to good yields (54 -72%) as shown in Table 1. While thin-layer chromatography analyses indicated an almost complete reaction, a chromatographic separation step was required, lowering the yields in isolated products. All the structures were characterized and confirmed by high resolution mass spectrometry, NMR, FTIR and UV-visible spectroscopy (See ESI † and Fig. S1-S8). The formation of 5,5-disubstituted barbiturates induces significant upfield shift of the H-anomeric involved in -Cglycoside formation that appears as a clean doublet at 4.5 ppm in GlcB and MalB (Fig. S9 † ). 18 The keto tautomer form for the series

O HO HO OH OH N N O NaO O R R Glucose N N O O O R R HO H2O2, H2O -NaOH Glucose N H O OH 2 2' 1 1' 4'
Glucose HO

3 3' N O O O R = H R = Me R
of GlcB(OH) and MalB(OH) was also confirmed by UV-visible spectroscopy with an expected max around 230 nm and nothing at 262 nm, such as for GlcB-1, which is the signing of the enol tautomer form (Fig. 2). 18 To investigate the self-assembling properties of GlcB(OH) and MalB(OH), tube-inversion experiments were realized, that involved cooling to induce gelation (self-assembly) after dissolution into aqueous solution via heating at 80°C (Table 2). In contrast to GlcB samples that didn't dissolved into water even after heating, GlcB(OH) samples dissolved upon heating and more importantly slightly opaque hydrogels were formed after cooling-down at room temperature (rt) with a minimal gelation weight concentration (MGC) of 2 wt% for GlcB(OH)-2, and 3 wt% for GlcB(OH)-3 and GlcB(OH)-4. GlcB(OH)-1 dissolved into hot water and upon cooling formed sedimented large spherical aggregates and could not be considered as hydrogel. For GlcB(OH)-4, the formation of the hydrogel occurred after 3 days at rest, highlighting particularly low self-assembly kinetics and aging helped for improving packing over time. In summary, these results clearly indicated that the additional hydroxyl function changed dramatically the solubility parameter in pure water and induced hierarchical self-assembly into supramolecular hydrogels. For MalB(OH) samples, opaque and transparent hydrogels were only observed for MalB(OH)-3 at 4 wt% and MalB(OH)-4 at 5 wt%, respectively. The obtention of a stable transparent gel for MalB(OH)-4 compared to MalB(OH)-3 reveals the formation of thinner aggregates that are shorter than the visible wavelength region. Decreasing the aliphatic chain from C18 (MalB(OH)-3) to C16 (MalB(OH)-2) and C14 (MalB(OH)-1) resulted in a full solubility in water. While MalB(OH)-3 is stable at room temperature for several months, MalB(OH)-4 becomes liquid after 1 day at rt. The hydrogels were all thermoreversibles, as confirmed by alternating heating (solution) and cooling (gel) cycles, that is one of characteristics of supramolecular hydrogels owing to the nature of weak interconnected bonds. Gel-to-sol transition (Tgel) of the hydrogel at MGC was investigated by differential scanning calorimetry (DSC) which showed an endothermic peak at 64, 61, 42, 38°C for GlcB(OH)-2 to -4, respectively and 38°C for both MalB(OH)-3 and -4, supporting the disruption of the gel phase (Table 2).

ARTICLE

To get a deeper insight into the morphology of self-assembled glyco-hydrogels, transmission electron microscopy observations were performed. As shown in Fig. 3, various well-structured nanoscale architectures entrapping solvent water molecules in the 3-D network were observed and featured by carbohydrates exposed at the periphery. GlcB(OH)-2 self-assembled into ribbons constituted of fine fibrils (white arrow in Fig. 3A) across their width, while GlcB(OH)-3 revealed uneven giant multilamellar vesicles ranging from 0,3 to 1 m in diameter (Fig. 3B) as already observed for phospholipids. 31 For the latter, the matrix of the hydrogel seems to be formed by the closely packed structures of multilamellar vesicles as already reported in our recent publication. 18 For GlcB(OH)-4, TEM showed micrometers long semi-rigid helical nanofibers with a uniform lateral diameter (17 ± 1 nm) close to the lateral arrangement of four GlcB(OH)-4 molecules (Fig. 3C). MalB(OH)-3 displayed a plate-like morphology of about hundred nanometers in diameter, (Fig. 3D) 32 while MalB(OH)-4 self-assembled into 5 ± 0,5 nm-wide and several micrometer-long tortuous wormlike cylinders consisting with the model of bilayer structures of interdigitated oleyl (C18:1(cis-9)) chains (Fig. 3E), that physically entangled into a network that forms the hydrogel matrix. These small wormlike aggregate dimensions confirmed our previous assumption related to the transparency of the hydrogel. If the control of noncovalent interactions with DNA, peptides or phospholipids is well mastered, the known difficulties in controlling the precise structures of carbohydrate-based amphiphiles are closely related the stereochemistry of carbohydrates that greatly impact hydroxyl-based hydrogen bonding (carbohydrate-carbohydrate interactions or CCIs) along with steric effects. However, it is wellestablished that the self-assembly process is roughly governed by geometrical constraints impacting the surface curvature. of individual uncharged amphiphile molecules (packing parameter), 33 in addition to the attractive hydrophobic interactions between the lipophilic segments and repulsive steric interactions between the hydrophilic headgroups. 34,35 The different morphologies observed between Glc-B(OH)-3 vs. MalB(OH)-3 and GlcB(OH)-4 vs. MalB(OH)-4 are undoubtedly attributed to molecular geometric considerations and subsequent molecular packing, since Mal disaccharide possesses one additional bulky glucose unit compared to Glc monosaccharide. Moreover, from dissipative particle dynamics simulations with amphiphilic carbohydrate-based metallacycles, G. Chen and coll. 36 revealed that a disaccharide displays stronger CCIs than a monosaccharide by four times, driving the self-assembly along the CCIs direction. Therefore, beside the simple molecular geometric considerations, the nature of the carbohydrate headgroup plays also an important role in driving hierarchical self-assembly towards diversiform glyco-nanostructures. Likewise, the morphologies drastically changed from stearyl-containing glycoamphiphiles (GlcB(OH)-3 or MalB(OH)-3) to the counterpart monounsaturated oleyl-containing glycoamphiphiles (GlcB(OH)-4 or MalB(OH)-4) only differing in the absence or presence of a cis double bond in the middle of the alkyl chain moiety. Once again, the intermolecular packing plays a significant role because one can expect that oleyl chains, due to the higher chain mobility (Tm = 13°C) than stearyl chain (Tm = 69°C) at RT, will form a tighter intermolecular packing promoting low mean-curvature aggregates such as fiber-or worm-like nanostructures. 37 After analyzing individually all the morphologies from the series of GlcB(OH)-2 to 4, that varies slightly of two C-C bonds in the hydrophobic chain length, it seems difficult to get direct relationships with the molecular packing parameters. Above all, our results emphasize that HLB still plays a major role for driving their self-assembly behavior regardless of the discrete molecular architectures.

Carbohydrate-based hydrogelators have found many applications in the biomedical field. For instance, supramolecular glycohydrogels exhibited inhibition towards biofilm formation and bacterial growth of P. aeruginosa, 38 served as biomimetic scaffolds for cell growth and adhesion, 39,40 promoted wound healing, 41 and selectively inhibited cancer cells. 42 For the latter, Pashkuleva et al. [43][44][45] originally proposed a simple N-Fmoc glucoamine-6-phosphate derivative as a precursor of self-assembly and hydrogelation triggered by the overexpressed alkaline phosphatase-catalyzed dephosphorylation and produced by osteosarcoma cell line. This innovative supramolecular-based therapeutic axis stems from pioneering works of Bing Xu's group with peptide-based hydrogelator precursors and it is commonly called "Enzyme-Instructed Self-Assembly" or EISA. 46,47 By carefully studying the gelation tests of our amphiphilic -C-glycosylbarbiturates, we identified a potential precursor of hydrogelation that could be induced by the action of -glucosidases. Indeed, while we demonstrated that MalB(OH)-2 is fully soluble in water, the monosaccharide analogue GlcB(OH)-2 formed an opaque hydrogel with a MGC of 2 wt%. So, we evaluated MalB(OH)-2 as a precursor to turn into a hydrogelator, GlcB(OH)-2, upon Aspergillus niger amyloglucosidase-catalyzed nonreducing D-glucosyl residue hydrolysis. We started our study by using 3 wt% of MalB(OH)-2 that should result in 2,3 wt% of GlcB(OH)-2 if the hydrolysis is total, therefore slightly above MGC. After overnight of enzymatic hydrolysis at 45°C, we were pleased to see that the clear solution turned into an opaque gel. TEM images revealed uniform self-assembled nanofibers (Fig. 4) that differ from the expected ribbons for GlcB(OH)-2 (Fig. 3A). In addition, if the weight concentration is divided by 5 to reach 0.6 wt% of MalB(OH)-2, the -glucosidase-induced gelation of the resulting GlcB(OH)-2 occurred and at theoretically 0.46 wt%, which is much lower than MGC. These results pointed out that nanostructure morphologies resulting from self-assembled hydrogelators are clearly dictated by kinetics of supramolecular polymerization (production rate of the hydrogelator) and not only by the tridimensional structure of hydrogelators. 48 This work reports for the first time the use of -glucosidase for sol-to-gel phase transition while it was only used for gel-to-sol phase transition. 49 The in-vivo preparation of self-assembled hydrogelators for selective inhibition of cancer cells by taking advantage of overexpressed endogenous enzymes could be envisioned for treating pancreatic cancer, a devastating gastrointestinal cancer, since a recent study demonstrated that an elevated level of lysosomal acid -glucosidase occurred in response to gemcitabine treatment, an anticancer nucleoside. 50

Conclusions

In conclusion, following our observation from a previous study that sodium salt of glycosylbarbiturate hydrogelators were effective if the keto form of barbiturate is predominant, we successfully developed a green chemistry derivatization to stabilize the keto form through H2O2-mediated oxidative hydroxylation of the barbiturate ring in water. Thus, we synthesized a series of hydroxylated glycosylbarbiturates and found hydrogelators enabling the supramolecular gelation of water at neutral pH and without the need of strong acid or divalent cations such as for sodium salt of glycosylbarbiturates. Glucose and maltose-based hydrogelators have been discovered resulting in various self-assembled glyco-nanostructures due to their different intermolecular interaction and packing parameters. And in the context of EISA, we showed that -glucosidase could trigger the formation of supramolecular nanofibers to result in hydrogels and even at a concentration below the minimal concentration of gelation. Indeed, a maltose-based precursor was hydrolyzed into a glucose-based hydrogelator with the use of Aspergillus niger amyloglucosidase. Since EISA found relevant applications in cancer therapy by selectively generating in situ self-assembled nanostructures or at the periphery of cancer cells, it can be anticipated that these C-glycosylbarbiturates have promising applications in medicine.

Fig. 1

 1 Fig. 1 Structures and conditions of hydrogelation of sodium salt of amphiphilic -C-glycosylbarbiturates

Scheme 2

 2 Scheme 2 H2O2-mediated oxidative hydroxylation of sodium salt of amphiphilic -C-glycosylbarbiturate GlcB and MalB towards neutral amphiphilic hydroxylated -Cglycosylbarbiturate GlcB(OH) and MalB(OH).

Fig. 2

 2 Fig. 2 UV spectrum of GlcB(OH) and MalB(OH) in water showing max at 230 nm corresponding to keto tautomer forms and of GclB-1 showing max at 262 nm corresponding to a mixture of enol and keto tautomer forms.

GlcB

  gel at room temperature, the numbers close to G indicate the MGCs; P: precipitation; S: soluble; O: opaque; T: transparent. Tgel represents gel-to-sol transition temperature of hydrogels at MGCs obtained from DSC.

Fig. 3

 3 Fig. 3 TEM image of nanostructures of hydrogels formed with GlcB(OH)-2 (A), GlcB(OH)-3 (B), GlcB(OH)-4 including a rectangle zoom (C), MalB(OH)-4 (D) and MalB(OH)-5 (E).

Fig. 4 A

 4 Fig. 4 A. niger amyloglucosidase-catalyzed glucosyl residue hydrolysis of MalB(OH)-2 at 3 and 0,6 wt% into GlcB(OH)-2 hydrogelator with hydrogel pictures and TEM image of fibrillar glyco-nanostructures.

Table 1

 1 Yields

	Glucose		R		Maltose	% yield a
		1	C14	1		58	72
	GlcB(OH)	2 3	C16 C18	2 3	MalB(OH)	64 56	54 67
		4	C18:1(cis-9)	4		63	63

of oxidative hydroxylation affording amphiphilic -C-glycosylbarbiturate GlcB(OH) and MalB(OH).

a Conversion in isolated product.

Table 2

 2 Pictures of the hydrogels and properties in water of

GlcB(OH) and MalB(OH).

R. Hamura, Y. Shirai, Y. Shimada, N. Saito, T. Taniai, T. Horiuchi, N. Takada, Y. Kanegae, T. Ikegami, T. Ohashi and K. Yanga, Cancer Sci., 2021, 112, 2335-2348.

Acknowledgements

We are grateful for financial support from CNRS, Université Grenoble Alpes, LabEx ARCANE and CBH-EUR-GS (ANR-17-EURE-0003), the Glyco@Alps program (ANR-15-IDEX-02), Institut Carnot PolyNat (No. 16-CARN-025-01), National Natural Science Foundation of China (51803227), S&T Innovation 2025 Major Special Program of Ningbo (2019B10063), CAS President's International Fellowship for Visiting Scientists (2019VBA0016). All authors acknowledge J. L. Putaux for his help in TEM images

ARTICLE

and NanoBio-ICMG Platform (UAR 2607, Grenoble) for granting access to the electron microscopy, NMR and mass spectrometry facilities. S.Y. thanks the China Scholarship Council (CSC) for his scholarship support (201908320428).