Quentin Gallouédec

Emmanuel Dellandréa

Cell-Free Latent Go-Explore

In this paper, we introduce Latent Go-Explore (LGE), a simple and general approach based on the Go-Explore paradigm for exploration in reinforcement learning (RL). Go-Explore was initially introduced with a strong domain knowledge constraint for partitioning the state space into cells. However, in most real-world scenarios, drawing domain knowledge from raw observations is complex and tedious. If the cell partitioning is not informative enough, Go-Explore can completely fail to explore the environment. We argue that the Go-Explore approach can be generalized to any environment without domain knowledge and without cells by exploiting a learned latent representation. Thus, we show that LGE can be flexibly combined with any strategy for learning a latent representation. Our results indicate that LGE, although simpler than Go-Explore, is more robust and outperforms state-of-the-art algorithms in terms of pure exploration on multiple hard-exploration environments including Montezuma's Revenge. The LGE implementation is available as open-source at https://github.com/qgallouedec/ lge.

Introduction

RL algorithms aim to learn a policy by maximizing a reward signal. In some cases, the rewards from the environment are sufficiently informative for the agent to learn a complex policy, and therefore achieve impressive results, including world level in Go [START_REF] Silver | Mastering the Game of Go with Deep Neural Networks and Tree Search[END_REF], StarCraft [START_REF] Vinyals | Grandmaster level in StarCraft II using multi-agent reinforcement learning[END_REF], or learning sophisticated robotic tasks [START_REF] Lee | Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning[END_REF]. However, many real-world environments provide extremely sparse (Bellemare et al., 2016), deceptive [START_REF] Lehman | Abandoning Objectives: Evolution Through the Search for Novelty Alone[END_REF] rewards, or none at all. In such environments, random exploration, on which many current RL approaches rely, may not be sufficient to collect data that is diverse and informative enough for the agent to learn anything. In these cases, the agent must adopt an efficient exploration strategy to reach high reward areas, which may require a significant amount of interactions.

Recently, [START_REF] Ecoffet | First return, then explore[END_REF] introduced a new paradigm in which a goal-conditioned agent is trained to reach states it has already encountered, and then explore from there. The agent thus iteratively pushes back the frontier of its knowledge of the environment. We call this family of algorithms return-then-explore. [START_REF] Ecoffet | First return, then explore[END_REF] provide Go-Explore, an algorithm of this new family, that outperforms by several orders of magnitude the state-of-the-art scores on the game Montezuma's Revenge, known as a hardexploration problem. Go-Explore relies on a grouping of observations into cells. These cells are used both to select target observations at the frontier of yet undiscovered states and to build a subgoal trajectory for the agent to follow to reach the final goal cell. As [START_REF] Ecoffet | First return, then explore[END_REF] initially spotted, the cell design is not obvious. It requires detailed knowledge of the observation space, the dynamics of the environment, and the subsequent task. If any important information about the dynamics of the environment is missing from the cell representation, the agent may fail to explore at all. For example, in Montezuma's Revenge, possession of a key is a crucial piece of information that when included in the cell representation increases exploration by several orders of magnitude. We also demonstrate in Appendix B that the cell design has a major influence on the results.

In this paper, we present Latent Go-Explore (LGE), a new algorithm derived from Go-Explore which operates without cells. This new algorithm meets the definition of a returnthen-explore family of algorithms since the agent samples a final goal state at the frontier of the achieved states, returns to it, and then explores further from it. Our main contribution consists of three major improvements.

• A latent representation is learned simultaneously with the exploration of the agent to provide the most up-todate and informative representation possible.

• Sampling of the final goal is based on a non-parametric density model in latent space. This leverages the learned latent representation for sampling the states of interest to be reached.

• The subgoal path pursued by the agent is reduced using a characteristic latent distance.

These three modifications, detailed in Section 3, allow us to generalize the Go-Explore approach to any continuous highdimensional environment. It also enables the automation of the encoding of observations into an informative latent representation, eliminating the need for manual cell design. The full LGE exploration workflow is presented in Figure 1.

To evaluate LGE, we conducted experiments in the context of reward-free exploration in various hard-exploration environments including a maze, a robotic arm interacting with an object, and two Atari games known for their high exploration difficulty: Montezuma's Revenge and Pitfall.

LGE can use various types of latent representation learning methods. In this study, we demonstrate the use of three such methods, including inverse dynamics, forward dynamics, and auto-encoding mechanism. We show in Section 4.4 that for the environments studied, LGE outperforms all state-ofthe-art algorithms studied in this paper, and in particular Go-Explore for the exploration task.

Preliminaries and Related Work

Preliminaries

Markow Decision Process This paper uses the standard formalism of a discounted Markov Decision Process (MDP) defined as the tuple (S, A, P, R, γ, ρ 0 ) where S is the set of states, A is the set of actions, P : S × A → S is the (unknown) transition function providing the probability distribution of the next state given a current state and action, R : S ×A×S → R is the reward function, γ is the discount factor and ρ 0 is the initial distribution of states. A policy, denoted π : S × A → R + is the probability distribution such that π(a|s) is the probability of choosing action a in state s.

We denote the previously defined values with discrete time t such that s t , a t and r t denote respectively the state, action, and reward at timestep t. The goal is to learn a policy π that maximizes the long-term expected reward E[

+∞ t=0 γ t r t ].
Goal-conditioned MDP We note that every MDP can be augmented into a goal-conditioned MDP with a goal space G and an initial goal distribution ρ g . At each timestep, the observation is augmented with a goal and the reward function depends on this goal. A goal-conditioned policy (Kaelbling, 1993), denoted π(•|•, •) also depends on the goal.

Related Work

Exploration in RL can be divided into three types (Ładosz et al., 2022): unstructured exploration1 , intrinsic rewardsbased methods, and goal-based methods.

Unstructured exploration In unstructured exploration, the agent does not adhere to a predetermined exploration plan and instead takes actions randomly or according to a simple heuristic. These actions may be sampled uniformly from the action space, or in the continuous case, they may be augmented by exploration noise that is parametrized by the current state [START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] or not [START_REF] Lillicrap | Continuous Control with Deep Reinforcement Learning[END_REF]. Unstructured exploration can be effective in some environments, but it may not be sufficient to explore more complex or sparsely rewarded environments.

Intrinsic rewards-based methods Intrinsic rewardsbased methods are inspired by the concept of intrinsic motivation in cognitive science [START_REF] Oudeyer | What is Intrinsic Motivation? A Typology of Computational Approaches[END_REF]. They involve the addition of an additional reward signal, called intrinsic, to the reward signal from the environment, called extrinsic. This intrinsic reward is designed to encourage exploration. It can be based on the state visitation count (Bellemare et al., 2016;[START_REF] Machado | Count-Based Exploration with the Successor Representation[END_REF] or on the prediction error of a model learned from the collected data (Houthooft et al., 2016;[START_REF] Pathak | Curiosity-Driven Exploration by Self-Supervised Prediction[END_REF][START_REF] Achiam | Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning[END_REF][START_REF] Pathak | Self-Supervised Exploration via Disagreement[END_REF][START_REF] Burda | Exploration by Random Network Distillation[END_REF]Tao et al., 2020).

Goal-based methods Methods that modify the reward of the environment provide no mechanism for distilling the knowledge gained from visiting various states. Agents may visit new states, but they quickly forget about them when other states become newer. To address this issue, recent work has suggested the use of a goal-conditioned autotelic agent specifically trained for the exploration task. This approach allows for the use of the knowledge gained during exploration to realize new user-specified goals [START_REF] Levine | Understanding the World Through Action[END_REF][START_REF] Colas | Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey[END_REF]. During the exploration phase, the reward signal is ignored, and after the exploration phase, the data collected by the agent is used to learn one or more subsequent tasks [START_REF] Jin | Reward-Free Exploration for Reinforcement Learning[END_REF]. Goal-based methods condition the agent with a goal that is used to guide exploration towards unknown areas. These methods rely on a goal generator to create goals for the agent. We divide goal-based methods into two categories: exploratory goal methods and goals to explore from methods (called post-exploration in [START_REF] Yang | First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation[END_REF]).

Exploratory goal methods follow the intuition that the agent discovers new areas of the observation space by pursuing goals that have been little or not achieved before. The challenge of these methods is to choose the goal to be neither too easy nor too hard. The literature contains several ways to approach this trade-off. Some methods sample goals that either maximize Learning Progress [START_REF] Colas | Intrinsically Motivated Modular Multi-Goal Reinforcement Learning[END_REF][START_REF] Portelas | Teacher Algorithms for Curriculum Learning of Deep RL in Continuously Parameterized Environments[END_REF] or value disagreement (Zhang et al., 2020). Other methods sample goals from the least visited areas using a parametric density model on the visited states [START_REF] Pong | Skew-Fit: State-Covering Self-Supervised Reinforcement Learning[END_REF]. It is also possible to imagine goals that have never been reached using a language model [START_REF] Portelas | Teacher Algorithms for Curriculum Learning of Deep RL in Continuously Parameterized Environments[END_REF], a generative model [START_REF] Racanière | Automated Curriculum Generation through Setter-Solver Interactions[END_REF] or a GAN [START_REF] Florensa | Automatic Goal Generation for Reinforcement Learning Agents[END_REF].

In goals to explore from methods the agent samples a goal from previously visited states. It returns to it, either by teleportation [START_REF] Ecoffet | Go-Explore: a New Approach for Hard-Exploration Problems[END_REF][START_REF] Matheron | Efficient Exploration and Exploitation Using a Synergy Between Reinforcement Learning and Motion Planning[END_REF], or using a goal-conditioned policy [START_REF] Ecoffet | First return, then explore[END_REF]. The challenge of these methods is to choose a goal that is of high exploratory interest. Similarly, some methods estimate the density of the encountered states, using either parametric methods [START_REF] Pitis | Maximum Entropy Gain Exploration for Long Horizon Multigoal Reinforcement Learning[END_REF] or non-parametric methods [START_REF] Ecoffet | First return, then explore[END_REF][START_REF] Matheron | Efficient Exploration and Exploitation Using a Synergy Between Reinforcement Learning and Motion Planning[END_REF], to target the low-density areas.

In summary, methods based on a goal reaching policy should facilitate scalable RL. The stunning results of Go-Explore illustrate this point but remain circumscribed to few environments and require a lot of domain knowledge to work. By bridging with concepts already used in the intrinsic reward literature, we show a way to make this approach more general and simpler.

Latent Go-Explore

LGE meets the definition of the return-then-explore family of algorithms. First, a final goal state is sampled from the replay buffer, then the agent learns a goal-conditioned policy to reach this goal. When the agent reaches the goal, the agent starts to explore. LGE learns a latent representation of observations and samples the goal pursued by the goalconditioned agent in priority in low latent density areas. In Section 3.1, we present how the latent representation of observations is learned. In Section 3.2, we show how the latent density is estimated and how the final goal state pursued by the agent is sampled. Finally, in Section 3.3, we show how to build a subgoal trajectory from the final goal to increase the agent's performance, in particular in far-away goal situations. The pseudo-code of the resulting algorithm is presented in Algorithm 1.

Learning a Latent Representation

The literature contains several latent representation learning methods for RL. Learning such a representation is orthogonal to our approach. Hence, LGE can be combined with any learning method without the need for further modifications. Choosing the best representation learning method given the environment is out of the scope of this paper. In this paper, we present three methods of representation learning that have been found to work well with our test environments. Two of these methods are inspired by the literature on intrinsic reward-based methods, Inverse dynamic representation learning [START_REF] Pathak | Curiosity-Driven Exploration by Self-Supervised Prediction[END_REF] proposed an intrinsic reward calculated based on the agent's prediction error of the consequence of its own actions. The representation is learned using two submodules. The first encodes the observation into a latent representation ϕ(s t ). The second takes as input ϕ(s t ) and ϕ(s t + 1) and outputs the prediction of the action taken by the agent at time step t. The parameters θ of the inverse model P inv θ are optimized by minimizing the loss function:

L = 1 |N | (st,at,st+1)∼D 1 2 ∥a t -P inv θ (s t , s t+1 )∥ 2 2 (1)
The inverse dynamics representation learning allows getting a latent representation of the states containing only the aspects of the state on which the agent can have an influence.

Forward dynamic representation learning In Achiam & Sastry (2017), the intrinsic reward is calculated based on the prediction error of a model approximating the transition probability function of the MDP. Two submodules are used.

The first one encodes the observation to a latent representation ϕ(s t ). The second takes as input ϕ(s t ) and a t and outputs the prediction of the next state ŝt+1 . The model parameters θ are optimized by minimizing the loss function:

L = - 1 |N | (st,at,st+1)∼D log P θ (s t+1 | s t , a t ) (2)
Vector Quantized Variational Autoencoder (VQ-VAE) Autoencoding [START_REF] Hinton | Reducing the Dimensionality of Data with Neural Networks[END_REF] aims to train a neural network to reconstruct its input by learning a compressed representation of the data. This approach is known to be effective in extracting useful features from the input, especially images. For Atari environments, we use a VQ-VAE (van den Oord et al., 2017), a technique that combines autoencoding with vector quantization, and has shown good results, while being simple to train. We use the coordinates of the embeddings in the embeddings table as the latent representation.

Density Estimation for Intrinsic Goal Sampling

The success of the proposed method relies on the agent's ability to generate for itself goals that it will be able to reach and then explore from there. For the agent to progress in the exploration of the environment, these goals must be at the edge of the yet unexplored areas. To identify these areas, we use an estimator of the density of latent representations of the encountered states. Moreover, we require the goal to be reachable. The set of reachable states is a subset of the state space that we assume to be unknown. The easiest way to satisfy the previous requirement is therefore to sample among the states that have already been reached.

We estimate the density of latent representations of the encountered states (called latent density) using the particlebased entropy estimator originally proposed by [START_REF] Kung | An optimal knearest neighbor for density estimation[END_REF] and used in the literature on intrinsically motivated RL (Liu & Abbeel, 2021b;a). This estimator has the advantage of being nonparametric and thus does not hinge on the learning capabilities of a learned model. Appendix C describes the details of the implementation of this estimator, denoted f .

The sampling of the final goal state follows a geometric law on the rank in the latent density sort R i . The probability to draw s i as the final goal state is

P(G = s i ) = (1 -p) Ri-1 p (3)
where G is the random variable corresponding to the final goal state, and 0 ⩽ p ⩽ 1 is a hyperparameter controlling the bias towards states with a low latent density.

This method has the advantage of being robust to approximation errors in the density evaluation, which can be particularly important in low density areas. In doing so, we only focus on the ability of the model to correctly order the observations according to their latent density.

The representation is jointly learned with the exploration of the agent. Therefore, the latent density must be regularly recomputed to take into account the most recent representation on the one hand, and the recently visited states on the other hand. However, considering the slow evolution of this value, we choose to recompute the latent density only once every 5k timesteps for maze and robotic environments, and every 500k timesteps for Atari environments. This allows us to significantly reduce the computation needs while having a low empirical impact on the results.

Subgoal Trajectory

As learning progresses, the sampled final goal states are increasingly distant. However, reaching a distant goal is challenging because it implies a sparse reward problem.

To overcome this problem, we condition the agent to successive intermediate goals τ g = (g 0 , g 1 ..., g L ) that should guide it to the final goal state g L . These intermediate goals are chosen from the trajectory that led the agent to the final goal state (s 0 , s 1 , . . . , s T ).

The trajectory that led the agent to the final goal state is unlikely to be optimal. Plus, if the agent is conditioned by the whole trajectory, it may fail to reach all of them, even though some of them may not be necessary to reach the final goal state. To allow the agent to find a better path to the final goal state, we remove some subgoals from this trajectory.

To decide whether a subgoal should be removed from the trajectory, we evaluate the latent distance to the previous subgoal. If the distance is less than the threshold, then the goal is removed.

∀i ⩽ L -1, ||ϕ(g i ) -ϕ(g i+1 )|| > d (4)
Unlike Go-Explore, LGE don't use the best known trajectory that leads to the sampled goal area (cell). The main reason is that the best known trajectory may be particularly difficult to reproduce, due to the dynamics and the stochasticity of the environment, or cause the early termination of the episode. For example, in Montezuma's Revenge, there are two ways to reach the bottom of the left ladder (a necessary step to get the first key). The first one consists in jumping right from the promontory, but causes the death of Panama Joe (the character) and as a result ends of the episode. The second one, longer, consists in going around by the right ladder. Therefore, if the agent always chooses the shortest path (like Go-Explore), it will most likely fail to reach the first key and to further explore the environment.

Once the goal is reached, the agent explores using any exploration strategy. For the sake of simplicity, we choose a random exploration strategy for our experiments. We also impose that the agent repeats the previous action with a probability of 90%. This technique has been shown to increase the results significantly [START_REF] Ecoffet | First return, then explore[END_REF].

Experiments

To demonstrate the effectiveness of our method, we apply it to a range of pure exploration tasks. We focus on environments for which naive random exploration is not sufficient to explore the rich variety of reachable states. We compare the results obtained with LGE with the results obtained using several algorithms based on intrinsic curiosity and others based on goal-directed strategies. For each environment, LGE uses the representation method that empirically gives the best results. Consequently, we use the forward dynamics for the maze environment, the inverse dynamics for the robotic environment, and the VQ-VAE for Atari.

In terms of infrastructure, each run was performed on a single worker machine equipped with one CPU and one NVIDIA® V100 GPU + 120 Gb of RAM.

Environments

Continuous maze We train an agent to navigate in a continuous 2D maze. The corresponding configuration is shown in Figure 2. The agent starts every episode in the center of the maze. At each timestep, the agent receives the current coordinates as an observation and chooses an action that controls its location change. If the agent collides with a wall, it returns to its previous position. The reachable space is a square of 12 × 12 and the agent's action is limited to [-1, 1] horizontally and vertically. The agent can interact 100 times with the environment (which is just enough to explore all the maze), after which the episode ends.

Robotic environment Robotic environments are interesting and challenging application cases of RL, especially since the reward is often sparse. We simulate a Franka robot under the PyBullet physics engine using panda-gym [START_REF] Gallouédec | panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning[END_REF]. The robot can move and interact with an object. The agent has access to the position of the end-effector and the position of the object, as well as to the opening of the gripper. The agent interacts 50 times with the environment and then the object and the robot arm are reset to their initial position.

Atari We train LGE on two high-dimensional Atari 2600 environments simulated through the Arcade Learning Environment (ALE, [START_REF] Bellemare | The Arcade Learning Environment: An Evaluation Platform for General Agents[END_REF]) that are known to be particularly challenging for exploration: Montezuma's Revenge and Pitfall. Details of the settings used are presented in Appendix A.

Baselines

RANDOM EXPLORATION

Most RL methods from the literature do not follow any structured exploration strategy. In a reward-free context, the performance of the latter is often equivalent to a random walk. We take as a reference a random agent, whose actions are uniformly sampled over the action space at each time step, Soft Actor-Critic (SAC, [START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]) and Deep Deterministic Policy Gradient (DDPG, Lillicrap et al. ( 2016)) for continuous action space environments.

INTRINSIC REWARD-BASED EXPLORATION

In this paper, we take as reference two widely used intrinsic reward-based methods combined with either SAC or DDPG. These methods stand out from the others because, despite their simplicity, they have demonstrated good performance on a wide variety of tasks.

Intrinsic Curiosity Module (ICM, Pathak et al. ( 2017))

The intrinsic reward is computed as the mean square error between the true latent representation and the one predicted by a learned dynamic model given the action taken. The encoder is trained jointly with an inverse dynamics model.

Surprise [START_REF] Achiam | Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning[END_REF] The intrinsic reward is the approximation of the KL divergence between the actual transition probabilities and a learned transition model.

GOAL-DIRECTED EXPLORATION

Go-Explore The agent divides the observation space into cells, prioritizes the cells that have been visited the least, returns to them using a goal-conditioned policy, and then continues exploring from that point. This is the policy-based and without domain knowledge variant of Go-Explore, but we refer to it simply as Go-Explore. The observations in continuous environments are converted into cell representations by discretizing them. In the maze environment, we use a 24 × 24 grid, and in the robotic environment, we use a grid with a 0.1m resolution for the position of the gripper and object. For Atari, we use the same fixed cell representation as proposed by [START_REF] Ecoffet | First return, then explore[END_REF] in the policy-based case: the observation is converted to grayscale and reduced to the size of 8 × 11 pixels. The depth is then reduced from 256 to 8 values according to ⌊ 8p 255 ⌋ where p is the pixel value. The resulting image is the representation of the cell. Go-Explore is the closest baseline to our algorithm. Appendix D details the differences between LGE and Go-Explore.

Diversity Is All You Need (DIAYN, Eysenbach et al. ( 2019a)) The agent is conditioned by a skill and a discriminator predicts the skill pursued by the agent. The more the discriminator predicts with certainty the skill pursued, the bigger the reward. Conjointly, the discriminator is trained to maximize the distinguishability of skills.

Skew-Fit [START_REF] Pong | Skew-Fit: State-Covering Self-Supervised Reinforcement Learning[END_REF] The agent's goal sampling is skewed to maximize the entropy of a density model learned on the achieved states.

For the goal-directed methods, we use Hindsight Experience Replay (HER, [START_REF] Andrychowicz | Hindsight Experience Replay[END_REF]) relabeling which has shown to significantly increase learning.

To nullify the variation in results due to different implementations, we implement all algorithms in the same framework: Stable-Baselines3 [START_REF] Raffin | Stable-Baselines3: Reliable Reinforcement Learning Implementations[END_REF]. The set of intrinsic reward-based methods and goal-directed methods are underpinned by the same off-policy algorithm. The hyperparameters for this algorithm are identical. For the maze environment, we use SAC, while for the robotic environment, we use DDPG as it gives better results for all methods. For Atari environments, we use QR-DQN [START_REF] Dabney | Distributional Reinforcement Learning With Quantile Regression[END_REF], as it commonly considered to be a strong baseline on it. For Atari, we only compare LGE to Go-Explore as it far outperforms the others. To negate the influence of a bad choice of hyperparameter on the results, the methodspecific hyperparameters are optimized. Appendix A details the optimization process and the resulting hyperparameters.

Measuring the Exploration

In this paper, we focus on the agent's ability to explore its environment in a pure exploration context, i.e. in the absence of extrinsic reward. This step is particularly important because, in the case of an environment with very sparse rewards, the agent can interact a large number of times with the environment without getting any reward. It is therefore necessary to follow an efficient exploration strategy to discover the few areas of the state space where the agent can get a reward. To be able to compare the results obtained by different methods in this context, it is necessary to use a common metric for the quality of exploration. The literature uses various metrics. Some papers use the average reward on a hard-exploration task [START_REF] Ali Taïga | On Bonus-Based Exploration Methods in the Arcade Learning Environment[END_REF], the zero-shot performance on a predefined task [START_REF] Sekar | Planning to Explore via Self-Supervised World Models[END_REF] or monitor specific identifiable events in the environment that indirectly informs the degree of exploration [START_REF] Gülc ¸ehre | Making Efficient Use of Demonstrations to Solve Hard Exploration Problems[END_REF]. We argue that these indirect measures are unsatisfactory as they rely on the subsequent learning ability of an online and offline agent respectively.

For simplicity, we use the number of visited cells as the metric, whose construction strategy is explained in Section 4.2.3. Therefore, the figures represent the number of cells explored, although Go-Explore is the only algorithm to explicitly maximize this metric. Following the guidelines of [START_REF] Agarwal | Deep Reinforcement Learning at the Edge of the Statistical Precipice[END_REF], we use for all plots in this paper the interquartile mean (IQM) with the 95% confidence interval.

Main Results

The exploration results for the maze environment are presented in Figure 3. A rendering of the positions explored by the agent is presented in Figure 2. We note that only LGE and Go-Explore significantly outperform the results obtained with random exploration. This demonstrates the effectiveness of the return-then-explore paradigm in this environment. We note that exploration based on intrinsic curiosity does not yield significantly better results than those obtained by random exploration. We hypothesize that the simple dynamics of the environment makes the intrinsic reward to quickly converge to 0.0. Surprisingly, neither Skew-Fit nor DIAYN performs significantly better than random exploration. For DIAYN, we find that most of the skills were concentrated in the initial position area of the agent. We hypothesize that this is the consequence of the lack of post-exploration described by [START_REF] Yang | First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation[END_REF]. Finally, we note that, although the cell size has been optimized, LGE significantly outperforms Go-Explore. LGE manages to cover almost the entire reachable space at the end of the runs while exhibiting low variability in the results.

The exploration results for the robotic environment are presented in Figure 4. We notice that LGE significantly outperforms all other methods. Notably, Go-Explore performs only slightly better than random exploration. We note that Go-Explore does not learn to grasp the object throughout the learning process. The results presented on a robotic environment by [START_REF] Ecoffet | First return, then explore[END_REF] are much better. We presume this is mainly due to the meticulous work done on the state space examination and the induced cell design.

Here, we use a naive grid-like cell design. Although the grid parameter is optimized, it does not yield good exploration results with this environment. We thus demonstrate the benefit of using a learned representation to automatically capture important features of the environment's dynamics.

The exploration results for Atari are presented Figure 5. We see that both LGE and Go-Explore quickly discover a large number of cells, then continue their exploration by regularly discovering new cells.

LGE slightly outperforms Go-Explore on both Pitfall and Montezuma's Revenge. Nevertheless, we note that the number of discovered cells is much lower than that of Go-Explore in its full configuration (around 5k for Montezuma's Revenge), including domain knowledge and the ability to reset the environment in any state. This shows the criticality of these settings for exploring these particular environments.

Ablation Study

We study the impact of the ablation of three key elements of LGE.

(1) LGE without further exploration (as exploratory goal methods, see Section 2.2): the environment is reset once the agent reaches the final goal instead of performing the exploration random interactions.

(2) LGE without skewing the final goal distribution in favor of low latent density areas, the final goals are sampled uniformly among the reached goals. (3) LGE without subgoals trajectory reduction: the agent is just conditioned by the final goal state.

We perform these ablation studies on the maze environment and use the same hyperparameters as in Section 4.4. The results are shown in Figure 6.

The impact of the three ablations on the outcome is significant. We find that exploration after reaching the final goal is crucial, confirming the results of [START_REF] Yang | First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation[END_REF]; without it, the agent reaches the limits of its knowledge but has little chance to explore further. Additionally, sampling goals with low latent density can significantly improve results by directing exploration to states with high exploratory value. Furthermore, we observe that conditioning the agent with successive subgoals greatly improves its exploration.

Discussion

Limitation and Future Work

Goal-achievement functions In LGE, an agent is considered to have reached a goal (whether final or intermediate)

when the latent distance between its state and the goal is below a threshold. This is a naive way of defining a goal achievement function [START_REF] Colas | Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey[END_REF] that depends crucially on the latent representation. We believe that the results could be improved by envisioning a more informative and suitable goal achievement function for our method.

The initial state must remain the same across episodes The approach we propose is based on the assumption that the agent is always initialized in the same state. This assumption guarantees that at the beginning of each episode, all the states previously reached are reachable and that the subgoal trajectory starts with the initial state of the agent. However, in some environments, especially in procedurally generated environments, this assumption is not fulfilled (Küttler et al., 2020). In this situation, trying to follow the subgoal trajectory may be counterproductive in reaching the final goal.

It is also possible that the final goal is not even reachable. However, note that even the pursuit of an unreachable goal can foster exploration. We believe that an approach inspired by generative networks such as [START_REF] Racanière | Automated Curriculum Generation through Setter-Solver Interactions[END_REF]) may be appropriate to overcome this problem.

High-dimension environments and representation learning Our main contribution consists in the generalization of the Go-Explore approach by using a latent representation.

LGE is notably effective in high-dimensional environments, specifically those with image observations. Representation learning is the keystone of the method. We provide a proof of concept for a forward model, an inverse model, and a VQ-VAE. We believe that the results can be greatly improved by choosing more finely the representation learning method for each environment by taking advantage of the many works dealing with this subject [START_REF] Lesort | State representation learning for control: An overview[END_REF].

Representations are expected to encapsulate transitional proximity between observations, a feature not guaranteed by most learning methodologies. Nonetheless, in practice, such transitional proximity is often exhibited in learned representations.

The representation used by Search on Replay Buffer (SoRB) (Eysenbach et al., 2019b) is directly that of the critic. Using the same reward structure as LGE, the critic thus has the nice property of basically learning the negative distance of the shortest directed path between two states. Overall, we believe that the use of SoRB in the "Go" phase can be a substantial improvement of LGE and is a promising way to solve the three limitations mentioned above.

Finally, we believe that the community should endeavour to find a relevant metric for exploration, especially for imagebased environments. We expect that such a metric would allow a more accurate comparison of different methods.

Conclusion

We introduce LGE, a new exploration method for RL. In this method, our agent explores the environment by selecting its own goals based on a jointly learned latent representation.

LGE can be used as pre-training in environments where rewards are sparse or deceptive. Our main contribution is to generalize the Go-Explore algorithm, allowing us to benefit from representation learning algorithms for exploration. We present statistically robust empirical results conducted on diverse environments, including robotic systems and Atari games, that demonstrate our approach's significant improvement in exploration performance.

C. On the Density Estimation

Let s 1 , . . . , s n be a sample of event locations in a R d -space. Assume that the event location s follows a common distribution with density function f (s). For any sample s i and s j in this sample, assume that D i (s j ) = ||s is j ||, denotes the euclidian distance between s i and s j . For any k ⩽ n, let D (k) (s i ) be the distance with k-th nearest neighbors of s i with respect to the euclidian distance. [START_REF] Kung | An optimal knearest neighbor for density estimation[END_REF] propose an optimal unbiased estimator f for the density:

f = kU * k k -1 (5)
where

U * k = (k -1) nC d D d (k) (6)
and

C d = π d/2 Γ(d/2 + 1) (7)
Hence we have

f = k nC d D -d (k) (8)
We follow the recommendation of [START_REF] Kung | An optimal knearest neighbor for density estimation[END_REF] to take k = 2n 1/d (9)

D. Comparing Go-Explore and LGE

The Go-Explore algorithm as presented by [START_REF] Ecoffet | First return, then explore[END_REF] has many components. All these components allow to obtain good results on test environments. In this article, we implement our own version of Go-Explore. We have tried to stick as much as possible to the initial implementation and to improve some aspects. We keep the essence of Go-Explore, but our implementation is not intended to be equivalent to the initial implementation. The main goal here is to compare LGE and Go-Explore. Thus, the two implementations differ only in the elements that make them unique. To the best of our knowledge, all the composnts that we did not implement are compatible with LGE. It is likely that they improve LGE and Go-Explore in a similar way. In this section, we describe the implementation of LGE and Go-Explore. We explain their differences if any.

Policy-based Go-Explore The initial implementation of Go-Explore distinguishes between the case where the environment can be reset to any desired state and the case where this is not possible. In this paper, we choose the general setting where the environment can't be reset to any desired state, and we therefore work with the so-called policy-based implementation of Go-Explore.

Exploration after returning In the original implementation of Go-Explore, once a cell is returned, exploration proceeds with random actions for a certain number of timesteps.

For both LGE and Go-Explore, we set this number of timesteps to 50 for all environments. Note that the agent can interrupt this exploration beforehand if the maximum num-

Figure 1 .

 1 Figure 1. LGE exploration workflow. The encountered observations are encoded in a latent space. A latent density is estimated. A final goal is sampled from the states already reached, by skewing the distribution with the density. A goal-conditioned agent is trained to reach this goal by pursuing a sequence of subgoals, derived from the experiment that led to the final goal. Once the agent has reached the final goal, it explores from it with any exploration strategy.

Figure 2 .

 2 Figure 2. Space coverage of the maze environment after 100k timesteps. In (f), the different colors are the different skills.

Figure 3 .

 3 Figure3. Comparison of the space coverage of the maze environment. Each experiment is run 10 times. The left plot represents the space coverage (number of cell divided by the total number of reachable cells) across timesteps. The solid lines are the IQMs and the shaded areas are the 95% confidence intervals. The right plot is the final performance profile (higher is better).

Figure 4 .

 4 Figure 4. Comparison of with the robotic environment.Each experiment is run 10 times. The left plot represents the number of explored bins across timesteps. The solid lines are the IQMs and the shaded areas are the 95% confidence intervals. The right plot is the final performance profile (higher is better).

Figure 5 .

 5 Figure 5. Comparison of exploration on the Atari environments. Each experiment is run 3 times. The solid lines are the IQMs and the shaded areas are the 95% confidence intervals.

Figure 6 .

 6 Figure6. Result of ablation study on the maze environment. Each experiment is run 10 times. The left plot represents the space coverage across timesteps. The solid lines are the IQMs and the shaded areas are the 95% confidence intervals. The right plot is the final performance profile (higher is better).

Table 2 .

 2 Hyperparameters specific to each method. Their value is identical for all experiments and have not been optimized.

	METHOD	HYPERPARAMETER	VALUE
	LGE	Encoder module trained every N timesteps 5k (Maze and Robotic), 500k (Atari)
		Learning	0.001
		Batch size	32
		Gradient steps	500 (Maze and Robotic), 5k (Atari)
		Exploration strategy	Random
		Repeat action probability	0.9
	GO-EXPLORE Exploration strategy	Random
		Repeat action probability	0.9
	ICM	Feature size	16
		Networks	[64, 64]
		Activation function	ReLU
	SURPRISE	Networks	[64, 64]
		Activation function	ReLU
	DIAYN	Discriminator networks	[256, 256]
		Activation function	ReLU
	SKEW-Fit	Gradient steps	100
		Batch size	2048
		Learning rate	0.01

Univ Lyon, Centrale Lyon, CNRS, INSA Lyon, UCBL, LIRIS, UMR5205, F-69130 Ecully, France. Correspondence to: Quentin Gallouédec <quentin.gallouedec@ec-lyon.fr>. Proceedings of the 40 th International Conference on Machine Learning, Honolulu, Hawaii, USA. PMLR

202, 2023. Copyright 2023 by the author(s).

We replace the terminology ofŁadosz et al. (2022) random exploration by unstructured exploration that we think is more accurate.

Acknowledgements

We would like to thank Adrien Ecoffet, Joost Huizinga, and Jeff Clune for their encouraging feedback and for the time they spent discussing this exciting topic. We would also like to thank the reviewers of our article for their efforts in reviewing and providing valuable comments. Their comments and suggestions were very helpful in converging on the final version of this work. This work was granted access to the HPC resources of IDRIS under the allocation 2022-[AD011012172R1] and 2022-[AD011013894] made by GENCI.

A. Hyperparameters and Environments Settings

To limit the impact of the large variability of results depending on the hyperparameters, we chose to optimize the hyparameters for each experiment. For maze and robotic environments, we selected 100 unique sets of hyperparameters from a search space presented in Table 1 using Optuna [START_REF] Akiba | A Next-generation Hyperparameter Optimization Framework[END_REF]. For each hyperparameter set, we train the model with 3 different seeds and keep the median score.

For Atari, we selected 10 unique sets of hyperparameters and train the agent just once.

The method-specific parameters that have not been optimized are presented in Table 2.

The hyperparameters used for the off-policy agent are identical for all algorithms. They are presented in Table 3.

For Atari, we mainly use the setting recommended by [START_REF] Machado | Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents[END_REF]. Like [START_REF] Ecoffet | First return, then explore[END_REF], we use both sticky actions and start no-ops.

B. On the Criticality of Cell Representation in Go-Explore

In Go-Explore, similar observations are grouped into cells and each cell encountered is stored in an archive. The cell representation is a critical aspect of Go-Explore. In the Montezuma's Revenge environment, a slight variation in cell representation results in an order of magnitude difference in the results. The cells are used to (1) estimate the density of states encountered in the observation space and sample a target cell against it;

(2) divide this goal reaching task into a sequence of subgoals.

We argue that building a cell representation to capture the relevant components of an environment to perform the desired task requires a significant amount of domain knowledge. In general, this cell representation cannot be generalized to other tasks or to other environments.

To support our claim, we present in Figure 7 the space coverage in a continuous maze for different cell design. We show that even in this simple environment, a small variation in cell design has a significant impact on the result.

On the left, the cells are small, and the agent must visit each of them. If the agent interacts long enough with the environment, it should eventually explore the whole space. On the right, the cells are large. We can observe some detached areas, because the agent has not visited the cell enough to discover the next one, but enough so that this cell is no longer listed as a target cell. ber of interactions with the environment is reached. [START_REF] Ecoffet | First return, then explore[END_REF] shows that action consistency generally allows for more effective exploration, especially in the robotic environment. For LGE and Go-Explore, we use the same trick: the agent chooses the previous action with a probability of 90%, and uniformly samples an action with a probability of 10%.

Cell design

The original implementation of Go-Explore provides two methods for generating the cell representation.

1. When the observation is an image, the observation is grayscaled and downscaled. The image produced is the cell representation. The parameters to get this representation (downscaling width and height and number of shades of gray) are optimized during training to maximize an objective function that depends on a target split factor.

2. When the observation is a vector, each component of the vector is discretized separately by hand before learning.

For all the environments presented in this paper, we use a naive method of cell generation corresponding to a discretization of the observation. The granularity of the discretization is a hyperparameter. The choice of this hyperparameter is crucial, we develop it in more detail in the Appendix B.

Goal-conditioning In the original implementation of Go-Explore, the agent is conditioned by the cell representation of the goal. We note that this representation can vary during learning, and even in size (see previous paragraph). It is not clear how to structure the agent's network when the size of the input varies during the learning process.

In our implementation, we choose to condition the agent by the goal observation rather than by the representation of its cell. We also condition the agent by the goal observation in LGE.

RL agent

In the original implementation of Go-Explore, the goal-conditioned agent is based on the on-policy PPO algorithm [START_REF] Schulman | Proximal Policy Optimization Algorithms[END_REF]. For both LGE and Go-Explore, we rather chose to use an off-policy algorithm (SAC or DDPG) to use a Hindsight Experience Replay (HER, [START_REF] Andrychowicz | Hindsight Experience Replay[END_REF]) relabelling, which has shown to perform better in a sparse reward environment.

Reward In the original implementation of Go-Explore, the agent gets +1 reward for reaching intermediate cells and +5 reward for reaching the final cell of a path. The rest of the time, it receives a 0 reward. For both LGE and Go-Explore, following the suggestions of [START_REF] Tang | Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning[END_REF], we rather choose the following structure for the reward: the agent gets a reward of 0 for a success (target cell reached for Go-Explore and latent distance with the goal state below the distance threshold for LGE) and -1 the rest of the time.

As noted by (Eysenbach et al., 2019b), in this setting, an optimal agent tries to terminate the episode as quickly as possible. We therefore set done = True only at the end of the post-exploration, and not when the agent reaches a final state.


Huan-Ting Shen

Claude Weisbuch

James S Speck

Yuh-Renn Wu

Three-dimensional modeling of minority-carrier lateral diffusion length including random alloy fluctuations in (In,Ga)N and (Al,Ga)N single quantum wells

For nitride-based InGaN and AlGaN quantum well (QW) LEDs, the potential fluctuations caused by natural alloy disorders limit the lateral intra-QW carrier diffusion length and current spreading. The diffusion length mainly impacts the overall LED efficiency through sidewall nonradiative recombination, especially for µLEDs. In this paper, we study the carrier lateral diffusion length for nitride-based green, blue, and ultraviolet C (UVC) QWs in three dimensions. We solve the Poisson and drift-diffusion equations in the framework of localization landscape theory. The full three-dimensional model includes the effects of random alloy composition fluctuations and electric fields in the QWs. The dependence of the minority carrier diffusion length on the majority carrier density is studied with a full three-dimensional model. The results show that the diffusion length is limited by the potential fluctuations and the recombination rate, the latter being controlled by the spontaneous and piezo electric fields in the QWs and by the screening of the the by the screening of the internal electric fields by carriers.

I. INTRODUCTION

The increasing demand for lighting, sterilization, and display units has led to traditional light sources being replaced by light-emitting diodes (LEDs), which offer high power efficiency. In LEDs, the diffusion length in the lateral direction and the defect density both impact the internal quantum efficiency and the possible current spreading, particularly as chip sizes decrease. The efficiency of nitride-based LEDs is limited in part by the density of nonradiative recombination (NR) centers. The natural random alloy fluctuations in InGaN are believed to cause carrier localization 1 and suppress the carrier diffusion to NR centers. However, the influence of random alloy potentials is difficult to analyze quantitatively due to computational limitations. Our recent studies with a full threedimensional (3D) simulation model have investigated the vertical carrier transport with respect to alloy fluctuations and thickness fluctuations, 2-4 even in ultraviolet C (UVC) LEDs, 5 in which fluctuating potentials occur in both quantum wells (QWs) and quantum barriers (QBs). Hence, it is important to further investigate the impact of random alloy fluctuations on the lateral carrier transport.

The existence of random alloy fluctuations has been extensively studied. Usually, optical methods can resolve alloy fluctuations at scales down to around 50 nm, [START_REF] De | Two distinct origins of highly localized luminescent centers within InGaN/GaN quantum-well light-emitting diodes[END_REF][START_REF] Mensi | Direct measurement of nanoscale lateral carrier diffusion: toward scanning diffusion microscopy[END_REF][START_REF] Sonderegger | High spatial resolution picosecond cathodoluminescence of InGaN quantum wells[END_REF][START_REF] Maruo | Three-dimensional microfabrication with two-photon-absorbed photopolymerization[END_REF] mainly limited by the optical resolution. A recent scanning tunneling luminescence (STL) study directly observed localized states induced by the intrinsic disorder on a scale of a few nanometers. [START_REF] Hahn | Evidence of nanoscale anderson localization induced by intrinsic compositional disorder in InGaN/GaN quantum wells by scanning tunneling luminescence spectroscopy[END_REF] Models including localized carrier effects in disordered systems indicate that these fluctuations change the carrier mobility and lifetime. 5,[START_REF] David | Droop in InGaN light-emitting diodes: A differential carrier lifetime analysis[END_REF][START_REF] Feix | Ga-polar (In, Ga) N/GaN quantum wells versus N-polar (In, Ga) N quantum disks in GaN nanowires: a comparative analysis of carrier recombination, diffusion, and radiative efficiency[END_REF] In addition, the forward a) Electronic mail: yrwu@ntu.edu.tw voltage V F is improved in LED structures with a strongly fluctuating potential, decreasing the effective barrier height for electron and hole injection into the QWs and increasing the radiative efficiency. [START_REF] Nath | Unipolar vertical transport in GaN/AlGaN/GaN heterostructures[END_REF][START_REF] Browne | Electron transport in unipolar InGaN/GaN multiple quantum well structures grown by NH 3 molecular beam epitaxy[END_REF][START_REF] Qwah | Theoretical and experimental investigations of vertical hole transport through unipolar AlGaN structures: impacts of random alloy disorder[END_REF][START_REF] Nakamura | High-power In-GaN single-quantum-well-structure blue and violet light-emitting diodes[END_REF][START_REF] Nakamura | History of gallium-nitride-based lightemitting diodes for illumination[END_REF] Therefore, the natural alloy disorder that causes potential fluctuations in materials plays an important role in limiting carrier diffusion, preventing them from being captured by defects or surface states. When considering carrier diffusion in QW systems, the issues are more complicated. [START_REF] Shklovskii | Electronic properties of doped semiconductors[END_REF] When a QW is formed in a nitride-based device, the strain-induced piezoelectric polarization charge [START_REF] Romanov | Straininduced polarization in wurtzite III-nitride semipolar layers[END_REF] and spontaneous polarization charge will lead to net polar charges at the interfaces, which results in internal electric fields in the QWs and reduced radiative recombination rates due to decreased electron-hole wave function overlap. This also significantly affects carrier diffusion because the diffusion length depends on the diffusion coefficient and carrier lifetime, which is inversely proportional to the total recombination rate.

The impact of the diffusion length is most critical for µLEDs, as with chip size decreasing, minority carriers have a greater chance of diffusing to the sidewall. High-density surface states become major NR centers. This effect is more severe in AlGaInP-based red LEDs than in nitride-based LEDs. As the carrier diffusion length determines the likelihood of carriers being trapped by surface states, the factors that influence the diffusion length in nitride-based LEDs should be investigated. Although we expect the potential fluctuations ∆V to be important, our studies show that this phenomenon may be affected by additional factors. For example, according to the continuity equation, if the carriers' local radiative recombination rate is less than the injected current density, the excess carriers will be delocalized and leave the local confining potential regions within the QWs, effectively increasing the diffusion length.

This work investigates the impact of random alloy fluctua-tions on the lateral diffusion within nitride alloy QWs. Simulations play a vital role in understanding LED behavior and help guide improved designs. However, addressing the alloy fluctuation effects in an entire LED structure is a difficult simulation task requiring 3D simulation software. Alloy fluctuations cannot be described by one-dimensional (1D) simulation tools, which do not take in-plane fluctuations into account. [START_REF] Karpov | Modeling of III-nitride light-emitting diodes: progress, problems, and perspectives[END_REF][START_REF] Kisin | Inhomogeneous injection in III-nitride light emitters with deep multiple quantum wells[END_REF][START_REF] Geng | Quantitative multiscale, multi-physics quantum transport modeling of gan-based light emitting diodes[END_REF][START_REF] Li | Non-local transport in numerical simulation of GaN LED[END_REF][START_REF] Der Maur | Multiscale approaches for the simulation of InGaN/GaN leds[END_REF] Traditionally, the Poisson, drift-diffusion, and Schrödinger equations are used to solve the problem. Solving the Schrödinger energy eigenvalue equation, however, consumes considerable computational resources and time. Recently, a novel simulation tool using localization landscape (LL) theory was proposed for 3D simulations, replacing the Schrödinger equation in solving the motion of carriers in fluctuating potentials. [START_REF] Filoche | Localization landscape theory of disorder in semiconductors. I. theory and modeling[END_REF][START_REF] Piccardo | Localization landscape theory of disorder in semiconductors. II. Urbach tails of disordered quantum well layers[END_REF][START_REF] Li | Localization landscape theory of disorder in semiconductors. III. application to carrier transport and recombination in light emitting diodes[END_REF][START_REF] Wu | Percolation transport study in nitride based led by considering the random alloy fluctuation[END_REF] Instead of solving the Schrödinger eigenvalue problem, LL theory converts the eigenvalue problem to an partial differential equation with boundaries, from which one can efficiently obtain an effective quantum potential. Use of the LL equations reduces the calculation time. In this work, we apply this method to investigate the diffusion length. Note that, due to the large simulation domain (3 µm) needed to study the lateral diffusion length (∼0.5-2 µm), we only model a single QW structure sandwiched by two QBs. The electron and hole pairs are generated at the middle of the QW, such as would be realized in resonant local optical excitation or local minority carrier injection in STL, and the characteristics of diffusion are studied. The details will be discussed later.

II. METHODOLOGY

As mentioned in the Introduction, modeling the 3D fluctuations in a large domain is difficult due to the high requirements for computing power and computer memory. Hence, the simulation domain size must be compromised. To understand the change of diffusion length in QWs, we design a large-area single QW measuring approximately 3 µm along the lateral xdirection with alloy fluctuations to mimic LED structures. The extent in the y-direction is limited to 15 nm due to memory constraints. The QB thickness is 10 nm and the QW width is 3 nm. The numerical complexity is depending on the mesh size and dimensionality. In our 3D modeling, the typical mesh has 4657552 nodes and 27462280 elements. The memory needed to solve this problem is about 150GB to 200GB. Moreover, the computing time is at least needs 4-8 hours with a 8 cores 16 thread Intel Xeon Silver 4110 CPU for each simulation case. Also, for each case, we need to run 15 different random seeding numbers to get the average results. To reduce the memory requirements, the excitation power (carrier generation) is imposed at one side of the simulation domain, as shown in Fig. 1(a), and we simulate half of the LED. The shaded volume indicates the virtual domain. The electrons and holes have different diffusion length, hence, we separate this study into n-i-n and p-i-p structures in which the holes and electrons are minority carriers, respectively. More precisely, these are n-doped QB/intrinsic-QW/n-doped QB (n-i-n) and p-doped QB/intrinsic-QW/p-doped QB (p-i-p) structures, respectively. The reason for not choosing the undoped case will be discussed later.

To construct the structure, we first use the "gmsh" method 29 to create finite-element meshes in the QW and QB regions, and set the parameters for the different layers. For the InGaN and AlGaN layers, which are the alloy materials, a random number generator selects the In or Al atoms and determines the local composition through the Gaussian averaging method. Details can be found in Refs. 26 and 27. We then use a 3D finite-element method strain solver to calculate the strain field ε i j (r) and polarization distributions P(r). These settings are in- put to the 3D Poisson solver [Eq. (1)], LL solver [Eqs. (2) and (3)], and drift-diffusion solver [Eqs. (4)- (6)] to ensure that the problem is solved self-consistently. The simulation flowchart is shown in Fig. 2(a).

∇(ε∇ϕ) = e(n -p + N - A -N + D ±ρ pol ), (1)
Ĥ = -h 2 2 ∇•( 1 m * e,h ∇) + E c,v , (2)
Ĥu e,h ( r) = 1, (3) J n (r) = nµ n ∇E Fn , (4) J p (r) = pµ p ∇E F p , (5)
∇•J n,p = -eG±e[A 0 + B 0 np + C 0 (n 2 p + np 2 )], (6)
A 0 = np -n 2 i τ n (p + n i e (E i -E t ) k B T ) + τ p (n + n i e (E t -E i ) k B T ) . ( 7
) n = +∞ 1/u e LDOS 3D (E) 1 1 + exp (E+∆E c )-E f n ) k B T (8)
p = 1/u h -∞ LDOS 3D (E) 1 1 + exp E f p -(E+∆E v )) k B T (9)
LDOS 3D (E) = √ 2m * 3 2 π 2 h3 E - 1 u e,h (10)
Equation ( 6) is the equation of continuity, with the Shockley-Read-Hall rate A 0 given in Eq. (7). Here, ε is the permittivity and ϕ is the electrostatic potential, E Fn and E F p are the electron and hole quasi-Fermi levels, and n and p are the free electron and hole densities in Eqs ( 8) and ( 9), respectively, which correspond to the LDOS formula given in Ref.

N -

A and N + D represent the ionized acceptor and donor densities, respectively. ρ pol is the density of polarization charge. J n,p (r), µ n,p , and τ n,p are the current density, mobility, and NR lifetime for electrons and holes, respectively. For a description of where the Eqs. ( 2), ( 8)-( 10) originate from, please see Refs. 25-27. The values for the lifetimes, radiative recombination rate B 0 , mobilities, and Auger coefficient C 0 are listed in Table I, and the ionization energies are given in Table II.

There is some uncertainty about the minority carrier mobilities, as very few measurements exist: Sohi 5 reported 180 cm 2 /Vs for majority electrons in HEMTs, while Solowan 30 deduced a mobility of 10 cm 2 /Vs, which we interpret as the hole mobility, in undoped MQW samples from diffusion length measurements using Einstein's relation. We therefore use conservative values of 30 and 5 cm 2 /Vs for the minority electron and hole mobilities, respectively. The use of mobili- ties significantly lower than those for pure GaN is of course due, at least in part, to alloy disorder effects, for which there is as yet no detailed theory. We shall see below that the diffusion length varies approximately as the square root of the mobility, as expected from the simple model of the usual diffusion equation and Einstein's relation. For Eq. ( 1) in Poisson solver, the fixed boundary condition is used on the top and bottom, which is biased at 0V. For the Eq. ( 3), since this is based on FEM, the Neumann boundary condition was applied where du dr = 0 along surface normal direction. For drift-diffusion equation, we also set the same fixed boundary condition on top and bottom contact, where E f n and E f p are zero at the boundary.

Equations ( 2) and (3) are the Hamiltonians of the Schrödinger equations and the LL theory, and are applied to solve the effective quantum potential. The landscape functions u e,h ( r) for electrons and holes are the solutions of Eq. (3), and 1/u e,h gives the effective potential incorporating the localization properties of the Schrödinger equation. [START_REF] Filoche | Localization landscape theory of disorder in semiconductors. I. theory and modeling[END_REF] After obtaining 1/u e and 1/u h for electrons and holes, respectively, these form the input potential for the 3D-DDCC solver, replacing the original terms E c and E v . This enables us to account for the quantum confinement effects caused by the fluctuating potential in the random alloy system. The detailed formalism and method can be found in Refs. 25-27. The effective mass, permittivity, polar charge, and bandgap are calculated for the local In or Al composition, using linear interpolation where parameters are listed in Table III.

III. RESULTS AND DISCUSSION

Blue, green, and UVC LEDs are studied in this work. [START_REF] Chen | Three dimensional simulation on the transport and quantum efficiency of UVC-LEDs with random alloy fluctuations[END_REF] For the blue LED, we set the average In composition x in the In x Ga 1-x N to 15% at the QW. For the green LED, the average In composition x is 22%. For the UVC-LED, the average Al composition x in the Al x Ga 1-x N composition is 40% in the QW and 60% in the QB. Figures 1(b)-1(d) show the composition maps and Figs. 1(e)-(g) are the x-plane side-views of the calculated conduction band profile, E c . In the QBs, different doping levels are applied to study the carrier diffusion length. The carriers are excited at the red region shown in Fig. 1(a), where we can observe carrier diffusion. The typical radiative recombination rate in the QW is about 10 26 -10 27 cm -3 s -1 . Hence, we set the generation rate in the local excitation area to 10 26 1/cm 3 s by default. When carriers are generated in the red region, they diffuse and eventually recombine in the QW. For the n-i-n structure, the majority carriers in the QW are electrons, and the holes determine the diffusion length. For the pi-p structure, electrons determine the diffusion length because the holes already fill the whole QW. Note that in the p-i-p structures, without putting the excitation power (generation term) in the simulation, the hole density is already very high due to the modulation doping effect from the barriers. Hence, although the generated carriers have electrons and holes, the generated hole density is not as high as the original hole density in QW. Therefore, even the hole will still diffuse out in the simulation under excitation, the hole density does not change too much in QW compared to the original high carrier density. Therefore, the carrier diffusion length is mainly determined by the electron diffusion in the p-i-p structure. For the n-i-n structures, it would be the same concept where the diffusion length would be mainly determined by the hole diffusion.

A. Influence of doping density and polarization field on diffusion length

Figure 3 demonstrates the typical radiative recombination rate decay [START_REF] Solowan | Direct observation of charge carrier diffusion and localization in an InGaN multi quantum well[END_REF][START_REF] Danhof | Lateral charge carrier diffusion in InGaN quantum wells[END_REF][START_REF] Pantha | Electrical and optical properties of p-type InGaN[END_REF] in QWs for the blue LED in the n-i-n structure. Figure 3(a) shows the radiative recombination distribution within a 100-nm region. The 10-nm region from the lefthand side is the excitation area. Figure 3(b) shows the hole (minority carrier) density. The carriers are highly localized due to the potential fluctuations. Figure 3(c) shows the 1D fluctuation potential along the x-direction (with the red cut region over a range of 500 nm). We observe the movement of the hole quasi-Fermi level towards the center of the bandgap due to the diminishing minority hole density away from the injection region. Holes occupy localized states, where they can recombine, but this is not contradictory with hole transport: at room temperature, holes move between localized and delocalized states and conduction can occur in the latter. This parttime contribution to the conduction is reflected by the rather low mobilities observed and used in the simulations.

We observe in Fig. 3(a) that the radiative recombination varies locally. It is hard to determine the diffusion length from the computation of a single random map. Hence, to obtain an averaged result, 15 different seeding numbers were used to generate random alloy composition maps for the simulations, and the results were averaged. Figure 4(a) shows the average recombination rate at each x-section. The shape indicates exponential decay. The diffusion length L can be extracted from the exponential equation

R(x) = Aexp -(x -x c ) L . ( 11
)
where A is a fitting parameter and L is the diffusion length. Figures 4(a) and 4(b) show the y-z averaged radiative recombination rate R(x) along the x-direction. We now discuss the various factors that affect the fitted diffusion length. First, the carrier diffusion lengths with and without alloy fluctuations are studied to clarify their effect on diffusion. Figure 4 shows the result for blue n-i-n QB/QW structures with different doping densities and for the cases with and without random alloy fluctuations. Figure 4(a) shows that, due to the fluctuations seen in Fig. 3, the recombination rate R(x) still fluctuates even when 15 random maps with different seeds are averaged, but is sufficiently regular to extract the diffusion length. Figure 4(b) is the case without alloy fluctuations, which leads to a smooth curve. Figure 4(c) shows the diffusion length versus doping level for cases with and without alloy fluctuations. The red line is the case with alloy fluctuations, but no polarization effects. As shown in Figs. 4(c) and 4(d), the case with alloy fluctuations has a shorter diffusion length than the case without fluctuations. The reason for these effects can be easily understood in terms of the potential fluctuations limiting the carrier diffusion in the QW: the indium composition fluctuations behave like quantum dots in the QW. The recombination in such localized sites is stronger, leading to a reduction in the diffusion length.

However, more interestingly, the diffusion length decreases significantly as the doping density increases. This effect of the doping density is even stronger than the influence of potential fluctuations. An increase in doping density enhances the majority carrier density, but this should not affect recombination rates too much because the recombination rate is determined by the minority carrier density when the majority carrier density is high and in the degenerate condition. As the diffusion length is affected by the diffusion coefficient and the carrier lifetime, the reason for reduction of diffusion length with higher doping in QB is likely to be the change in carrier lifetime with doping, because different doping densities will not strongly affect the diffusion coefficient. The quantumconfined Stark effect (QCSE) strongly influences carrier recombination in nitride QW LEDs. Hence, we set the polarization in the simulation process to zero to investigate the influence of the QCSE. The red line shown in Fig. 4(c) indicates the diffusion length without polarization effects and with random alloy fluctuations: the diffusion length decreases by almost an order of magnitude without the polarization field. The p-i-p structures [see Fig. 4(d)] exhibit a similar trend. Moreover, the diffusion length becomes less dependent on doping density (or majority carrier density in the QW), which is to be expected as the major contributor, i.e., the change in the QCSE, is suppressed. Hence, this shows that, although random alloy fluctuations limit the diffusion length, the QCSE plays a much stronger role in the opposite direction.

We hereafter use a 1D simulation to demonstrate the impact of the QCSE on the radiative rate: Figs. 4(e)-4(g) show the approximated 1D simulation [START_REF] Kawaguchi | Semipolar (2021) single-quantum-well red light-emitting diodes with a low forward voltage[END_REF] for electron and hole overlap in the QW with n = 5× 10 18 cm -3 , n = 4× 10 19 cm -3 , and without polarization effects. The square of the overlap integral for n = 5× 10 18 cm -3 , n = 4× 10 19 cm -3 , and without polarization effects increases to 0.12, 0.23, and 0.898, respectively. The reduction of the QCSE increases the square of the overlap by almost one order of magnitude, which decreases the carrier lifetime by about the same amount. With such a change, the application of the simple diffusion equation implies a factorof-three change in the diffusion length, which is in reasonable agreement with our computations. If there is no majority carrier doping in the QW, the overlap would be further decreased and the diffusion length could even be one order of magnitude larger, extending over a few micrometers, as observed by David, [START_REF] David | Observation of long-range carrier diffusion in InGaN quantum wells, and implications from fundamentals to devices[END_REF] and exceeding our simulation range. Here lies the reason for not presenting cases without doping-the long diffusion lengths that would result are beyond our current computational capabilities. This result indicates that the interpretation of carrier diffusion lengths in experimental work should carefully analyze the carrier screening conditions in each case. For the green and UVC cases, It has similar trends as the blue case. Hence we put simulation results in Fig. 7 in Appendix.

To further investigate the influence of potential fluctuations due to random alloys, we study the influence of the doping level at different emission wavelengths for both electron and hole systems. Figure 5(a) shows the dependence of the diffusion length on the average majority carrier (electron) concentration in the QW for different doping densities (different symbols). Due to the modulation doping effect, whereby the activated carriers all transfer into the QW, the carrier density in the QW (3 nm) is much larger than the dopant density in the two 10-nm QB regions. The results show that the diffusion length decreases as the doping density increases. As already discussed, the ionized dopants screen the field that results from fixed polarization charges (nominally at the QB/QW interface) bending in the QW (i.e., the QCSE), [START_REF] Onuma | Quantum-confined stark effects in the m-plane In 0[END_REF] and the carriers injected into the QW further screen the QCSE. The reduction in the QCSE will improve the electron and hole overlap and increase the recombination rate. Hence, the carriers will recombine more easily before they diffuse out. A more interesting result can be found in the p-i-p structures, where electrons are the minority carriers, as shown in Fig. 5(b). The electron diffusion length depends on the hole density in the QW in LEDs with different wavelengths. They all exhibit a similar trend, except that the UVC QW does not reach large carrier We now look at the different factors affecting the diffusion length when including the effect of alloy fluctuations. We simulate blue and green LED structures with 15% and 22% average In compositions. Due to the limited size of the simulation domain, we explore cases with diffusion lengths of less than 1 µm to avoid any boundary effects. We set the doping concentration to be 2 × 10 19 cm -3 in the QB. Figure 6(a) shows that, as the injected local power density increases, the electron diffusion length increases slightly. At very high excitation powers, the excess carriers cannot fully recombine locally, and so the diffusion current increases. However, this effect is not significant for the holes, possibly because of their larger effective mass. Figures 6(b)-6(d) illustrate the influence of the nonradiative lifetime 37 and carrier mobility. 5,[START_REF] Sze | Physics of semiconductor devices[END_REF] Let us examine the impact of the NR lifetime and mobility in a simple model. We set the excitation power density to 10 W/cm 2 . From the diffusion equation, the diffusion length L is obtained as 39

L = √ Dτ, (12)
and Einstein's equation relates D to µ as where D is the minority carrier diffusion coefficient, τ the minority carrier lifetime, k B is the Boltzmann constant, T is the temperature, and µ is the minority carrier mobility. The diffusion length is proportional to the square root of the mobility and nonradiative lifetime, √ µτ. Figure 6(b) shows the changes in the nonradiative lifetime. With a shorter lifetime, the diffusion length decreases as the lifetime decreases. However, τ is not only limited by the nonradiative lifetime, but also the radiative recombination and Auger recombination. When the nonradiative lifetime is longer than 10 -7 s, the increase in diffusion length becomes saturated; this is because the radiative recombination becomes the dominant factor since B 0 is fixed at 6×10 -11 cm 3 /s. Figures 6(c) and 6(d) show the influence of the mobility parameters on the diffusion length, which broadly scales with √ µ. From the preceding discussion, we can conclude that the large differences in diffusion lengths observed in the literature can largely be attributed to differences in carrier mobility and doping density, and marginally to the excitation conditions.

D = k B T e µ, (13)

IV. CONCLUSIONS

In this study, we have found that alloy fluctuations effectively limit the diffusion length in nitride alloy QWs. The random composition fluctuations confine the carriers and increase the recombination rate. The mobility and nonradiative lifetime also affect the diffusion length. Higher doping concentrations increase the majority carrier, which screens the QCSE and enhances the recombination rate, thus dimin-ishing the diffusion length. Finally, a higher local excitation power density causes the electron diffusion length to increase slightly, with the effect being even smaller in the hole case. Our results clarify the major limiting factors on the diffusion length, and differences in diffusion lengths measurements as due to different materials properties. For µLED applications or devices with a large density of defects, a shorter diffusion length is needed, and so designs with large fluctuating potentials and small QCSEs, such as semipolar structures, might be a good choice whenever possible. Sticking to c-plane structures, a good solution would be the use of field screening by fixed charge planes on both sides of the active layer as demonstrated by Young [START_REF] Young | Germanium doping of GaN by metalorganic chemical vapor deposition for polarization screening applications[END_REF] . Our more recent experimental and theoretical work by Y. C. Chow et al. 41 points to improvements in radiative rates due to field screening around 10 fold depending on the well width, corresponding to smaller diffusion lengths by a factor 3.

  FIG. 1. (a) Simulation structure, with dashed line representing the virtual structure. The electrons and holes are generated at the red area with a width of 10 nm. (b), (c) Indium composition maps of single QW structures for blue and green LEDs with random alloy fluctuations. (d) Single QW structure for UVC. The red arrow marks the respective average composition in the QW to the color bar. (e)-(g) Side-views of the calculated potential E c for blue, green, and UVC structures at x = 0, respectively. The barrier doping of this case is 2.0 × 10 19 cm -3 . White dashed lines separate the QW from the QB.

  FIG. 2. (a) Simulation flowchart. First step constructs the LED structure, then the alloy fluctuation model is used in the calculation. Finally, all parameters are input to the 3D-DDCC solver to obtain the results. The average indium composition is 15%. (b), (c) Indium maps with and without alloy fluctuations, respectively.

  FIG. 3. Blue LED for n-i-n structure. (See Fig. 2)The QW in-plane view is at z = 11nm, as recombination at the 11nm plane is stronger than in the middle of the QW at 11.5nm.The QW is in the 10nm to 13nm region. (a) Radiative recombination and (b) minority carrier density (holes). (c) Band diagram affected by alloy fluctuation [along the red line in (a)], (d), (e) averaged radiative recombination and hole density along the x-direction, respectively.

  FIG. 4. (a), (b) Averaged radiative recombination rate distribution along the lateral direction for the cases with and without random alloy fluctuations, respectively. (c) Extracted diffusion length with hole (minority) diffusion in the n-i-n structure. (d) Extracted diffusion length with electron (minority) diffusion in the p-i-p structure. The black line indicates the case with random alloy fluctuations, the blue line indicates the case without fluctuations, the red line indicates the case without polarization, and the green line indicates the case without fluctuations and without the polarization effect. (e)-(g) Electron and hole overlap for the three different situations using 1D-DDCC 34 simulations in the n-i-n structure.

26 FIG. 6 .

 266 FIG. 6. Comparison of different factors affecting the carrier diffusion length at a doping concentration of 2×10 19 cm -3 in the QB for the blue and green structures. (a) Comparison of diffusion length versus local excitation density (generation rate) in blue LEDs. (b) Influence of nonradiative lifetime on diffusion length. In green and blue n-i-n structures, the diffusion length is almost unchanged. (c), (d) Influence of mobility on diffusion length.

FIG. 7 .

 7 FIG. 7. Comparison of minority diffusion lengths with and without fluctuations, without polarization effect, and with different doping concentrations in the QB. (a), (b) Green structure, (c), (d) UVC structure. The black line indicates the alloy fluctuation case, the red line indicates the case without polarization, the blue line indicates the case without fluctuations, and the green line indicates the case without fluctuations and without polarization.

TABLE I .

 I Default parameters used in the modeling.

		Blue	Green	UVC
	Generation rate (1/cm 3 s)	1×10 26	1×10 26	1×10 26
	Electron mobility (cm 2 /Vs)	30	30	30
	Hole mobility (cm 2 /Vs)	5	5	5
	τ n , τ p (s)	5×10 -8	5×10 -8	5×10 -8
	Average composition in QW	15%	22%	40%
	Average composition in QB	0%	0%	60%
	Auger coefficient (cm 6 /s) B 0 (cm 3 /s)	6×10 -31 2×10 -11	6×10 -31 2×10 -11	2×10 -31 2×10 -11

TABLE II .

 II Activation energies used in the modeling.

		GaN	Al 0.6 Ga 0.4 N
	Acceptor ionization energies (meV)	170	432
	Donor ionization energies (meV)	25	76

TABLE III .

 III Materials parameters used in the modeling.

				0	In%	0.23	0	In%	0.29	0	Al%	0.72
				potential Ec	potential Ec	potential Ec
				-0.5		0.34 -0.5		0.52	-0.5	0.45
		GaN	InN	AlN
	Permittivity (ε r )	10.4	15.3	10.31
	Effective mass (m * e /m 0 ) Effective mass (m * h /m 0 )	0.21 1.87	0.07 1.61	0.32 2.68

  : when increasing the doping concentration, the majority carrier density does not increase much. The major reason for this is the high activation energy of the Mg dopant in the AlGaN layer. The activation energy of Mg acceptors for GaN is 170 meV, but is a much higher 432 meV for the Al 0.60 Ga 0.40 N layer. This leads to a low density of ionized acceptors in the UVC QW and a smaller carrier density in the QW. At all wavelengths, the majority carrier density in the QW appears to play a major role in limiting the diffusion length.

	600 500 400 300 200 Diffusion length (nm) 100	n-i-n	(Hole is minority carrier) 5×10 18 1×10 19 2×10 19 4×10 19 n-type doping	Diffusion length (nm)	3000 2500 2000 1500 1000 500	p-i-p	(Electron is minority carrier) 5×10 18 1×10 19 2×10 19 4×10 19 p-type doping
	Majority carrier density in 1×10 20 3×10 20		Majority carrier density in 1×10 20 3×10 20
	(a)	QW (cm -3 )	(b)		QW (cm -3 )
	FIG. 5. Comparison of minority carrier diffusion lengths for differ-
	ent doping concentrations in the QB for different structures. (a), (b)
	Diffusion lengths for holes and electrons versus the majority carrier
	density in QWs, respectively. The indium compositions for green and
	blue QWs are 22% and 15%, respectively. The average Al compo-
	sition of the UVC LED is 40% and 60% for the QW and QB, re-
	spectively. The generation rate representing light excitation is 10 26
	cm -3 s -1 .
	B. Impact of excitation power, mobility, and NR lifetime on
	diffusion length

densities

ACKNOWLEDGMENTS

This work was supported by the Ministry of Science and Technology, Taiwan (Grant Nos. MOST 108-2628-E-002-010-MY3 and MOST 110-2923-E-002-002). The work at UCSB was supported by the National Science Foundation (Grant No. DMS-1839077), and by grants from the Simons Foundation (Grant Nos. 601952 to J.S. and 601954 to C.W.). This work was also supported by the TECCLON project, grant ANR-20-CE05-0037-01 of the French Agence Nationale de la Recherche (ANR). The data used in this paper can be requested by contacting the corresponding author. The modeling tool was developed by our laboratory, and more detailed information can be found on our website (http://yrwu-wk.ee.ntu.edu.tw/).

Appendix A

Figure 7 shows the diffusion length of green and UVC LEDs affected by alloy fluctuations, without fluctuations, and without the polarization effect. The trend is similar to that of blue LEDs when increasing the doping concentration, leading to shorter diffusion lengths. (d) QB doping concentration (cm -3 ) Diffusion length(nm)