Rémy Sun

Diane Lingrand

Frédéric Precioso

Exploring the Road Graph in Trajectory Forecasting for Autonomous Driving

 dataset, with additional experiments on the LaneGCN [13] framework and Argoverse 1. 1 [2] dataset.

Introduction

The last decade has seen Deep Learning revolutionize how we approach fundamental problems like Computer Vision [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] and Natural Language Processing [START_REF] Vaswani | Attention is all you need[END_REF]. As such, it is perhaps no surprise that neural architectures are now spearheading a new generation of techniques that will directly impact our daily lives (e.g. Chatbots [START_REF] Chowdhery | PalM: Scaling language modeling with pathways[END_REF], Fraud detection systems [START_REF] Roy | Deep learning detecting fraud in credit card transactions[END_REF] and Autonomous Vehicles [START_REF] Kuutti | A survey of deep learning applications to autonomous vehicle control[END_REF]). Some, like Autonomous Vehicles, are even now motivating new inquiries into difficult questions [START_REF] Kuutti | A survey of deep learning applications to autonomous vehicle control[END_REF][START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF].

One of the core issues of Autonomous Vehicles, trajectory forecasting [START_REF] Salzmann | Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data[END_REF][START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF], has recently emerged as a difficult and complex task for neural networks. At its core, this task aims to forecast possible trajectories of pedestrians or vehicles. In its most common iterations, it takes as inputs both bounding boxes of active agents (pedestrians, cars, bikes, ...) pre-extracted from sensor data and known map information about the roads [START_REF] Kuutti | A survey of deep learning applications to autonomous vehicle control[END_REF].

Interestingly, trajectory forecasting techniques have proven heavily reliant on HDMaps [START_REF] Liu | Vectormapnet: End-to-end vectorized hd map learning[END_REF] that encode knowl-Figure 1. Investigating the Road Graph. We propose to investigate 3 aspects of the Road Graph used in trajectory forecasting: its resolution, how it can correct predictions, and how persistent knowledge can be embedded in it.

edge about the network of roads, signs, and objects, the agents might interact with. HDMaps are notable both due to the high complexity of the information they encode, and due to the fact they require encoding prior human knowledge into a neural system which is a notoriously difficult problem. Therefore, the question of how to encode this information has been the focus of much debate in the literature [START_REF] Salzmann | Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data[END_REF][START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF][START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF].

Graph (or vectorial) representations of the road network have slowly emerged as the consensus solution over the last few years [START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF][START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF]. Contrarily to previous solutions like rasterized representations [START_REF] Salzmann | Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data[END_REF], road graphs depicting the relation between the various lanes and agents offer a flexible and powerful input that can be directly fed to a graph neural network [START_REF] Veličković | Graph Attention Networks[END_REF]. Better, the road graph structure can be leveraged by teaching the network to traverse the graph before yielding a continuous trajectory prediction [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF].

It is therefore surprising that the particulars of this road graph have seen very little study. While recent works propose general heuristics regarding the optimal spatial resolution used by the graph [START_REF] Park | Leveraging future relationship reasoning for vehicle trajectory prediction (frm)[END_REF] or the information to encode in the nodes [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF], we can find no specific work directly exploring these questions. Indeed, most existing work merely takes advantage of the graph structure to acquire good features and instead focuses on feature extraction [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF]14] or decoding schemes [10].

In this paper, we explore 3 possible ways this road graph can interact with trajectory forecasting solutions: the impact of spatial resolution, the graph's relation to trajectory outputs and how persistent or static knowledge can be embedded into the graph. The first is a straightforward exploration of how the number of nodes in the road graph directly impacts the model's understanding of distances and of the world. The second aims to study how relating the output to the graph can allow for even better trajectories. The last point explores how the model can acquire "persistent" knowledge about the characteristics of cities and lanes.

Contributions

We propose these 4 contributions:

• We investigate the spatial resolution of the road graph in relation to trajectory forecasting.

• We check how examining output predictions with regards to the graph can help further improve the final trajectories given by models.

• We explore a few ways to embed more persistent knowledge into the graph that can recur across multiple scenarios (e.g. knowledge on a city).

• We evaluate our propositions experimentally on two well-established graph based frameworks (PGP [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF], LAformer [14]) over the nuScenes [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF] dataset, with additional experiments on the LaneGCN [START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF] framework and the Argoverse 1.1 [START_REF] Chang | Argoverse: 3d tracking and forecasting with rich maps[END_REF] dataset.

Context

Before elaborating in more details regarding the construction of the Road Graph in Sec. 3, we start by offering here some brief context on trajectory forecasting and HDMap representations.

Trajectory forecasting Trajectory forecasting [START_REF] Salzmann | Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data[END_REF][START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF] -ie, the task of predicting possible trajectories for a given agent (pedestrian or vehicle) -is in itself a fairly new problem for autonomous driving with the recent push for fully automated vehicles [9]. As opposed to more traditional autonomous driving problems like planning [START_REF] Schwarting | Planning and decision-making for autonomous vehicles[END_REF], forecasting is inherently a multimodal problem that requires outputing a vast spectrum of possible solutions to model most likely possibilities (see Fig. 2). The exact solutions developed are beyond the scope of this paper but they all require combining traditional perceptual inputs (detected agent trajectories [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF] or raw records [START_REF] Gu | Vip3d: End-to-end visual trajectory prediction via 3d agent queries[END_REF]) with symbolic knowledge bases (HDMaps).

Map representation

In order to help navigate the complexities of the real world, known information about the road network is typically embedded in rich HDMaps provided to the forecasting systems [9]. This map information is however difficult to feed directly to a deep architecture. It must therefore be processed into a suitable representation. While converting the map to an image -a process known as rasterization [START_REF] Salzmann | Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data[END_REF] -is a convenient solution that has been used by early solutions, such representations lose much in translation [START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF].

As such, a trend has emerged to modelize the map as a set of vectorial (or graphical) entities with different relationships to each other [START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF][START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF]. VectorNet [START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF] first introduced this idea in 2020 by modeling each lane in a map as a node and learning an embedding of the map through an attention mechanism between the nodes. This representation has since been refined into a more detailed road graph that mimics the road network by LaneGCN [START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF].

The information contained in HDMaps has proven crucial to the functioning of modern trajectory forecasting solutions. Indeed, a number of graph based methods like PGP [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF] and LAformer [14] explicitly reason on the road graph and therefore cannot be used without map information. Even for a non graph based method (e.g. mmTransformer [START_REF] Liu | Multimodal motion prediction with stacked transformers[END_REF]), [START_REF] Liu | Vectormapnet: End-to-end vectorized hd map learning[END_REF] shows a 10% drop in performance for most commonly used metrics. There is therefore a pressing need to better understand how Road Graphs leverage HDMaps information to improve in current Deep trajectory forecasting systems.

Meet the road graph

As this study focuses on exploring the impact of the Road Graph on trajectory forecasting methods, we provide here a brief introduction both of the graph proper (Sec. 3.1) and of how it is leveraged in current methods (Sec. 3.2).

Building a road graph

The Road Graph is meant to encode all the complex information encoded in HDMaps in a way readily understandable to a neural network. As such, quite a bit of preprocessing must go into its construction and a number of choices must be made by the implementer as to what needs to be encoded and how. For simplicity's sake, we mostly discuss here the road graph as it is proposed in methods that rely heavily on it like PGP [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF].

Notations

In the following, we use common notations in the literature that we introduce now. We refer to lanes centerlines -ie, the line at the center of a drivable lane -by a sequence of vectors C i = {C 0 i , . . . , C l i }. We denote agents by sequences of descriptive vectors A i = {A -T 0 i , . . . , A 0 i } describing its past history for some time horizon T 0 . We characterize the road graph G = (V, E) by its set of nodes V and its the set of edges in the graph E.

At a very high level, the road graph encodes the centerlines C as nodes V , and includes in E edges that indicate whether it is possible to navigate from one node to the other according to the HDMap. Agents A can be added to the nodes of this graph and linked to lane nodes they are close to spatially. This exact representation is used in some works of the literature like the seminal VectorNet [START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF], but it lacks in detail and does not quite reflect the topology of the problem [START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF].

In practice, many modern methods discretize lane centerlines C into segments such that a centerline C i = { Ĉi,0 , . . . , Ĉi,s } is comprised of s sub-centerline segments Ĉi,j = { Ĉ0 i,j , . . . , Ĉl i,j } (see Fig. 3). From this, we can extract nodes -one node per segment -such that a node in the road graph denotes a spatial location in the world, stretching across a few meters on a lane. These segments Ĉi,j typically get represented as a sequence of vectors (to then be encoded by some neural layer like a GRU [4]), with each vector representing a point on the segment. What each of those "point" vector Ĉk i,j contain depends on the method, but a typical implementation encodes spatial coordinates x, y, yaw angle θ as well as some semantic flags indicating whether a crosswalk or stop sign is near.

We then connect lane segment nodes V by edges to reflect the road network on the HDMap. Typically, the set of edge E contains both edges connecting successive segments in a given lane ("E succ ") but also connections between segment nodes belonging to different lanes ("E prox "). The generally adopted heuristic is to connect two nodes if a vehicle could legally move from one node to the other. Concretely, this could mean connecting two segment nodes that are within a certain radius of each other spatially and have similar yaw orientations (to avoid allowing impossible Uturns).

The end result is shown on Fig. 3, where the original network of HDMap lane centerlines if faithfully represented by the constructed road graph. This graph provides a lightweight representation that can easily be processed by standard neural architectures like GNNs. Contrarily to rasterized representations, it allows for a clear and unambiguous depictions of complex situations where lanes intersect and overlap (e.g the blue and violet lanes do not obscure other lanes). As such, it has emerged over the last few years as the de facto solution for HDMap representations in Autonomous Driving trajectory forecasting.

While this fundamental definition of the Road Graph does not include agents, agents A of course play a role in most methods and can be regarded as additional nodes in a heterogeneous graph. In such cases, these additional agent nodes are connected to segment nodes that come within a certain radius of the agent node. This can be explicitly formulated as such [14] or processed as a separate type of Figure 4. Graph traversal for trajectory forecasting. After encoding a meaningful road graph, methods like PGP [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF] can sample likely traversal on the graph. These traversals can then be converted to real world trajectories.

graph relation [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF].

Using a road graph in practice

To better understand the utility of the now defined Road Graph representation, we quickly illustrate here how it can be leveraged by forecasting solutions in the literature.

Baseline use: GNN features At the very least, the graph structure of the Road Graph can be processed by Graph Neural Networks [START_REF] Veličković | Graph Attention Networks[END_REF] to yield relevant features [START_REF] Gao | VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation[END_REF][START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF]. These features can then be used to enrich agent representations or be directly fed to a trajectory generator. This single improvement has led to significant improvements in performance over previous rasterized representations [START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF].

Reasoning on the graph: PGP The Road Graph can also be leveraged further by directly reasoning on the graph to guide the target agent's movement over the lanes. A good example of this is provided by the PGP framework [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF] which provides a solid foundation still used to this day by very recent works [5,14]. As such, we provide in Fig. 4 and the rest of this section a brief summary of the graphtraversal mechanism in the PGP framework.

The PGP framework can be broken down to three main components as shown on Fig. 4: graph encoding, graph traversal and trajectory decoding. The first step -graph encoding -simply refines the feature representation of the segment nodes through a combination of GRU encoders and GNN layers. The last step roughly translates a traversal on the road graph into a complete trajectory in real world coordinates.

Direct reasoning on the graph takes place through the second step of PGP: graph traversal. For each node on the graph, the model learns to predict transition probabilities to adjacent nodes. Given these transition probabilities, PGP samples possible graph traversal. These traversals can then be translated to a trajectory in the third step. As such, the method explicitly takes advantage of the road graph's structure to learn plausible predictions. We therefore propose in this paper to further study how changes to this road graph can directly impact results.

Contribution: Exploring the Road Graph

We investigate in this study three possible ways the road graph can directly influence the predicted trajectory. First, we look into how the graph resolution influences the prediction in Sec. 4.1. Afterwards, we attempt to contextualize the proposed trajectories with the road graph to improve performances in Sec. 4.2. Finally, we discuss how persistent knowledge could be embedded to further enhance the Road Graph in Sec. 4.3.

Spatial resolution in the Road Graph

The most natural questions when considering the road graph pertain to its construction: How much do the lengths of the segment nodes and connecting edges matter ? At a very high level, this problem is analogous to resolution in images. The shorter the distances involved, the more detailed the representation is. This comes with both advantages and disadvantages. On the one hand, performance in Computer Vision tends to improve with higher resolutions. On the other, this comes with both higher dimensionality and more intensive computations [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF]. While this is sometimes considered in ablative studies [START_REF] Pollak Zuckerman | Across scales & across dimensions: Temporal super-resolution using deep internal learning[END_REF], we propose specifically investigating the question as formulated in Fig. 5.

This analogy to Computer Vision however fails to account for a particular aspect of Autonomous Driving: Vehicle Movement. Indeed, the end goal is to describe the movement of vehicles moving at varying speeds. In principle, long road segments are perfectly appropriate to describe the If resolution is to be linked to speed, it introduces the difficult question of how models can accommodate graphs of various resolutions. Changing the graph resolution between one image and the other leads to representing the world very differently within a dataset, confusing the model (as is well known in Computer Vision [START_REF] Touvron | Fixing the train-test resolution discrepancy[END_REF]). Technically, the increasingly distant coordinates of segment nodes should indicate the changes to the model but it is not clear this would be enough. This should therefore be accounted for when designing a model to deal with such situations.

Relating predictions back to the Road Graph

A natural second question to consider relates to how the final continuous predictions relate to the Road Graph. A typical framework goes as follows: the world is discretized into a Road Graph, feature extraction or even reasoning takes place on the graph, and a continuous prediction is then output. Conspicuously, once we output a trajectory, no further use is made of the graph. We propose to look into this matter more closely here.

Interestingly, a number of work have recently taken to refining initial trajectory prediction by predicting corrections to be made to the original estimate [10]. Unfortunately, these attempts have until now been made largely independently from the Road Graph. While some works do incorporate information on the Road Graph [14], the role of the Road Graph in the refinement process is never focused on. As such, it is important to study this refinement mechanism further.

We propose to examine the importance of information on the Road Graph to the trajectory refinement process. Be-Figure 7. 3rd Exploration: Persistent node features. We study how "persistent" node features can be learned to characterize permanent objects like cities and lanes. yond simply verifying that adding Road Graph information to the process is beneficial, there is a wealth of unexplored questions regarding the type of Road Graph information that should be taken into account. As such, we explore the different types of information that can be embedded into the process like the features of the graph itself or information on the spatial structure tied to the graph (e.g. the coordinates of the nodes involved in a graph traversal).

Embedding persistent knowledge

Finally, we tackle the complex question of how persistent knowledge can be learned and embedded in the Road Graph. Can we learn that a road is slow? Fast? Can we learn how people drive in different parts of the world? This is a fairly under explored part of the literature as of yet, one which has significant long term ramifications for the future of autonomous driving.

A first simple consideration at this level is to tell the network which city the situation takes place in. Indeed, this is interesting as a number of driving rules are local to various places in the world. For instance, the nuScenes [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF] dataset collects scenarios from two cities: Singapore and Boston. Driving habits between the two cities differ significantly due to a number of considerations ranging from the typical width of the lanes to the driving directions (left in Singapore, right in Boston). To leverage this, we propose adding a simple embedding to nodes in the lane graph depending on the city of the scenario.

A more ambitious project would be to learn characteristics at the lane level: each lane has its own characteristics such as the different driver speeds and behavior. This can be due to a number of real world factors. For instance, within the same city, it is not rare to find both very narrow streets and wide boulevard lanes, with the resulting differences in driver behavior. We explore this line of thought by reprising the previous embedding mechanism. In other words, for each lane in a city, we create a unique embedding that gets added to any road graph node belonging to that lane. The goal behind this mechanism is to acquire unique signatures for each lane that can be continually updated as part of a flexible knowledge base in an autonomous driving system.

Experiments

We investigate here the three lines of inquiry discussed in Sec. 4. After briefly discussing the experimental setting, we provide our results and conclusions regarding the impact of graph resolution (Sec. 5.1), the relation between the predicted trajectories and the graph coordinates (Sec. 5.2), as well as our efforts to embed persistent knowledge in the road graph (Sec. 5.

3).

Setting To ensure the validity of our observations, we check the impact of the Road Graph across three graph based trajectory forecasting systems: PGP [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF], LAformer [14] and (to a lesser extent) LaneGCN [START_REF] Liang | Learning lane graph representa-tions for motion forecasting[END_REF]. These three methods cover different levels of graph involvement in forecasting. The PGP method is completely reliant on the Road Graph to sample graph traversals, and is still used to this day as a backbone in some recent methods. LAformer is the current State of the Art solution on nuScenes (alongside FRM [START_REF] Park | Leveraging future relationship reasoning for vehicle trajectory prediction (frm)[END_REF]), and reasons on the graph to find the most likely lane segment used by the target vehicle. Finally, LaneGCN is a seminal work that solely uses the Road Graph to extract meaningful features. All experiments are run by modifying official code repositories 1 for the three methods studied.

We base our conclusions on the well established nuScenes [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF] dataset for the most part, with a complementary experiment on the Argoverse 1 [START_REF] Chang | Argoverse: 3d tracking and forecasting with rich maps[END_REF] dataset to shed further light on results in Sec. 5.3. The nuScenes dataset pro- As such, it has been a staple of trajectory forecasting efforts since its introduction in 2021. The dataset provides as inputs logs of the past two seconds of driving for each agent and HDMaps. The expected output is a set of predictions matching the next 6 seconds of the target vehicle trajectory.

In line with the literature, we choose to evaluate the Mi-nADE and Miss Rate metrics at K on the nuScenes val set. For the MinADE metric, given the K likeliest predicted trajectories, we compute each trajectory the Average Deviation Error to the ground truth ("ADE") and keep the best score out of the K predictions ("Min"). Similarly, the Miss Rate (at K) indicates whether at least one of the K likeliest trajectories remained within 2 meters of the Ground Truth. Following nuScenes convention, we show metrics at K=5 predictions, and K={1,6} for Argoverse 1.

Spatial resolution in the Road Graph

In accordance with the discussion conducted in Sec. 4.1, we explore how "resolution" (ie, the length of the lane segment nodes) impacts the performance of models. Our general experimental results are summarized in Tab. 1. Tab. 2 provides additional details to better examine the effect of changing resolution. Experiment We train models with different road graph resolutions (5, 10, 15 and 20 meters) and check the impact on the resulting performance. For PGP, we perform an additional experiment where we adapt the resolution (in a range between 5 to 15 meters) of the graph depending on the speed of the vehicle at the start of the prediction: slow target vehicles navigate high resolution graphs (short segments) and faster vehicles work with lower resolutions (longer segments). It should be noted that LAformer (resp. PGP) natively uses 5 (resp. 20) meter lane segments.

Method

Tab. 1 shows that lower resolutions (e.g. 20 meter segments) tend to perform worse. In particular, LAformer performs very poorly with 20 meter segments. Interestingly, the best performance seems to be obtained with 10 meter lane segment nodes which is not the resolution used by either of those methods natively. The slightly worse performance with 5 meter segments shows that increasing the resolution does not necessarily improve results, and that other factors must be considered.

To better understand the effect of resolutions, we check in Tab. 2 the minADE (at K=5) of PGP when predicting samples where the vehicle moves little (less than 75 meter long trajectory), moves moderately (less than 150 meter long trajectory, but more than 75) or moves a lot (more than 150 meter long trajectory). The resolution seems to have fairly little importance when predicting trajectories that span less than 150 meters. Indeed, trajectories longer than 150 meters account for most of the difference in performance between resolutions. In line with expectations, 5 meter segments struggle with long trajectories as they depict shorter spatial locations. This reasoning fails to explain why 20 meter segments perform worse with longer trajectories. A possible explanation might be that lower resolution induces more inaccuracies, and that these inaccuracies compound the longer the trajectory is.

Finally, our attempt to dynamically tie the resolution to initial vehicle speed produces very poor results across the board. This does not seem to simply be an issue of the method striking a compromise between the performance of high and low resolution models. Indeed, the performance over moderately long trajectories appears far worse than that obtained by any single resolution model. As such, this could be caused by the network having difficulty interpreting different network resolutions.

Relating predictions back to the Road Graph

As discussed in Sec. 4.2, we try to link predicted trajectories in "the real world" back to the structured graph representation to see how this affects performance. Our general experimental results are summarized in Tab. 3.

Experiments

We add a refinement head (3 layer CNN) to PGP, and reprise the native refinement module of LAformer. Refinement heads are fed as input the predicted scratch trajectories to refine along with some other features. These other features do not traditionally include graph information. We check two ways of injecting graph knowledge: an attention-based summary of relevant graph features, or the (x,y) coordinates of the samples graph traversal (in PGP).

As shown in Tab. 3, including graph knowledge of any kind significantly improves the MinADE. Simply adding a refinement head fails to improve performance for PGP (and might actually deteriorate it). This result stands in sharp contrast to the performance of both variants of the graph based refinement models we propose. Interestingly, introducing the (x,y) coordinates of the sampled graph traversal seems to yield even better results than using an attention based summary of graph features as is done natively in LAformer. This indicates that not all the spatial information contained in the graph gets translated to the predicted trajectory. Further work is therefore needed on the matter.

Embedding persistent knowledge

Building on the ideas developed in Sec. 4.3, attempt to inject some "persistent" knowledge in the graph. Ideally, this should provide us with a way to link many different scenarios that happen in the same city, district or even large avenue. Our general experimental results are summarized in Tab. 4 and Tab. 5.

Experiments We tackle two types of persistent knowledge here: city-wide knowledge and lane-wide knowl- edge. For the first, we conduct a preliminary experiment on nuScenes using the PGP backbone: we add city identifying binary features to the nodes of the road graph to indicate whether the scene takes place in Singapore or Boston.

As to the second, we catalogue all the known lanes in the Argoverse 1 dataset. For each lane segment node, we add a unique learnable embedding characterizing the lane from which it comes. These experiments on lane embeddings are performed on the Argoverse 1 dataset as the nuScenes dataset proved too small. Since the PGP framework translates poorly to the Argoverse 1 format, we use LAformer and LaneGCN as baseline methods.

Preliminary experiments on the city features seem to indicate some improvement as indicated by Tab. 5. Interestingly, this improvement only manifests when accounting for the influence of horizontal flip augmentation: city features need to be switched off when the sample is flipped. This seems to indicate the city features help the model learn whether one needs to drive on the left (Singapore) or on the right (Boston). When the horizontal augmentations is performed indiscriminately, the model learns to be invariant to this information and therefore the city features are useless.

Tab. 4 shows consistent improvements for both LaneGCN and LAformer on the Argoverse 1 dataset. Given the minimal cost of adding lane embeddings to graph nodes, it seems worthwhile to further explore the idea. Greater improvements could be obtained by better focusing which lane embeddings should be updated. As it stands, every lane embedding gets updated when training on a scenario even if a lane has nothing to do with the prediction of interest.

Interestingly, this improvement seems to require a large amount of data. Preliminary experiments on the nuScenes dataset -where a lane might appear in at most a few hundred scenarios -invariably lead to clear overfitting. From that, we can glean two conclusions: lane embeddings provide a lot of expressive power to models, and only well traveled roads should be considered for real-time updates of lane embeddings. These results show that adding persistent knowledge may improve MinADE in some cases and merits further study.

Conclusion

In this paper, we explore three questions regarding the use of road graphs in trajectory forecasting for autonomous vehicles. What is the importance of the graph resolution? Can we further improve predictions by relating them back to the road graph? How should we embed learnable "persistent" knowledge in the graph?

After formally introducing the road graph and its role for autonomous vehicle trajectory forecasting, we develop these three general questions and identify key points to elucidate in our experiments. Through experiments on the well-known nuScenes dataset and through 2 recent forecasting methods (PGP and LAformer), we manage to provide some insights on our questions: graph resolution seems to mostly affect predictions of longer trajectories, graph information (node coordinates in particular) is very valuable when refining trajectories, and it seems possible to learn some "persistent" node features that can be shared across multiple scenarios.

Through this exploratory study, we have striven to shed some light on the road graph that lies at the center of many modern trajectory forecasting methods. Nevertheless, we have focused on three specific facets of the issue without considering the many others that would affect model performance. The properties of the road graph remain to be explored. We hope that our work can lead to further investigation both of new questions and of the interrogations that remain on the ones we have studied.

Figure 2 .

 2 Figure 2. Trajectory forecasting takes complex inputs (both perceptual detections and symbolic knowledge) to predict possible trajectories for a given vehicle of interest.

Figure 3 .

 3 Figure 3. Construction of a typical Road Graph. We discretize lane centerlines into segments before converting the segments into nodes. The nodes are then connected to reflect the HDMap topology.

Figure 5 .

 5 Figure 5. 1st Exploration: Graph Resolution. We study the importance of the graph resolution on prediction results.

Figure 6 .

 6 Figure 6. 2nd Exploration: Trajectory correction. We further refine final predictions using the graph.

Table 1 .

 1 Influence of resolution on nuScenes. Results using PGP and LAformer over the nuScenes dataset illustrate how resolution affects model performance. Default settings:

	Method	PGP		LAformer
		MinADE MissRate MinADE MissRate
	20 meter lane segments *	1.27	0.51	1.26	0.41
	15 meter lane segments	1.26	0.51	1.20	0.37
	10 meter lane segments	1.25	0.52	1.19	0.37
	5 meter lane segments †	1.26	0.52	1.20	0.36
	Dynamic resolution (5 to 15 meter segments)	1.28	0.51	N/A	N/A

* PGP, † LAformer

Table 2 .

 2 Detailed influence of resolution on PGP over nuScenes MinADE for different types of vehicle motions. p is the trajectory length in meters. Default setting:

	1 PGP: github.com/nachiket92/PGP
	LAformer: github.com/mengmengliu1998/LAformer
	LaneGCN: github.com/uber-research/LaneGCN

*

vides high quality annotation made by humans of 5 hours of driving footage (2Hz) in 2 cities (Singapore and Boston).

Table 3 .

 3 Influence of graph knowledge on trajectory refinement. Results using PGP and LAformer on the nuScenes dataset indicate clear performance improvement from referring to the road graph.

	PGP	LAformer
	MinADE Miss Rate MinADE Miss Rate

Table 4 .

 4 Influence of lane embeddings. Results of LaneGCN and LAformer on the Argoverse 1 dataset show consistent benefits of lane embeddings.

			LaneGCN			LAformer
	Method		K=1	K=6		K=1		K=6
		MinADE Miss Rate MinADE Miss Rate MinADE Miss Rate MinADE Miss Rate
	No persistent feature	1.35	0.49	0.72	0.10	1.23	0.46	0.65	0.09
	Lane embedding	1.34	0.49	0.71	0.10	1.22	0.45	0.65	0.09
	Method		PGP
			MinADE Miss Rate
	No persistent feature		1.27	0.51
	City features		1.28	0.52
	City features w/ proper flip	1.26	0.51

Table 5 .

 5 Influence of city features. Results on the nuScenes Dataset using the PGP framework indicate city features improve performance.

Acknowledgements This work was realized by the Mul-tiTrans project funded by the Agence Nationale de la Recherche under grant reference ANR-21-CE23-0032. The authors are grateful to the OPAL infrastructure from Université Côte d'Azur for providing resources and support.


Arthur Floquet

Member, IEEE Sayantan Dutta

Emmanuel Soubies

Member, IEEE Duong-Hung Pham

Senior Member, IEEE Denis Kouame

Automatic Tuning of Denoising Algorithms Parameters without Ground Truth

Keywords: Bilevel optimization, Denoising, Hyperparameter tuning

Denoising is omnipresent in image processing. It is usually addressed with algorithms relying on a set of hyperparameters that control the quality of the recovered image. Manual tuning of those parameters can be a daunting task, which calls for the development of automatic tuning methods. Given a denoising algorithm, the best set of parameters is the one that minimizes the error between denoised and ground-truth images. Clearly, this ideal approach is unrealistic, as the ground-truth images are unknown in practice. In this work, we propose unsupervised cost functions -i.e., that only require the noisy image -that allow us to reach this ideal gold standard performance. Specifically, the proposed approach makes it possible to obtain an average PSNR output within less than 1% of the best achievable PSNR.

I. INTRODUCTION

N OISE is inherent to any imaging device. It comes from a variety of sources and is modeled in a variety of ways. When considering additive noise, the corrupted measurements y ∈ R N follow the model :

y = x + n, (1)
with x ∈ R N the clean image we wish to recover, and n ∈ R N the noise. Many denoising algorithms, denoted A θ , have been proposed to address this task and provide estimates given by A θ (y), e.g, [START_REF] Dabov | Image denoising by sparse 3-D transform-domain collaborative filtering[END_REF]- [START_REF] Dutta | A novel image denoising algorithm using concepts of quantum many-body theory[END_REF], to cite few. The quality of these estimates depends on the chosen parameters θ ∈ R d . In practice, however, manual tuning of these parameters is far from being trivial, even for low numbers of parameters such as two or three. As such, finding ways to automatically tune these parameters is of major importance and constitutes an active area of research. Most existing approaches use a mapping Θ λ : y → θ, itself parameterized by λ ∈ R d ′ , that maps an image and/or its features (e.g., noise level, noise type, image dynamic, image content) to a set of parameters θ. The best λ, i.e. λ * , is found by minimizing the expectancy E of a discrepancy measure L(•, •) between the denoised images and ground truth images:

λ * := arg min λ∈R d ′ E (x,y) [L(A Θ λ (y) (y), x)]. (2)
There are several possibilities to define Θ λ with increasing degree of sophistication:

This work was supported by the CIMI Excellence Laboratory, ANR grant ANR-11-LABX-0040 within the French State Programme "Investissements d'Avenir" A. Floquet, E. Soubies, D.-H. Pham, and D. Kouamé are with the IRIT Laboratory, Université de Toulouse, and CNRS, Toulouse 31400, France. Email: firstname.lastname@irit.fr. (Corresponding author : Arthur Floquet) S. Dutta is with the Department of Radiology, Weill Cornell Medicine, New York, NY 10022, USA. Email: sdu4004@med.cornell.edu.

• Θ λ is a constant mapping (i.e., Θ λ (y) = θ ∈ R d ). Solving (2) then consists in finding a fixed set of parameters θ such that, on average, the estimates are good, e.g, [START_REF] Calatroni | Bilevel approaches for learning of variational imaging models[END_REF]. • Θ λ is defined using features extracted from y. For instance, Θ λ (y) = λ 1 +λ 2 σ(y) where σ(y) is an estimator of the noise variance. Here, solving (2) allows us to learn a mapping Θ λ that adjusts algorithm parameters according to a known and predefined model, e.g, [START_REF] Nguyen | MAP-informed unrolled algorithms for hyper-parameter estimation[END_REF]. • Θ λ is a neural network. In this case, solving (2) allows us to learn a mapping Θ λ that adjusts algorithm parameters according to an unknown model, e.g, [START_REF] Afkham | Learning regularization parameters of inverse problems via deep neural networks[END_REF], [START_REF] Kofler | Learning regularization parametermaps for variational image reconstruction using deep neural networks and algorithm unrolling[END_REF]. This greater flexibility comes at the price of a larger number of parameters λ to learn. Typically, all these methods require a dataset and work in a supervised way. Since this is not always feasible, unsupervised alternatives have been developed [START_REF] Lehtinen | Noise2Noise: Learning image restoration without clean data[END_REF]- [START_REF] Pang | Recorruptedto-Recorrupted: Unsupervised deep learning for image denoising[END_REF]. The main idea behind these approaches is to define an unsupervised loss (i.e., that does not depend on x) achieving the same minimizers as the supervised counterpart. Nevertheless, constructing Θ λ and finding λ remains challenging.

To discard the need of defining and training a mapping Θ λ on a dataset, one could directly fit θ on individual images y. The ideal estimate would be:

x * := A θ * (y) with θ * := arg min θ∈R d C * y (θ) , C * y (θ) := L(A θ (y), x). (3)
Yet, this formulation is impractical as it requires knowing x to obtain x * . In the following, x * will be our gold standard, that is, the best estimate we can expect for a given image y and algorithm A θ . For Gaussian noise, methods such as the famous generalized cross-validation (GCV) [START_REF] Golub | Generalized crossvalidation as a method for choosing a good ridge parameter[END_REF] and its variants or the Stein's unbiased risk estimate (SURE) [START_REF] Stein | Estimation of the mean of a multivariate normal distribution[END_REF], which depends only on the noisy data, can be used in place of the true mean-squared error (MSE). The SURE optimization is, however, challenging in the general case and requires the use of approximations [START_REF] Deledalle | Stein unbiased gradient estimator of the risk (sugar) for multiple parameter selection[END_REF] or Monte Carlo approaches [START_REF] Ramani | Monte-Carlo Sure: A Black-Box Optimization of Regularization Parameters for General Denoising Algorithms[END_REF]. It is noteworthy to mention that other metrics that do not require the reference image have also been proposed [START_REF] Zhu | Automatic Parameter Selection for Denoising Algorithms Using a No-Reference Measure of Image Content[END_REF], [START_REF] Kong | No-reference image quality assessment for image auto-denoising[END_REF].

Here, inspired by [START_REF] Lehtinen | Noise2Noise: Learning image restoration without clean data[END_REF]- [START_REF] Pang | Recorruptedto-Recorrupted: Unsupervised deep learning for image denoising[END_REF], we propose alternative unsupervised cost functions C and inference schemes I such that:

x * ≈ x := I( θ, y) with θ := arg min

θ∈R d C y (θ). (4)
Let us emphasize that the inference scheme in (4) is not directly A θ(y). Indeed, as detailed in Section II, the proposed unsupervised cost functions may require adapting the inference scheme. As such, we will systematically specify both the cost function and the inference scheme. Remark 1: The use of the cost function C * could be avoided in (3) and replaced with L directly. We write (3) this way to be consistent with the following unsupervised cost functions C, which don't reduce to simple discrepancy measures between two images. Moreover, we will use • * and • for supervised and unsupervised objects, respectively.

Paper Outline

The paper is structured as follows. The proposed method is detailed in Section II. Then, in Section III, we deploy our approach to tune the parameters of a recently published denoiser [START_REF] Dutta | A novel image denoising algorithm using concepts of quantum many-body theory[END_REF] that can handle a variety of noise types. Finally, discussions and conclusions are provided in Section IV.

II. METHOD

Let us start this section by restating the key difference between our approach in Eq. ( 4) and the more standard one in Eq. ( 2) (with its unsupervised counterparts [START_REF] Lehtinen | Noise2Noise: Learning image restoration without clean data[END_REF]- [START_REF] Pang | Recorruptedto-Recorrupted: Unsupervised deep learning for image denoising[END_REF]). Notably, there is no training phase in (4). We don't need to create and train a mapping Θ λ over a dataset that will then be used to infer estimates on new data. Instead, our inference is done by solving (4) directly for individual images y.

We also emphasize that our aim is not to build a new denoiser, but an automatic way to select hyperparameters of a given denoiser, using only the input noisy image. For illustration purposes, we use the DeQuIP algorithm [START_REF] Dutta | A novel image denoising algorithm using concepts of quantum many-body theory[END_REF] described in Section II-C.

A. The Cost Functions and Inference Schemes Definition

We drew our inspiration from existing unsupervised learning methods. Instead of training a denoising neural network f of parameters ω, say f ω , on noisy-clean pairs (y, x), Noise2Noise (N2N) [START_REF] Lehtinen | Noise2Noise: Learning image restoration without clean data[END_REF] proposes to train it on pairs of noisy images (y, y ′ ) with y and y ′ two noisy versions of the same clean image x. Lehtinen et al. [START_REF] Lehtinen | Noise2Noise: Learning image restoration without clean data[END_REF] showed that for several noise types, L can be chosen accordingly so that:

arg min ω E (y,y ′ ) L(f ω (y), y ′ ) = arg min ω E (x,y) L(f ω (y), x). (5)
For example, if y, y ′ are two corrupted versions of x with independent additive zero-mean noise (i.e., y

= x + n, y ′ = x + n ′ with E[n] = E[n ′ ] = 0),
letting L be the quadratic error leads to [START_REF] Nguyen | MAP-informed unrolled algorithms for hyper-parameter estimation[END_REF]:

ω := arg min ω E (y,y ′ ) ∥f ω (y) -y ′ ∥ 2 2 = arg min ω E (y,x,n ′ ) ∥f ω (y) -x -n ′ ∥ 2 2 = arg min ω E (y,x,n ′ ) ∥f ω (y) -x∥ 2 2 -2 ⟨f ω (y) -x, n ′ ⟩ = arg min ω E (y,x) ∥f ω (y) -x∥ 2 2 := ω * , (6)
where the expectation of the dot product cancels as E[n ′ ] = 0, and n and n ′ (i.e., y and n ′ ) are independent.

In practice, training is performed using a finite dataset Ω I,J := {y j i = x i + n j i } I,J i,j= [START_REF] Dabov | Image denoising by sparse 3-D transform-domain collaborative filtering[END_REF][START_REF] Dabov | Image denoising by sparse 3-D transform-domain collaborative filtering[END_REF] . This means that for a noiseless image x i , J noisy versions are available, with additive noises n j i . As such, the true loss in ( 5) is replaced by an empirical one. This results in a solution ω ≈ ω * whose quality depends on I, J:

• When both I and J are large, the empirical loss yields a good approximation of the true one. • When I is small (e.g., I = 1), but J is large, we can expect f ω to perform well on the training set, but not to generalize well. It will effectively learn to output the mean of the y j 1 , which is x 1 . • When J is small (e.g., J = 2, with different n 1 i , n 2 i for each i), f ω could overfit the dataset Ω I,J to the point that we systematically obtain f ω (y

1 i ) = y 2 i , f ω (y 2 i ) = y 1 i , ∀i.
Yet, a large I (e.g., I = 1000) seems to be sufficient in preventing overfitting as demonstrated by Lehtinen et al, [START_REF] Lehtinen | Noise2Noise: Learning image restoration without clean data[END_REF]. Other factors can be exploited to avoid overfitting when limited data are available. These include network architecture, image type, noise type, or a low number of parameters relative to image size, card(ω)/P , with P the number of image pixels [START_REF] Ying | An Overview of Overfitting and its Solutions[END_REF]. For instance, it has been shown that U-Net-like networks have a hard time recreating noise [START_REF] Ulyanov | Deep Image Prior[END_REF].

• When both I and J are small (e.g., I = 1, J = 2), if not carefully designed, f ω would simply learn to map y 1 1 to y 2 1 , and vice-versa. The proposed method, i.e., fitting algorithm parameters to a single image, corresponds to the last extreme case. We argue that if, instead of a neural network (which can recreate noise), we use a denoising algorithm A θ with low card(θ)/P , we can define:

C N2N y,y ′ (θ) := ∥A θ (y) -y ′ ∥ 2 2 , xN2N := A θ(y), (7)
and have xN2N ≈ x * (we remind the reader that here, as for the rest of the paper, θ := arg min C y (θ)). This claim, consistent with [START_REF] Afkham | Learning regularization parameters of inverse problems via deep neural networks[END_REF], is supported by our numerical experiments in Section III where we consistently obtain an estimate xN2N of the same quality as the gold standard x * . Remark 2: The cost C N2N y,y ′ is presented using the L 2 norm. Yet, other distances L can be considered, for instance, to handle other types of noise as in [START_REF] Lehtinen | Noise2Noise: Learning image restoration without clean data[END_REF].

Still, N2N requires two independent noisy versions of the same clean image, which is not very standard in practice. Thus, we explored alternative ideas that extend the N2N one such that a single noise realization y per clean image x is required. They revolve around strategies of renoising y into z so as to create pairs of noisy images. These ideas and the corresponding costs and inferences we propose in our context are described below.

Noisy-as-Clean (NaC) [START_REF] Xu | Noisy-as-Clean: Learning self-supervised denoising from corrupted image[END_REF]: The underlying idea is to create a doubly noisy image z = y + n s , with n s being simulated noise drawn from the same distribution as n according to (1), and learn to go from z to y. Inference is then done on regular noisy images: x = f ω (y). From the continuity of L, f ω , p y|x , p z|y and assuming that the noise is low, the authors in [START_REF] Xu | Noisy-as-Clean: Learning self-supervised denoising from corrupted image[END_REF] argue that ω ≈ ω * . Since f ω is continuous, this implies x ≈ x * . Building upon this idea, we propose the following scheme:

C NaC y (θ) := E z ∥A θ (z) -y∥ 2 2 , xNaC := A θ(y). (8)
Noisier2Noise (Nr2N) [START_REF] Moran | Noisier2Noise: Learning to denoise from unpaired noisy data[END_REF]: As in NaC, it also proceeds by creating a doubly noisy image z = y + n s , with n s following the same distribution as n according to (1), and learn to go from z to y. However, it does not require any other assumptions than the sole additive noise. In particular, it is not restricted to a low noise level. As such, ω ̸ ≈ ω * , so inference cannot be performed on y. It is instead done on z and includes a correction step:

x = 2f ω (z) -z = E[x|z]
. This amounts to supervised denoising of the image z, which has a higher noise level than y. To mitigate the effect of this artificially high noise level, one solution is to lower the variance of n s . The correction step will then depend on the new variance. For Gaussian noise, with n ∼ N (0, σ), NaC uses n s ∼ N (0, ασ),

with α ∈]0, 1]. The inference becomes x = (1+α 2 )f ω (z)-z α 2
. A tradeoff has then to be made between a lower simulated noise, and an amplification of denoising errors due to the α 2 division. Exploiting the Nr2N idea in our context leads to

C Nr2N y (θ) := E z ∥A θ (z) -y∥ 2 2 xNr2N := (1 + α 2 )A θ(z) -z α 2 . ( 9
)
Recorrupted-to-Recorrupted (R2R) [START_REF] Pang | Recorruptedto-Recorrupted: Unsupervised deep learning for image denoising[END_REF]: Here, the authors propose to create two doubly noisy images z 1 = y + D T n s and z 2 = y -D -1 n s , with D being any invertible matrix and n s drawn from the same distribution as n, according to [START_REF] Dabov | Image denoising by sparse 3-D transform-domain collaborative filtering[END_REF]. Note that for R2R, n is assumed to be Gaussian. This way, we can write

z 1 = x + n 1 , z 2 = x + n 2 ,
with n 1 and n 2 being zero-mean, independent noise vectors. Training can then be done as in N2N. Yet, as in NaC and Nr2N, such unsupervised training can be seen as supervised training with a higher noise level. To deal with this, R2R uses a Monte-Carlo scheme in inference to limit the effect of n s :

x = 1 M m f ω (z m 1 )
where the superscript m simply represents the m th drawing of n s . For our fitting problem (4), we propose the scheme:

C R2R y (θ) := E (z1,z2) ∥A θ (z 1 ) -z 2 ∥ 2 2 xR2R := 1 M M m=1 A θ(z m 1 ). (10)

B. Optimization Aspects

To solve Problem (4), we exploit automatic differentiation to perform gradient descent steps. This implies that A θ must be differentiable with respect to θ, or at least subdifferentiable. For instance, as explained in Section II-C, hard thresholds need to be replaced with soft thresholds. For C NaC , C Nr2N and C R2R , we approximate E z by drawing a new random z k at each iteration k ∈ 1; K . For initialization, we use a fixed θ 0 , found by manually tuning θ for a single image. 1

C. Use case: Denoising via Quantum Interactive Patches

We illustrate our method, without loss of generality, to a denoiser, denoted DeQuIP (Denoising via Quantum Interactive Patches) [START_REF] Dutta | A novel image denoising algorithm using concepts of quantum many-body theory[END_REF] which can handle various types of noise. 1 Codes and supplementary materials can be found here It is based on two main concepts of quantum mechanics: (i) exploits the Schrödinger equation of quantum physics to construct an adaptive basis, which enables DeQuIP to deal with various noise models; (ii) treats the image as patches, and formalizes the self-similarity between neighbor patches through a term akin to quantum many-body interaction to efficiently preserve the local structures. Each patch behaves as a single-particle system, which interacts with other neighboring patches, thus the whole image acts as a many-body system. Under a potential V, represented by the image pixel values [START_REF] Dutta | Quantum mechanics-based signal and image representation: Application to denoising[END_REF], the adaptive basis vector ψ describes the characteristics of a virtual quantum particle with energy E, and satisfies the non-relativistic stationary Schrödinger equation, written as:

- ℏ 2 2m ∇ 2 ψ = -Vψ + Eψ, (11)
with ℏ being Planck's constant, m the particle mass, and ∇ 2 the Laplacian operator. The image-dependent basis is constructed from the wave solutions ψ i of eq. ( 11) by plugging the noisy image y as the potential V of the system. These wave solutions are oscillating functions with a local frequency proportional to (E -V)/(ℏ 2 /2m), thus the frequency is locally adapted to the image pixels' values. The exact behavior of these basis vectors with respect to the potential is determined by the constant ℏ 2 2m , which is a parameter of DeQuIP. For a more detailed illustration of these oscillating wave vectors, we refer readers to [START_REF] Dutta | Quantum mechanics-based signal and image representation: Application to denoising[END_REF], [START_REF] Dutta | Image denoising inspired by quantum many-body physics[END_REF]. Finally, for denoising, the noisy image y is projected onto this adaptive basis, and the low-value coefficients are thresholded using a soft threshold:

Υ c1,c2 (x) =        0 if |α| ≤ c 1 c2 c2-c1 x -c1c2 c2-c1 if -c 2 ≤ x ≤ -c 1 c2 c2-c1 x + c1c2 c2-c1 if c 2 ≥ x ≥ c 1 1 elsewhere
The second key idea of DeQuIP is the integration of quantum interaction theory to consider the image as overlapping patches, where each patch is a single-particle system that interacts with other patches [START_REF] Dutta | A novel image denoising algorithm using concepts of quantum many-body theory[END_REF]. The interaction between two patches A and B is defined as: I AB = p |A-B| d(A,B) 2 with p a parameter controlling the strength of the interaction term, and d the euclidean pixel distances between the center of A and B. This term is designed to promote local self-similarity, based on the hypothesis that neighboring patches in an image are likely to be more similar than distant ones.

Therefore, DeQuIP has 4 parameters θ = {-ℏ 2 2m , p, c 1 , c 2 } shared across patches.

III. EXPERIMENTAL RESULTS

A. Material & Data

Implementation is done on PyTorch 1.12.0. We used 180 × 180 clean grayscale photographic images, extracted from the BSD400 dataset, as ground truth.

B. Comparison of Denoised Images

The estimates xN2N , xNaC , xNr2N , xR2R and the gold standard x * are compared in the zero-mean Gaussian noise case since it is the only case covered by the four proposed schemes. We report on Fig. 2 the average performance (measured by output PSNR) and standard deviation over 35 test images as a function of the input PSNR. Examples of noisy and denoised images are presented in Fig. 1. We see, both in metric and visually, that xN2N and xR2R achieve the gold standard performance x * . As expected given the assumptions it relies on, xNaC yields better results when noise is low enough (i.e., high PSNR), but still underperforms. Finally, xNr2N does not perform as well as the other unsupervised methods.

C. Poisson Noise Denoising

As DeQuIP can deal with various noise models, we present in Fig. 3 Poisson noise denoising results, where we can observe that xN2N is similar to x * . We used C N2N because of its performances, and its capacity to handle Poisson noise.

IV. CONCLUSION

We have proposed a method for the automatic tuning of denoising algorithm parameters, leveraging only the noisy measurements targeted for enhancement. Specially, we introduced several cost functions and inference schemes, two of which yielded results comparable to those obtained with ground truth-tuned parameters. However, our method comes with certain limitations. The first one arises when only a single noisy image is available, requiring the noise to follow a zero-mean Gaussian distribution. For other types of noise, two independent noisy versions are needed. The second limitation is the need to resolve an optimization problem that involves differentiating the denoising algorithm with respect to its parameters. Although this can be achieved using a simple backpropagation, it can be computationally expensive.

Moving forward, there are potential avenues for exploration. For algorithms A that cannot replicate the identity function, exploring unsupervised losses inspired by Noise2Self [START_REF] Krull | Noise2voidlearning denoising from single noisy images[END_REF] and [START_REF] Batson | Noise2self: Blind denoising by self-supervision[END_REF] could be fruitful, given their advantage of using a single noise realization y without renoising it. Another idea worth exploring is the extension of our method to other inverse problems such as image deblurring or super-resolution, possibly building on [START_REF] Xia | Training image estimators without image ground truth[END_REF], [START_REF] Tachella | Unsupervised learning from incomplete measurements for inverse problems[END_REF].

Fig. 1 :

 1 Fig.1: Exemple of denoising results. The upper scripts of x refer to the corresponding method; x * and x denote respectively the result for the supervised (gold standard) approach and the ground-truth.

Fig. 2 :

 2 Fig. 2: Comparison of the proposed strategies with the gold standard x * . The average performance and standard deviation is computed over 35 test images.

Fig. 3 :

 3 Fig. 3: Poisson noise denoising.