Yunong Wu

Thomas Gilles

Bogdan Stanciulescu

Fabien Moutarde
email: fabien.moutarde@minesparis.psl.eu

TSGN: Temporal Scene Graph Neural Networks with Projected Vectorized Representation for Multi-Agent Motion Prediction

Predicting future motions of nearby agents is essential for an autonomous vehicle to take safe and effective actions. In this paper, we propose TSGN, a framework using Temporal Scene Graph Neural Networks with projected vectorized representations for multi-agent trajectory prediction. Projected vectorized representation models the traffic scene as a graph which is constructed by a set of vectors. These vectors represent agents, road network, and their spatial relative relationships. All relative features under this representation are both translationand rotation-invariant. Based on this representation, TSGN captures the spatial-temporal features across agents, road network, interactions among them, and temporal dependencies of temporal traffic scenes. TSGN can predict multimodal future trajectories for all agents simultaneously, plausibly, and accurately. Meanwhile, we propose a Hierarchical Lane Transformer for capturing interactions between agents and road network, which filters the surrounding road network and only keeps the most probable lane segments which could have an impact on the future behavior of the target agent. Without sacrificing the prediction performance, this greatly reduces the computational burden. Experiments show TSGN achieves state-of-the-art performance on the Argoverse motion forecasting benchmark.

I. INTRODUCTION

Predicting the motion of multiple agents is an essential step in autonomous driving. It helps autonomous driving vehicles to plan their actions and prevent accidents. However, the traffic scene is highly complex. It contains agents, road network, and interactions among them. The prediction model needs to take these entities as inputs and outputs reasonable multimodal trajectories that the target agent could take in the future.

The traffic scene is viewed differently from the perspective of different agents. However, most exsiting methods select one target agent each time and take it as the central reference to perform the coordinate transformation on the entire scene, which is not symmetric for the other agents and results in only one prediction at a time. It is inefficient and not robust to the coordinate transformation. To alleviate these problems, HiVT [START_REF] Zhou | Hivt: Hierarchical vector transformer for multi-agent motion prediction[END_REF] takes a symmetric way to encode the traffic scene. It divides the traffic scene into a set of local regions, each corresponding to one agent and centered on it, and uses a local-global mechanism to encode agents' features.

We adopt the symmetrical representation of HiVT and its local-global mechanisms. Based on them, we propose an extended scene representation method: Projected vectorized representation and an optimized prediction framework: TSGN. Our representation method introduces more features about agents and lane segments. It projects all relative spatial features between entities into the direction of the target entity for every timestamp, such that, all these relative features under this representation are also both translation-and rotation-invariant. TSGN consists of diverse interaction modules and temporal dependencies modules. These modules are constructed based on the transformer structure. TSGN treats the traffic scene in its entirety and encodes the agents' dynamics, surrounding lane segments, and interactions among them across time steps. In addition, we present a Hierarchical Lane Transformer module inside TSGN to capture the influence of the road network on the future motion of target agents. Unlike most approaches that use the full structural information of the surrounding road network, the Hierarchical Lane Transformer only selects the most probable lane segments which could have an impact on the future behavior of the target agent. It greatly reduces the computation burden and enables the model to do faster predictions without sacrificing the prediction performance.

Our contributions are summarized as follow:

• We extend the scene representation method of HiVT by adding new features about agents, lane segments, and their spatial relative relationships. We adopt a projected vectorized representation for all relative features of the traffic scene for every timestamp, which describes the relative spatial relationships in a more detailed way. • We propose a Hierarchical Lane Transformer which only selects lane segments that affect the future behaviors of target agents the most for modeling the interactions, enabling much faster and equally accurate predictions. • Our designed TSGN can make plausible and accurate predictions. We validate the performance of TSGN on Argoverse forecasting benchmark. It ranked 1 st in terms of minADE on the leaderboard on September 13, 2022.

II. RELATED WORK

Recently deep learning-based models have brought great progress to the motion prediction. According to the different representation methods, these learning-based models can be divided into rasterized-representation-based models and vectorized-representation-based models.

Rasterized representation of traffic scenes has benefits of unitedly rendering various types of input information (e.g., HD map, agents' trajectories, spatial relationships) into an image with multiple channels. In [START_REF] Djuric | Uncertainty-aware short-term motion prediction of traffic actors for autonomous driving[END_REF], [START_REF] Cui | Multimodal trajectory predictions for autonomous driving using deep convolutional networks[END_REF], different types of entities in traffic scenes are assigned different colors. HOME [START_REF] Gilles | Home: Heatmap output for future motion estimation[END_REF] rasterizes elements of HD map in different semantic channels and adds historical trajectories of the target agent and other agents on multiple channels to represent agents' physical states at each timestamp. In the same branch, [START_REF] Hong | Rules of the road: Predicting driving behavior with a convolutional model of semantic interactions[END_REF], [START_REF] Chai | Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction[END_REF] adopt a similar rasterized representation method. However, rasterized representation methods are limited to restricted image size and CNNs have constrained receptive fields. Besides, they cannot capture the complex structural information of HD map and it is difficult for them to represent the agents' dynamics.

More recently, vectorized representation methods are proposed to capture the structural information of traffic scenes. They use graph neural networks (GNNs) to capture the spatial features of agents and HD map and to learn the relationships between them. VectorNet [START_REF] Gao | Vectornet: Encoding hd maps and agent dynamics from vectorized representation[END_REF] is the first to use the vectorized representation method to encode both HD map and agents' dynamics. TNT [START_REF] Zhao | Tnt: Target-driven trajectory prediction[END_REF] and DenseTNT [START_REF] Gu | Densetnt: End-to-end trajectory prediction from dense goal sets[END_REF] adopt the same representation method and use VectorNet as their backbone network. LaneGCN [START_REF] Liang | Learning lane graph representations for motion forecasting[END_REF] treats lane segments as nodes and constructs a lane graph from the lane segments. [START_REF] Gilles | Thomas: Trajectory heatmap output with learned multi-agent sampling[END_REF]- [START_REF] Deo | Multimodal trajectory prediction conditioned on lane-graph traversals[END_REF] adopt similar lane graph representation methods as LaneGCN. TPCN [START_REF] Ye | Tpcn: Temporal point cloud networks for motion forecasting[END_REF] and its extension DCMS [START_REF] Ye | Dcms: Motion forecasting with dual consistency and multi-pseudo-target supervision[END_REF] treat trajectory prediction problems as joint temporal point cloud learning and use PointNet++ [START_REF] Qi | Pointnet++: Deep hierarchical feature learning on point sets in a metric space[END_REF] in the point representation learning. HDGT [START_REF] Jia | Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding[END_REF] models the traffic scene as a heterogeneous graph with different types of nodes and edges. The approach most related to this paper is HiVT [START_REF] Zhou | Hivt: Hierarchical vector transformer for multi-agent motion prediction[END_REF], which uses a translation-invariant scene representation method that avoids using absolute positions and characterizes the geometric entities with relative positions and constructs rotation-invariant transformers to model the different interactions between the vectorized entities locally and globally.

III. TEMPORAL TRAFFIC SCENE REPRESENTATION

We adopt the feature representation of HiVT and extend it. In this section, we introduce our feature representation in details. Our introduced extended features of the traffic scene and the original features of HiVT in their definitions are presented in blue and black respectively.

A. Node Feature Representation

For traffic agents, we extract their trajectory segments which are in a form of directed splines with respect to time. We consider each agent at each timestamp as a node and we characterize these temporal geometric nodes' attributes as

n a = {R T i d t i , R T i v t i , s t i , R T i a t i , ∆t t , b i |i = 1, ..., N t , t = 0, ..., 19} with
p t i = [x t i , y t i ] s t i = v t i (1)
d t i = p t i -p t-1 i α t i = arctan d y t i d x t i
(2)

d x t i = x t i -x t-1 i a t i = [cos(α t i ), sin(α t i )] (3)
d y t i = y t i -y t-1 i ∆t t = t t -t t-1 (4)
v t i = d t i ∆t t (5)
where N t represents the total number of agents at timestamp t, p t i is the location vector which consists of the lateral and longitudinal position of agent i at timestamp t, d t i is the displacement vector from timestamp t -1 to timestamp t, v t i is the velocity, s t i is the speed, a t i is the heading vector which consists of the cosine and sine of the agent's heading α t i , ∆t t is the timestamp difference, R T i is the rotation matrix parameterized by the heading α i at the current timestamp T [START_REF] Vaswani | Attention is all you need[END_REF], and b i corresponds to semantic attribute. The sampling rate is 10Hz, but we found it deviates in different scenes, thus we include the timestamp difference into the node feature.

For road network, we extract coordinates of lane segments and their semantic attributes, such as turn directions. We similarly vectorize lane segments as the vectorization for agents. The geometric node attributes of lane segments n l = {d ξ , a ξ , l ξ , b ξ |ξ = 1, ..., M } consist of displacement vector d ξ of each lane segment, heading vector a ξ , length of the lane segment l ξ , and also the semantic attribute b ξ , where

d ξ = p 1 ξ -p 0 ξ l ξ = d ξ (6) a ξ = [cos(α ξ ), sin(α ξ )] (7)
M is the total number of lane segments, p 0 ξ and p 1 ξ represent the start point and end point of the lane segment ξ respectively. Instead of using absolute representations, using relative representations ensure these state quantities are not affected by coordinate translation transformations.

B. Edge Feature Representation

The node features do not preserve relative spatial relationships. Thus, we further introduce the geometric edge attributes between entities. We characterize the edge attributes between agents as

e aa = {R T i d t ij , R T i v t ij , l ij , s ij , d t j2i , v t j2i , a t j2i |t = 0, ..., 19; i, j = 1, ..., N t , i = j}, where d t ij = p t j -p t i α t ij = α t j -α t i (8) l t ij = d t ij a t j2i = [cos(α t ji ), sin(α t ji )] (9)
v t ij = v t j -v t i d t j2i = R t i d t ji ( 10
)
s t ij = v t ij v t j2i = R t i v t ji ( 11
)
where d t ij is the relative position vector between the target agent i and agent j at timestamp t, l t ij is the distance, v t ij is the relative velocity and s ij is the relative speed. R t i is the rotation matrix parameterized by the heading α t i of the target agent i at the timestamp t. the projected relative position vector and projected relative velocity respectively, and a t j2i is the relative heading vector. The projected relative position is defined as projecting relative position in the direction of the heading of the target agent. An example is shown in the left side of Fig. 1. Similarly, we obtain the projected relative velocity by projecting relative velocity in the direction of the heading of the target agent.

We characterize the edge features between agents and lane segments as

e al = {R T i d t iξ , d t i2ξ , l t iξ , v t i2ξ , a t i2ξ |t = 0, ..., 19; i = 1, ..., N t ; ξ == 1, ..., M }, where d t iξ = p 0 ξ -p t i v t i2ξ = R t ξ v t i ( 12
)
d t i2ξ = R t ξ d t iξ α t iξ = α t ξ -α t i (13) l t ij = d t iξ a t i2ξ = [cos(α t iξ ), sin(α t iξ )] (14)
where d t iξ is the relative position vector between agent i and lane segment ξ at timestamp t, l t iξ is the distance and R ξ is the rotation matrix parameterized by the heading α ξ of the target lane. d t i2ξ and v t i2ξ represent the projected relative position vector and projected relative velocity respectively, and a t i2ξ is the relative heading vector. The projected relative position between lane segment ξ and agent i is defined as projecting relative position in the direction of the heading of the target lane segment ξ. An example is shown in the right side of the Fig. 1. Similarly, we obtain the projected relative velocity v t i2ξ by projecting the agent's velocity in the direction of the heading of the target lane segment.

The projected relative position and the projected relative velocity depict how close two individual nodes are and how fast one node moves towards another node in both lateral and longitudinal directions. Compared to relative representation, projected relative representation describes the interaction between entities in a more detailed way, allowing downstream networks to better understand their behavior. Moreover, such a projected relative representation guarantees translation invariance and rotation invariance naturally.

Temporal Scene Graph

𝑡 19 𝑡 1 𝑡 2 𝑡 0 … Temporal 1 AA 1 AL 1 Update Agent Nodes Update Edges between Entities h AA 1 AA 1 AA 1 AL 1 AL 1 Temporal 2 AA 2 AL 2 AL 1 … … Fig. 2.
Local Encoder of TSGN. The local encoder processes the temporal traffic scene in two stages. In the first stage, it models the agent-agent interactions (A-A) and agent-lane interactions (A-L) per time step and uses a temporal transformer module to capture the temporal dependencies across traffic scenes. In the second stage, the refreshed agents' features are used to update the relative spatial relationships between entities, and the agent-agent interaction is modeled again per time step. Then a temporal transformer with a classification token for summarizing the temporal information and an agentlane transformer focusing only on the current timestamp with a larger receptive field are applied. h is the final local representation for all agents.

IV. PREDICTION FRAMEWORK

In this section, we introduce our prediction framework TSGN which consists of a local encoder, a global encoder, and a multimodal decoder as well as the components of the local encoder. Following that, we cover details about our proposed Hierarchical Lane Transformer.

A. Local Encoder

The local encoder operates the temporal scene graph in two stages. In the first stage, it models agent-agent interactions and agent-lane interactions per time step and uses a temporal transformer module to capture temporal dependencies across traffic scenes. Since the number of agent-lane edges is large, we use a reduced receptive field of view in this stage for efficiency. In the second stage, it uses the refreshed agents' features to update relative spatial relationships between entities and models agent-agent interactions per time step again, followed by a temporal transformer with a classification token for summarizing the temporal information and an agent-lane transformer only focusing on the current timestamp with a larger receptive field, we obtain the final updated features for each agent. In summary, the local encoder of TSGN contains the following modules: Agent-Agent Transformer in the first stage (AA 1 ) and second stage (AA 2 ), Agent-Lane Transformer in the first stage (AL 1 for all timestamps) and second stage (AL 2 only for the current timestamp), and Temporal Transformer in the first stage (Temporal 1 ) and second stage (Temporal 2 ). An overview of the local encoder is shown in Fig. 2. In the remainder of this subsection, we first describe the Agent-Agent Transformer, Agent-Lane Transformer, and Temporal Transformer separately, and then we detail how these transformers are going to be used in the two different stages.

Agent-Agent Transformer

We use a multi-head cross-attention block to learn the interaction relationships and the importance of the influence of different surrounding agents on the center agent for each local region at each time step. Specifically, first, we apply Multi-Layer Perceptrons (MLPs) on its node attributes and edge attributes, then we obtain the central agent i's temporal node embedding z t i and any of its associated temporal edge embedding z t ij which is connected with neighboring agent j:

z t i = φ center [R T i d t i , R T i v t i , R T i a t i , s t i , ∆t t , b i ] (15)
z t ij = φ nbr [R T i d t j , R T i d t ij , R T i v t ij ,l t ij ,s t ij ,d t j2i ,v t j2i ,a t j2i ,b j ] (16)
where φ center and φ nbr are MLP blocks. Both node and edge attributes are translation and rotation invariant, their embeddings also inherit these properties. Then, a cross attention block takes z t i as query, z t ij as key, z t ij as value, performs the scaled dot-product attention [START_REF] Vaswani | Attention is all you need[END_REF] with a gating function [START_REF] Zhou | Hivt: Hierarchical vector transformer for multi-agent motion prediction[END_REF] , and outputs ẑt i . We further apply an MLP block in conjunction with residual connections [START_REF] He | Deep residual learning for image recognition[END_REF] and takes ẑt i as input to obtain the first updated node embedding s t i of agent i at timestamp t which incorporates interactions between agents.

Agent-Lane Transformer

To capture implicit interactions between agents and lane segments, we employ another multi-head cross-attention block. First, we encode the local lane segments, and their associated edge attributes which are connected with the central agent i by an MLP block φ lane to obtain the edge embedding:

z iξ = φ lane [R T i d ξ , R T i d iξ , d i2ξ , l iξ , v i2ξ , a i2ξ , b ξ ] (17)
Second, we employ a multi-head cross-attention block that takes agent i's node embedding s t i as query, the edge embedding z iξ as key and value, and performs the scaled dot-product attention the same as the Agent-Agent Transformer, followed by an MLP block in conjunction with residual connections. We then obtain the updated node embedding s t i which incorporates interactions between agents and interactions between agents and lane segments.

Temporal Transformer

We treat the encoded traffic scene at each timestamp as a whole and use a temporal transformer to capture temporal dependencies of all historical encoded traffic scenes. First, we add the learnable positional embeddings [START_REF] Vaswani | Attention is all you need[END_REF] to the central agent i's node embedding { s t i } T t=1 at each timestamp and stack them into a matrix S i . Then, we use a self-attention block to perform the scaled dot-product attention, in which S i is taken as query, key and value. We obtain the output of the temporal transformer Ŝi which summarizes the spatial-temporal features of agent i for every timestamp.

1) First Stage

The local encoder models agent-agent interactions and agent-lane interactions per time step. These two interaction modules aggregate the spatial information of the traffic scene for every timestamp. As explained before, due to the great computational complexity, we use a reduced receptive field of view in this stage for efficiency. Taking the encoded temporal traffic scene as input, the temporal transformer further summarizes the temporal information. Unlike most approaches which only capture temporal dependencies of agents' historical dynamics, we propose to incorporate agent-agent interactions and agent-lane interactions to update agents' node embedding for every timestamp, since the interactions in the past will affect the agents' pattern in the future.

2) Second Stage

We use refreshed node embeddings of agents in the first stage to update the edge embeddings between agents as:

z t ij 2 = φ nbr2 [ŝ t i , z t ij ] (18)
where {ŝ t i } T t=1 is the unstacked result of Ŝi and φ nbr2 is an MLP block. We use refreshed agents' nodes embeddings {ŝ t i } T t=1 and edges embeddings z t ij 2 to repeat an agent-agent interaction encoding for every timestamp. In the first modeling of agent-agent interactions, the agents lacked information about road network, which could in turn affect their behaviors and interactions. Thus, we apply it again, after all agents are incorporated with the road network information. Afterwards, we apply another temporal transformer with an extra learnable classification token [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] and a temporal mask that enforces the tokens only focus on the preceding time steps to summarize the whole temporal sequence into one single token si . Then, the edge embeddings between agents and lane segments are calculated as:

z iξ 2 = φ lane2 [s i , z iξ ] (19)
where φ lane2 is an MLP block. We input the updated extra classification token si as query and updated edge embeddings z iξ 2 between agent i and lane segment ξ as key and value to another same agent-lane interaction module with a larger receptive field. We obtain the final node embedding h i of the central agent i. h i summarizes the fused rich spatial-temporal features of the agent i. It incorporates agent i's dynamics and the iterative interactions between agent i and its surrounding environment. The final local representation for all agents h is defined as h = {h i |i = 1, ..., N }, where N is the number of agents.

B. Global Encoder

The local encoder output represents the features of the central agent within its local region, but it does not incorporate the pairwise interactions between local regions and lack the long-range dependencies in the scene. Thus, we apply a global encoder [START_REF] Zhou | Hivt: Hierarchical vector transformer for multi-agent motion prediction[END_REF] for high-order interactions.

Similar to the Agent-Agent Transformer, we use an MLP φ rel to obtain the edge embedding g ij between agent i's and agent j's local regions:

g ij = φ rel [R T i d T ij , R T i v T ij , l T ij , s T ij , d T j2i , v T j2i , a T j2i ] (20)
where T represents the current timestep. We take h i as query, [h j , g ij ] as key and value. A multi-head cross-attention block receives them as input and outputs agent i's updated node embedding which incorporates global scene information. Afterwards, we employ an MLP block to map the updated node embedding of each agent to the final global representation h with shape [F, N, D], where F is the number of mixture components, representing the multimodality of agents' trajectories in the future, and D is the number of agents' features.

C. Multimodal Decoder

In complex traffic scenes, agents can behave multimodally. To characterize this phenomenon, we apply a mixture density network [START_REF] Bishop | Mixture density networks[END_REF] with GRU [START_REF] Cho | On the properties of neural machine translation: Encoder-decoder approaches[END_REF] as the decoder to output multimodal trajectories for each agent. Specifically, we tile the global representation h with shape

o = GRU([ h, h]) µ = φ loc (o) (21) b = φ unc (o) π = φ prob ([ h, h]) (22)

D. Loss Function

The tracking data of Argoverse dataset 1 contains noisy samples. Compared to Gaussian negative log-likelihood loss, Laplace negative log-likelihood loss is more robust to outliers, thus we design the multimodal decoder in that way so that it predicts a parameterization of the location and its associated uncertainty corresponding to a Laplace distribution [START_REF] Meyer | Learning an uncertainty-aware object detector for autonomous driving[END_REF]. We use the winner-take-all strategy, and only apply the loss to the best-predicted trajectory which is defined as the one that has the minimum endpoint error. Our final loss consists of the regression loss L reg and the classification loss L cls :

L reg = - 1 N H N i=1 T +H t=T +1 log P R T i (p t i -p T i )| μt i , bt i ( 23
)
π i = softmax - T +H t=T +1 µ t i , R T i (p t i -p T i ) (24)
L cls = N i=1 CE(π i , πi ) L = L reg + L cls ( 25
)
where μt i and bt i are the locations and their associated uncertainty of the best-predicted trajectory of agent i at timestamp t. µ t i is the locations of all predicted trajectories of agent i at timestamp t. CE is the cross-entropy loss.

E. Hierarchical Lane Transformer

Fig. 3. Hierarchical process. First, the road network is constructed by lanelets instead of lane segments. Second, Agent-Lanelet Transformer selects the lanelets which score are higher than threshold. Finally, only lane segments that belong to the selected lanelets are taken into account in calculating agentlane interactions. In this example, a lanelet consists of 3 lane segments. The selected lanelets are shown in black, the unselected lanelets are shown in gray, and the past trajectory of the target agent is shown in blue.

The number of agent-lane edges in the local region is still very large, even if we limit the radius of the local region. When the model performs the agent-lane interaction for every timestamp, it suffers from great time complexity and space complexity. In addition, not all lane segments in the local region have an impact on the agents' future behavior. Since a lanelet [START_REF] Bender | Lanelets: Efficient map representation for autonomous driving[END_REF] can be viewed as a set of lane segments, we can filter out the unuseful lanelets and lane segments belonging to them to reduce the computational complexity. In our case, a lanelet is defined as a sequence of 10 centerline points. We propose the Hierarchical Lane Transformer and apply it in the agent-lane interaction module to only select lanelets that are most important to the target agent. As the other unselected lanelets would have scarcely any influence on the target agent, this does not decrease performance. We will demonstrate it in the ablation studies.

Specifically, the Hierarchical Lane Transformer consists of an Agent-Lanelet Transformer and an Agent-Lane Transformer. The Agent-Lanelet Transformer has the same architecture as the Agent-Lane Transformer. We take the average position and the average position vector of all lane segments of lanelet Ξ as the position p Ξ and position vector d Ξ of the lanelet Ξ. We create the edge embedding z iΞ between agent i and lanelet Ξ as follows:

z iΞ = φ lanelet [R T i d Ξ , R T i d iΞ , d i2Ξ , l iΞ , v i2Ξ , h i2Ξ , a Ξ ] (26)
In the first step, the Agent-Lanelet Transformer performs the scaled dot-product attention and yields a score α t iΞ for each lanelet to identify the most probably lanelets that can affect the future motion of the target agent i. In the second step, only lane segments whose score are greater than threshold are taken into account in calculating agent-lane interactions. To enhance the influence of the lanelet's score α t iΞ , we multiply the scaled dot-product in Agent-Lane Transformer with α t iΞ before softmax function: 1.3640 0.8679 0.1634 Scene Transformer [START_REF] Ngiam | Scene transformer: A unified architecture for predicting multiple agent trajectories[END_REF] 1.2321 0.8026 0.1255 HOME+GOME [START_REF]Gohome: Graph-oriented heatmap output for future motion estimation[END_REF] 1.2919 0.8904 0.0846 DenseTNT [START_REF] Gu | Densetnt: End-to-end trajectory prediction from dense goal sets[END_REF] 1.2815 0.8817 0.1258 TPCN [START_REF] Ye | Tpcn: Temporal point cloud networks for motion forecasting[END_REF] 1.2442 0.8153 0.1333 MultiPath++ [START_REF] Varadarajan | Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction[END_REF] 1.214 0.790 0.13 HiVT [START_REF] Zhou | Hivt: Hierarchical vector transformer for multi-agent motion prediction[END_REF] 1.1693 0.7735 0.1267 DCMS [START_REF] Ye | Dcms: Motion forecasting with dual consistency and multi-pseudo-target supervision[END_REF] 1.1350 0.7659 0.1094 Wayformer [START_REF] Nayakanti | Wayformer: Motion forecasting via simple & efficient attention networks[END_REF] 1.1615 0.7675 0.1186 TSGN 1.1370 0.7537 0.1223

α t iξ = α t iΞ • softmax q t i √ d k • k t iξ (27)
where the lane segment ξ belongs to the lanelet Ξ. An example of this hierarchical process is shown in Fig. 3.

V. EXPERIMENTS

A. Experimental Settings 1) Dataset

We use the Argoverse Motion Forecasting Dataset [START_REF] Chang | Argoverse: 3d tracking and forecasting with rich maps[END_REF]. It is a dataset with agent trajectories and HD map which contains 324557 real traffic scenarios. The training, validation, and test sets contain 205942, 39472, and 78143 scenarios respectively. Each scenario is a 5-second sequence long sampled at 10 Hz and contains the position of all agents in the past 2 seconds. In Argoverse Motion Forecasting Challenge, we need to predict the 3-second future positions of one target agent given the initial 2-second observations of the scene.

2) Metrics

We follow the Argoverse benchmark and use minimum average displacement error (minADE), minimum final displacement error (minFDE), and miss rate (MR) to evaluate our model. These metrics allow models to forecast up to 6 trajectories for each agent.

3) Implementation Details

We train all models for 64 epochs using AdamW optimizer [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] initialized with a learning rate of 0.0005. We employ the cosine annealing scheduler for the learning rate decay. We set the number of layers for Agent-Agent Transformer, Agent-Lane Transformer, Agent-Lanelet Transformer, Temporal Transformer, and Global Encoder to 1,1,1,4,3. The number of hidden units is 128, the number of heads in all multi-head attention blocks is 8. The radius of local regions in the first stage is set to 20 meters, while in the second stage is set to 50 meters. We drop the agent which moved less than 6 meters unless it is the target agent [START_REF] Ngiam | Scene transformer: A unified architecture for predicting multiple agent trajectories[END_REF].

B. Comparison with State-of-the-art

We show in Tab. I the results of TSGN compared to other state-of-the-art models on the Argoverse motion forecasting test set. The data in Tab. 1 is collected from the Argoverse leaderboard on 13/09/2022. TSGN outperforms all the other methods in terms of minADE, and remains competitive ranking on minFDE, which verifies the superior prediction performance of our method.

C. Ablation Studies

Our ablation studies consist of four parts: the importance of each module of TSGN, the importance of the extended scene reresentation and the importance of the Hierarchical Lane Transformer. We conduct these experiments on the Argoverse validation set.

1) Importance of Each Module

HiVT consists of AA 1 , AL 2 , Temporal 1 , and Global modules. The functions of these 4 modules can be seen in [START_REF] Zhou | Hivt: Hierarchical vector transformer for multi-agent motion prediction[END_REF]. As shown in Tab. II, AL 1 , AA 2 , and Temporal 2 modules can improve minADE, minFDE, and MR to a certain degree. First, every moment of interaction will affect future behavior. AL 1 captures the agent-lane interaction at each timestamp, it noticeably improves the prediction performance. Second, the components of the second stage of the local encoder (AA 2 , Temporal 2 ) have a significant impact on the performance. As discussed before, AA 1 does not consider the influence that the agent-lane interaction has on the agent-agent interaction. Based on AA 1 , AA 2 further captures the agent-agent interaction incorporated with the map information.

2) Importance of the Extended Scene Reresentation In this ablation study, we use HiVT [START_REF] Zhou | Hivt: Hierarchical vector transformer for multi-agent motion prediction[END_REF] as the prediction framework. As shown in Tab. III, compared to the original representation, the extended scene representation yields better performance. Our proposed representation provides more information about the entities and more finely describes the relative relationships between these entities, allowing downstream networks to better understand the interactions between individuals in a complex scene.

3) Importance of the Hierarchical Lane Transformer

In this ablation study, we use TSGN as the prediction framework. We present the performance evaluation of the Hierarchical Lane Transformer in Tab. IV. In practice, we select the lanelets whose score α t iΞ is greater than 0.75 • α t iΞ , where α t iΞ represents the scatter mean score of all lanelets which have a shared edge with agent i. We measure the resulting average proportion of selected lanes to be 45.8%, dividing the number of processed lane features by more than 2. As a consequence, the computational complexity of agentlane interaction is significantly reduced and the measured inference speed improves greatly. Moreover, the Hierarchical Lane Transformer also keeps the prediction performance. As shown in Tab. IV, the minFDE, minADE and MR are barely significantly worse.

D. Qualitative Results of TSGN

From Fig. 4(a) to Fig. 4(d), we visualize the qualitative results of TSGN in 4 traffic scenes. One can conclude that TSGN can make multimodal predictions for all agents simultaneously and the prediction in each maneuver is reasonable. Comparing the best-predicted trajectory with the ground truth, we see that TSGN can make accurate predictions.

E. Comparison with HiVT in Worse Cases

To compare with HiVT, we present the qualitative results of TSGN and HiVT in some worse cases of HiVT, in which the minFDE of HiVT is greater than 10 meters. For clarity, we only visualize end points of the 6 predicted trajectories of the central agent. We compare these two models in two different traffic scenes. TSGN is more sensitive to speed-related information. When the agent is accelerating or decelerating, it can better perceive the current motion state of the agents. conclude that it's hard for TSGN to make accurate predictions when the agents intend to perform maneuver 'lane change' or 'U turn'. One possible cause of the failure is, compared with conventional maneuvers, such as 'Go straight', 'left turn' and 'right turn', the proportion of maneuver 'lane change' and 'Uturn' is very low. It is difficult for the model to learn to predict these kinds of intentions. Second, in Fig. 6(c), we observe that the prediction performance of TSGN is not robust, if the tracking information of the agent in the past is unstable. The prediction results seem to be overfitting to past perception errors. Third, when the agent moves in a complex traffic scene where the surrounding lane segments are extraordinarily tangled, e.g., the traffic scene in Fig. 6(d), TSGN fails in understanding the map structure and makes unreasonable multimodal predictions. We want TSGN to output distinct maneuvers, but it only presents predictions in one maneuver.

G. Visualization of Selected Lanelets by Hierarchical Lane Transformer and Their Scores

In Fig. 7(a) and 7(b), we present the selected lanelets by Hierarchical Lane Transformer of two different traffic scenes. Fig. 7(c) and 7(d) show the corresponding selected lanelets' scores. We observe that our model can select the most important lanelets to the target agent. The selected lanelets contain the possible future trajectories of the target agent. In addition, the prediction of lanelets' scores is reasonable.

VI. CONCLUSION

In this paper, we propose TSGN with projected vectorized representation for multi-agent trajectory prediction. It models the temporal traffic scene as a temporal scene graph and uses diverse transformer-based interaction modules and temporal dependencies modules to operate this temporal scene graph. Moreover, we present a Hierarchical Lane Transformer module inside TSGN, which effectively and efficiently captures the influence of the HD-map on the future motion of target agents. Experiments show our method achieves state-of-the-art performance on the Argoverse motion forecasting benchmark.

d t j2i and v t j2i areFig. 1 .

 1 Fig. 1. In the figure on the left, the blue agent i is defined as the target agent, d ji is the relative position vector between agent i and agent j, d j2ix and d j2iy is the lateral and longitudinal projected relative position. In the figure on the right, d i2ξ is the relative position vector between agent i and the black lane segment ξ, d i2ξx and d i2ξy is the lateral and longitudinal projected relative position.

  [F, N, D] and the local representation h with shape [N, D] H and F times, then we obtain the tiled global representation with shape [H, F ×N, D] and the tiled local representation with shape [1, F × N, D], where H is the number of future steps. The GRU receives the tiled global representation as inputs and the tiled local representation as the initial hidden state and outputs o with shape [H, F × N, D]. We transpose the first dimension and second dimension of o and apply two MLP blocks φ loc , φ unc which take the GRU output o as input and output locations µ with shape [F, N, H, 2] and associated uncertainties b with shape [F, N, H, 2]. We apply another MLP block φ prob with the concatenation of h and h as input to obtain the probability π of possible future trajectories of all agents:

Fig. 4 .

 4 Fig.4. Qualitative Results of TSGN. For clarity, we visualize only agents which are observed at current timestamp and move more than 6 meters. We present the past trajectories in orange, the ground-truth trajectories in red and the predicted trajectories in green.

Fig. 5 .Fig. 6 .

 56 Fig. 5. Qualitative results of HiVT and TSGN are shown in the first row and second row respectively. The central agent's past trajectory is shown in orange, the ground-truth trajectory is shown in red, and the endpoints of predicted trajectories are shown in blue. The past trajectories of other agents are shown in green.

F

  . Failed Cases From Fig. 6(a) to Fig. 6(d), we present some classic failed cases of TSGN. First, as shown in Fig. 6(a) and Fig. 6(b), we

Fig. 7 .

 7 Fig. 7. Selected lanelets by Hierarchical Lane Transformer and their' scores.The agent's past trajectory is shown in orange, and the ground-truth trajectory is shown in red. In the first row, selected lanelets and unselected lanelets are presented in black and grey respectively. In the second row, the depth of the selected lanelet's color represents the level of the score. The higher the score, the more important the lanelet is to the target agent.

TABLE I RESULTS

 I ON ARGOVERSE MOTION FORECASTING LEADERBOARD.

	Models	minFDE minADE	MR
	LaneGCN [10]

TABLE II IMPORTANCE

 II OF EACH COMPONENT OF OUR FRAMEWORK. AA 1 AL 2 Temporal 1 AL 1 AA 2 Temporal 2 minFDE minADE MR

			0.959	0.652 0.090
			0.930	0.644 0.085
			0.916	0.636 0.084
	TABLE III
	ABLATION STUDIES ON THE EXTENDED SCENE REPRESENTATION.
	Representation method	minFDE minADE MR
	raw scene representation	0.967	0.662 0.092
	extended scene representation 0.959	0.652 0.090


Amandine Brunetto

Sascha Hornauer

Stella X Yu

Fabien Moutarde

The Audio-Visual BatVision Dataset for Research on Sight and Sound

: The BatVision dataset contains large scale audio-visual data from a robots' perspective. For its creation, a robot traversed corridors, offices, lecture halls and driveways at a historic campus and modern office building like a bat, emitting chirping sounds with a speaker. A binaural microphone recorded their echoes which carry rich scene information of objects, materials and layout. With this paper we provide echoes, camera images and depth maps, shown overlayed on typical scenes on the right. Echoes are shown under images. This dataset will help investigate fundamental questions on how sound interacts with spaces, how it can be harnessed for robotic navigation and what in general can be understood about a scene from how it sounds.

Abstract-Vision research showed remarkable success in understanding our world, propelled by datasets of images and videos. Sensor data from radar, LiDAR and cameras supports research in robotics and autonomous driving for at least a decade. However, while visual sensors may fail in some conditions, sound has recently shown potential to complement sensor data. Simulated room impulse responses (RIR) in 3D apartment-models became a benchmark dataset for the community, fostering a range of audiovisual research. In simulation, depth is predictable from sound, by learning bat-like perception with a neural network. Concurrently, the same was achieved in reality by using RGB-D images and echoes of chirping sounds. Biomimicking bat perception is an exciting new direction but needs dedicated datasets to explore the potential. Therefore, we collected the BatVision dataset to provide large-scale echoes in complex real-world scenes to the community. We equipped a robot with a speaker to emit chirps and a binaural microphone to record their echoes. Synchronized RGB-D images from the same perspective provide visual labels of traversed spaces. We sampled modern US office spaces to historic French university grounds, indoor and outdoor with large architectural variety. This dataset will allow research on robot echolocation, general audio-visual tasks and sound phaenomena unavailable in simulated data. We show promising results for audio-only depth prediction and show how state-of-the-art work developed for simulated data can also succeed on our dataset. The data can be downloaded at https://forms.gle/W6xtshMgoXGZDwsE7

I. INTRODUCTION

Large-scale datasets propelled research in past decades, providing first static images and later an abundance of videos for tasks from object detection to activity recognition. Sounds, correlated with visual data from their source, provide exploitable information about an action or context, often at marginal computational overhead.

A novel research direction aims to listen to the environment for improved task performance. Simulated room impulse response (RIR) datasets allow researchers to investigate the interaction of sounds with known space layouts for e.g. depth prediction, obstacle avoidance or to drive towards an alarm beyond the line of sight. They have been used successfully to predict 3D layouts from simulated chirps, similar to how bats find their prey. Robots mastering this echolocation could create instant maps beyond their immediate surroundings without LiDAR, overcoming their field of view and operate even in darkness and smoke.

While some datasets provide recorded RIRs within limitations, there exists no large-scale real dataset to investigate audio-visual 3D scene understanding for robots. This hinders domain adaptation from simulation as well as learning to be robust against or even exploit real world background sounds.

Bats are capable of navigating and localizing prey in flight using echolocation. We showed successfully how to adopt this feat for machine listening, using only audible echoes of chirps for depth prediction [START_REF] Jesper Haahr Christensen | Batvision: Learning to see 3d spatial layout with two ears[END_REF][START_REF] Jesper Haahr Christensen | BatVision with GCC-PHAT Features for Better Sound to Vision Predictions[END_REF]. To understand the extend to which this method can complement failing vision sensors and even extend beyond the field of view, more real data in typical robotic scenarios is needed. We therefore present the BatVision Dataset providing publicly available large-scale real data of complex scenes for research on audio-visual 3D scene understanding and robotic echolocation.

Sound-augmented Task Performance. Sound arrives omnidirectional and provides rich information about the space it crosses. Due to the correlation of sound and vision it can provide complementary information when visual sensors fail. Cameras and LiDAR struggle with occlusions, reflections and smoke. Depth-from-stereo algorithms produce artifacts in low-light conditions and when objects lack textures. Combining audio-visual information can improve task performance with one modality providing labels for the other [START_REF] Schulz | Hearing what you cannot see: Acoustic vehicle detection around corners[END_REF][START_REF] Rivera Valverde | There is more than meets the eye: Self-supervised multi-object detection and tracking with sound by distilling multimodal knowledge[END_REF]. Biomimicking the echolocation principle can even improve 3D scene understanding using sound [START_REF] Jesper Haahr Christensen | Batvision: Learning to see 3d spatial layout with two ears[END_REF][START_REF] Gao | Visualechoes: Spatial image representation learning through echolocation[END_REF][START_REF] Zhang | Stereo Depth Estimation with Echoes[END_REF].

Audio-visual data suitable for these tasks is sparse because real-world recordings require complex setups and long sessions [START_REF] Čmejla | Mirage: Multichannel database of room impulse responses measured on highresolution cube-shaped grid in multiple acoustic conditions[END_REF][START_REF] Hadad | Multichannel audio database in various acoustic environments[END_REF][START_REF] Dai | Binaural SoundNet: Predicting Semantics, Depth and Motion With Binaural Sounds[END_REF]. In consequence, researchers often create small datasets tailored to their tasks [START_REF] Schulz | Hearing what you cannot see: Acoustic vehicle detection around corners[END_REF][START_REF] Majumder | Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations[END_REF]). For research on robotic echolocation, only very few simple datasets exists.

The BatVision Dataset. Motivated by the success of simulated datasets and the lack of real data for robotic echolocation and audio-visual scene understanding, we recorded the real-life BatVision dataset. We provide large-scale recordings from traversing real campus spaces with a robot, covering a wide range of materials, room shapes and objects, shown in Figure 2. We provide synchronized RGB-D camera images and recorded echoes of chirping sounds emitted with a forward facing speaker. Similar to RIRs, echoes can be used to infer the geometry of the room to allow the robot bat-like perception even in darkness, smoke and fog.

Data was collected at UC Berkeley (52,220 instances) and Mines Paris PSL (3,120 instances). Emitted chirp signals range between 20Hz and 20kHz. RGB-D images were recorded with an Intel Realsense camera mounted on a robot (Mines Paris) and a ZED camera (UC Berkeley).

In the following we place the BatVision dataset among similar ones and argue its unique utility. We then detail the collection process, data and showcase performance of depth prediction methods trained on it. UC Berkeley data will be referred to as BV1 and Mines Paris data as BV2.

II. RELATED WORK

Audio-visual data is available in many datasets, collected in reality or simulation, suitable for different tasks. However, to our knowledge, no existing dataset supports audio-visual 3D scene understanding with large-scale real data.

Acoustic Room Impulse Response Datasets. RIR datasets are used in room acoustics, speech and audio processing, sound localization and separation and virtual reality. RIRs account for the effect of room acoustics on audio signals and describe sound propagation. By convolving with RIRs, acoustic properties, such as reverb, can be transferred onto arbitrary dry sounds. Several datasets exist with RIRs, sampled in real spaces using a complex measurement routine.

The Acoustic Multichannel RIR Dataset [START_REF] Hadad | Multichannel audio database in various acoustic environments[END_REF] contains samples from one rectangular room. Panels, moved between recording sessions, emulate environments such as a small office, meeting or lecture room. The dEchorate dataset [START_REF] Di | dEchorate: a calibrated room impulse response dataset for echo-aware signal processing[END_REF] provides 1.8k sampled RIRs with annotated early echo timings and 3D positions of microphones and sound sources. For the MIRaGe dataset [START_REF] Čmejla | Mirage: Multichannel database of room impulse responses measured on highresolution cube-shaped grid in multiple acoustic conditions[END_REF] 371k RIRs are measured in a dense grid of 4104 source positions in different rooms. [START_REF] Kim | Acoustic room modelling using a spherical camera for reverberant spatial audio objects[END_REF] estimates RIRs from spherical camera images. RIRs were sampled as ground truth in living room style environments using a custom array of 48 microphones and a soundfield microphone. Object shapes are simple rectangles, aligned towards the main axis of the room.

While these datasets were sampled with high quality equipment and allow room geometry and RIRs estimation their environments are rectangular rooms with cuboid objects and simple materials. In contrast, for BatVision we sampled data in real public spaces, selected for their variety of materials, shape and architectural properties.

Audio-Visual Simulation.

Datasets containing simulated data are widely used, costeffective, allow large scale generation, controlled conditions and easy data annotation [START_REF] Sun | SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation[END_REF], [START_REF] Gaidon | Virtual worlds as proxy for multi-object tracking analysis[END_REF]). Simulation of sound propagation has been extensively used in games and AR applications.

SoundSpaces 1.0 [START_REF] Chen | Soundspaces: Audio-visual navigation in 3d environments[END_REF] allows simulating sound propagation by providing RIR renderings built with bidirectional path-tracing in 3D-scanned apartment models. They were generated for discreet positions and orientations in a grid in two 3D environments, Matterport3D [START_REF] Chang | Matterport3D: Learning from RGB-D Data in Indoor Environments[END_REF]) and Replica [START_REF] Straub | The Replica dataset: A digital replica of indoor spaces[END_REF]). SoundSpaces 2.0 [START_REF] Chen | Soundspaces 2.0: A simulation platform for visual-acoustic learning[END_REF] provides continuous on-the-fly rendering and improved the sound propagation. It has since become a widely used benchmark showing impressive performance on a wide range of tasks [START_REF] Zhu | Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision[END_REF][START_REF] Gao | Visualechoes: Spatial image representation learning through echolocation[END_REF][START_REF] Zhu | Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision[END_REF][START_REF] Zhang | Stereo Depth Estimation with Echoes[END_REF][START_REF] Kumar Parida | Beyond image to depth: Improving depth prediction using echoes[END_REF].

While it is a huge contribution to the community, bridging the gap between simulation and the real world stays a significant challenge. Models trained in simulation often overfit to characteristics of the simulator in surprising ways [START_REF] Kadian | Sim2real predictivity: Does evaluation in simulation predict real-world performance?[END_REF]. However, for audio-based 3D scene understanding there is no real-life dataset of a size, comparable to SoundSpaces which motivated our data collection.

Sound Event Localization and Detection Datasets. Sound event localization and detection (SELD) aims to infer the azimuth, elevation and distance of sounds relative to an observer, with or without additional classification. Datasets in the domain contain one or several, moving or static, clear or noisy sounds in different environments. The TUT Acoustic Scenes 2016 dataset [START_REF] Mesaros | TUT database for acoustic scene classification and sound event detection[END_REF] consists of 15 real acoustic scenes, such as City center or Metro station, with annotated sound event classes, onset and offset times, and spatial information. The STARSS22 dataset [START_REF] Politis | STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events[END_REF] is a collection of 22 real-world acoustic scene recordings with sound event annotations, captured with a high-resolution spherical microphone array. Annotations cover 13 classes and direction of arrivals.

For 3D scene understanding and especially reconstruction, SELD datasets often lack sufficient spatial ground truth information of the environments the sounds occur in. For the BatVision dataset forward facing camera images and depth information is provided to allow exploring the correlations in the audio and visual modality.

Real-World Audio-Visual Datasets.

The huge EGO 4D dataset [START_REF] Grauman | Ego4d: Around the world in 3,000 hours of egocentric video[END_REF] contains videos from the perspective of persons performing a wide range of activities. Parts contain audio and 3D meshes of the environment. It is an impressive effort and contribution for many research areas such as activity recognition. In contrast, for BatVision we emit chirps into recorded scenes to allow inferring RIRs and finally the 3D scene. It is magnitudes smaller but better suited for our tasks.

[5] proposed "The Greatest Hits" dataset. By filming objects being struck with a drumstick, they aim to study physical interactions with a visual scene and synthesize plausible impact sounds. It includes 46,577 actions of hitting and scratching objects. The research is a step towards understanding the link between material, action and emitted sound. Similarly, with our recorded echoes we collect interactions between chirps and scene materials.

The authors in [START_REF] Schulz | Hearing what you cannot see: Acoustic vehicle detection around corners[END_REF] predict approaching vehicles at blind intersections from sound before they enter the line-of-sight. They captured crossing vehicles at intersections with a custom microphone array and a front-facing camera mounted on a car. [START_REF] Balajee Vasudevan | Semantic object prediction and spatial sound super-resolution with binaural sounds[END_REF] recorded the "Omni Auditory Perception Dataset" standing next to streets with eight binaural microphones and a 360°camera. Pseudo ground truth depth is predicted from monocular images. The authors emphasize its mid-size and the great effort required to create it.

[17] introduced the "Quiet Campus Dataset" of ambient sounds from a variety of quiet indoor scenes. Paired with RGB-D images, they predict distances to walls from the whirling of a fan or noise coming through a window. While this unique idea shows impressive task performance with passive observations, we focus on active sounds which are humanly audible for improved performance.

Recording interactions of chirps with spaces has a recent history. The BatVision depth-from-binaural audio idea [START_REF] Jesper Haahr Christensen | Batvision: Learning to see 3d spatial layout with two ears[END_REF] inspired [START_REF] Tracy | Catchatter: Acoustic perception for mobile robots[END_REF] to collect similar data with added 360°LiDAR scans for depth and four ear-shaped microphones. With 5,000 samples the dataset falls between BV1 and BV2. [START_REF] Majumder | Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations[END_REF] predict an occupancy map from conversations in spaces. Beyond using SoundSpaces they also captured real data in a mockup apartment, citing the lack of publicly available real world data. To compute RIRs they capture chirps from a speaker with an Eigenmike. The "Studio Dataset" [START_REF] Irie | Co-Attention-Guided Bilinear Model for Echo-Based Depth Estimation[END_REF] records 1478 samples of chirp echoes with four microphones and RGB-D images with a RealSense camera. Everything is fixed into one metal frame facing in one direction. While also testing in SoundSpaces, their real-world recordings are done in one rectangular room showing simple cuboid objects.

The recording of so many datasets, tailored to specific needs show the potential impact that more and larger publicly available datasets can have. Presented tasks from depth prediction to occupancy mapping can be investigated with our BatVision dataset. We will describe in the following how and give details on the extend of our data collection.

III. THE AUDIO-VISUAL BATVISION DATASET

A. Dataset Overview

We collected the Audio-Visual BatVision Dataset at various locations at the UC Berkeley (ICSI), and Ecole des Mines Paris using small robots, carrying all sensors. The sites provide varied architectural styles, room shapes, materials and therefore acoustic impressions.

We crossed lecture halls, corridors, offices and cobblestone paths recording while emitting humanly audible sine swept signals (chirps) with a forward facing speaker (Figure 3). We recorded their echoes, forward facing camera images and provide depth-maps from the same perspective. Chirps ascend from 20Hz to 20kHz in 3 ms. Their use is motivated by their counterparts in nature, helping animals echolocalize, but they are also common in sound engineering to record RIRs [START_REF] Farina | Simultaneous measurement of impulse response and distortion with a swept-sine technique[END_REF]. Binaural microphones record audio at 44.1kHz with 24 bits to keep the full frequency range of the chirps. Each instance is one chirp synchronized with one 1280x720 RGB-D image. An overview of this dataset and other real-world dataset is available in Table I The distribution of the collected depth can be seen at fig. 4 and fig. 5. By comparing the average depth value per instances it shows BV2 is more long-tailed. BV1 depth values are clipped to the BV2 max depth value to remove outliers of the stereovision algorithm used. Pixel-wise average depth shows a more complex scene distribution in BV2. Data collected at the UC Berkeley (BV1). In Berkeley, 52,220 instances were collected at two floors of an institute containing hallways, open areas, conference rooms and offices. At two distinct areas of one floor, 39,564 and 7,618 instances were collected for training and validation and 5,038 instances on another floor for testing. While similar, the floors' spatial layout, furniture, occupancy, and decorations are different, see Figure 2. For details, please see [START_REF] Jesper Haahr Christensen | Batvision: Learning to see 3d spatial layout with two ears[END_REF] showcasing the initial depth prediction-from-audio idea.

A JBL Flip4 Bluetooth speaker emitted chirps while two USB Lavalier MAONO AU-410 microphones, embedded into silicone ears, recorded their echoes. They were mounted 23.5cm apart while the speaker sat between (Figure 3). We excluded motor sounds by pushing the robot on a trolley.

We used a ZED stereo camera to record images of the scene ahead and calculated depth maps with the camera API. Depth-from-stereo fails for some pixel which are NAN. We provide depth maps with RGB images from the left camera.

Designed for smaller spaces, audio recordings were cut at 72.5ms, including echoes from objects at 12m distance. This trades-off perception at a relevant distance while excluding later noisy reverberations. We also clip depth to 12m during training even though further distances are available.

Data collected in Mines Paris (BV2). In Ecole des Mines Paris, we collected 3,120 instances with large visual and acoustic variety (See Figure 2).

We split data into sets of 1,911 train, 625 validation and 584 test instances. Given the multi-modal scene distribution we aim to balance task difficulty. We split instances by time of recording when moving through rooms, avoiding loops. That way, our incremental coverage of rooms will lead to poses being sufficiently separated in the sets. Even if we revisit parts of rooms we chose different trajectories thereby avoiding repeated poses. Tasks on this split will be harder than if instances were randomly assigned, which could result in neighbouring poses ending up in training and test. Simple interpolation will not yield best performance. Admittedly our split is easier than separating complete rooms as done for BV1. We observed outdoor reconstruction performance is acceptable but sub-par suggesting domain shift. If very different features, i.e. outdoors, carpeted rooms or stone corridors, are not in the training data further domain adaptation would be necessary which is left for future work.

We provide monocular RGB images and depth from an Intel RealSense D455 camera. The chirp-emitting speaker is identical to BV1 but we recorded echoes with a binaural 3Dio Free Space microphone. A robot carried all hardware and an Nvidia Jetson TX-1 to run all recording software (see Figure 3). We excluded motor noise by switching between stopping and driving and filtered out instances while moving.

Synchronization was achieved using ROS timestamps and chirps detection with manual checks. Audio data instances were cut to be 0.45s long which includes echoes from objects up to 75m away. Because of larger spaces in BV2, we keep the long tail to sense far away reflection. The maximum depth value of the camera is ≈64m.

B. Limitations

Unlike in simulation, physically recording introduces a number of limitations. Echoes were recorded in real rooms next to noisy streets and with typical sounds of busy academic institutions. Some data may therefore contain audible noise, typical for the recording context. Models trained will need to learn to be robust to these noise profiles.

Some approaches predicting RIRs need independently changing emitter and microphone positions to record sounds on a direct path [START_REF] Di | dEchorate: a calibrated room impulse response dataset for echo-aware signal processing[END_REF][START_REF] Luo | Learning Neural Acoustic Fields[END_REF]. In our data collection by driving, the robot carries all the equipment so the emitter is fixed close to the microphone and only the echoes change.

After filtering instances in motion few remain from the same pose. We kept these to allow learning a noise-model but they can be filtered by thresholding optical flow.

Collection with different hardware for BV1 and 2 leads to different formats. Audio instances collected at UC Berkeley are shorter. Some very bright images from Mines Paris shows slight purple discoloration due to an issues with the Intel RealSense D455 RGB-D camera. Finally, our consumergrade speaker can not produce the full frequency spectrum.

IV. DEPTH PREDICTION ON BATVISION DATA

The data can be used for depth-prediction from audiovisual data with approaches developed for real or simulated data. For illustration we trained a U-Net audio-only depth prediction baseline and compared with one state-of-the-art audio-visual approach developed by its authors in simulation.

Beyond Image to Depth. Recent work improves audiovisual depth prediction using a pre-trained material classifier to decide which modality to pay attention to. The authors [START_REF] Kumar Parida | Beyond image to depth: Improving depth prediction using echoes[END_REF] extract latent material features from SoundSpaces 1.0 camera images and generate attention maps for the visual and audio-stream in the network. They convolve a dry-chirp sound with SoundSpace RIRs to simulate emitting it into spaces from different poses. Captured echoes and RGB-D images make up their dataset with which they predict depth from RGB images and audio.

Training their code on the BatVision dataset shows similar performance as on SoundSpaces (see Table II). Apart from changing the spectrogram resolution, no further adjustments were needed when switching to real data. We ablate results in a study using the audio signal only by setting the RGB images to zero. That way we keep the architecture unchanged.

Qualitatively, fine structures are harder to predict which may stem from our coarser ground truth depth compared to simulation (See Figure 6). Measuring depth in reality, either from stereo (ZED camera) or active stereo (RealSense camera), limits the accuracy depending on the object distance.

Comparable results show that an approach developed for U-Net Baseline. The extend to which audio can complement visual sensor data is not clear. Related work showed repeatedly the quality of the visual signal strongly dominates task performance. The compared work [START_REF] Kumar Parida | Beyond image to depth: Improving depth prediction using echoes[END_REF] investigates material-based attention to let the network choose the influence of each modality more explicitly. We investigate the claim that audio can help in bad visual conditions with a baseline, based on the audio signal alone. This serves to investigate the contribution of the audio signal in isolation with clear quality attribution. However, in all practical applications, the visual signal should be used as well.

We train a U-Net from audio only, similar to related work [START_REF] Jesper Haahr Christensen | Batvision: Learning to see 3d spatial layout with two ears[END_REF][START_REF] Gao | Visualechoes: Spatial image representation learning through echolocation[END_REF] and isolated on BV2 and BV1. We achieve solid results, correctly predicting free space, obstacles and the general room layout (see Figure 7). Without further tweaks such as GCC-Phat features or using a GAN [START_REF] Jesper Haahr Christensen | Batvision: Learning to see 3d spatial layout with two ears[END_REF][START_REF] Jesper Haahr Christensen | BatVision with GCC-PHAT Features for Better Sound to Vision Predictions[END_REF], the performance shows the exploitable quality of the data itself.

Implementation Details. The only input are spectrograms, generated from waveforms with 512 frequency bins (nfft), 64 window and 16 hop length, resized to 256x256.

For BV2, best performance is obtained with depth clipped to 30m and audio cut accordingly. This can be explained as audio energy decays with distance so far traveling echoes are hard to distinguish from noise. For BV1, depth is clipped to 12m and the audio accordingly. Ground truth depth is always normalized using the chosen max depth value.

We use an 8 block U-Net with skip connection. Each encoder block is composed of 2D convolutions, batch normalization and leaky ReLUs. Each decoder block is composed of 2D transposed convolution, batch normalization and ReLUs (see Figure 8 ). We found skip-connections improve performance though contrary to a segmentation task their contribution is not yet clearly understood. We train with 256 batch size, 0.002 learning rate for BV2 and 0.001 learning rate for BV1 with the AdamW optimizer and L1 loss.

Results on Depth Prediction. With this simple baseline we retrieve the general geometry of the space (see Table II and fig. 7 ). On BV2, complex obstacles such as chairs are visible. When the visual sensors fails (e.g on glass), audio gives correct information about the depth. The network generalizes well between different acoustic environment. It reproduces corridors robustly built with various material (e.g carpet floor and glass wall, stone and tiled floor, see Figure 7). Outdoors the performance is diminished with the network underestimating depth systematically. This shows some inevitable bias of the data having a majority of indoor data. Trained on BV1 data, corridors and obstacles are well reconstructed even though finer structures are equally lost.

V. CONCLUSION

Recent success in using simulated audio-visual data for scene understanding shows the potential of the audio modal-ity. Depth prediction from sound alone or in addition to vision is possible, allowing to perceive the environment like a bat. The BatVision dataset will support the research community with large-scale real audio-visual data to improve task performance and uncover novel uses. We present an audio-only depth-prediction baseline as starting point and obtain good results when training a state-of-the-art approach on our data. Future datasets could include ultrasound chirps to enable inaudible human-robot collaboration.

VI. ACKNOWLEDGMENT

Fig. 2 :

 2 Fig. 2: Example scenes from BV1 (left two columns) and BV2 (right four columns). BV1 contains typical office scenes with many corridors and some open spaces. BV2 columns show a wide variety of corridors, with and without carpet, maintenance areas, antique conference rooms and outdoor scenes.

Fig. 3 :

 3 Fig. 3: Recording robot used for BV1 at UC Berkeley (left) and BV2 at Mines Paris (right). Both record binaural audio and RGB-D images yet the hardware setup differs.

Fig. 4 :

 4 Fig. 4: Histogram of average depth per instance. BV2 depth distribution is more long-tailed than BV1, which is consistent with the variety of data.

Fig. 5 :

 5 Fig. 5: Average depth per pixel. In BV1 corridors are often centered, in BV2 depth distribution is more complex.

Fig. 6 :

 6 Fig. 6: Test set results when training Beyond Image to Depth on BatVision V1 (top) and BatVision V2 (bottom), depth in meters. Same hyperparameters as in simulation show well visible general layout and obstacles. Fifth row shows free space between two desks even based on audio-only. Fine structures such as cables are still hard to reconstruct.

Fig. 7 :

 7 Fig. 7: U-Net Baseline depth predictions on BatVision V1 (left) and BatVision V2 (right) ('turbo' colormap). Columns are f.l.t.r. RGB image, depth prediction and ground truth depth. Units in meters. Left, beyond showing correct structures, results improved quantitatively in this larger dataset. Right, obstacles like chairs are reconstructed and corridor layouts are correct, even though wall materials, and hence echoes, vary strongly from carpet to stone.

Fig. 8 :

 8 Fig.8: U-Net Architecture. Input spectrograms are processed to depth maps. Skip-connections used show better performance even though the spatial correspondence between input and output features is not clear.

TABLE I :

 I Overview of Real-World Dataset. Mic. stands for microphone, A. for active and P. for passive sound. Here, by chirp we mean frequency swept signals with different range and parameters. Passive sound means no controlled sound is emitted during the recording. Instances are given either by their number or by the total amount of hours or videos recorded.

	Dataset # Instances	Locations		Visual Labels	Perspective	Audio Labels	Audio Sensors Sound Type
	BV1	52,220		Hallways, areas, conference open rooms, office spaces	Monocular RGB Depth from stereo clipped at 12m	Robot	72.5ms (44.1kHz) binaural separated audio long	2 Hear-shaped Mic.	A. (chirp)
	BV2	3,120		Hallways, outdoors, narrow underground corridors, conference rooms	Monocular RGB Depth from infrared at maximum 64m	Robot	0.45s binaural audio long (44.1kHz)	Hear-shaped binaural Mic.	A. (chirp)
	Acoustic Room Impulse Response Datasets
				Rectangular room.
	[2]	234		Panels typical	emulate acoustic			8 channels RIR (48kHz)	Mic. array	A. (chirp)
				environments
	[8]	371k		Rectangular room. Panels emulate typical acoustic environments			RIR (48kHz)	6 Mic. arrays, One additional mic.	A. white noise) (chirp,
	[16]	≈ 1.8k		Cuboïd room with different wall con-figurations			RIR (48kHz)	Mic. Array	A. white anechoïc speech)	(chirp, noise,
	[7]	?		Living room, con-trolled acoustic en-vironment	Stereo 360°RGB, Depth from stereo	?	RIR		Mic. circle	A. (chirp)
	Sound Event Localization and Detection Datasets
								30s	long
	[4]	≈77k		15 outdoor and in-door acoustic scenes			binaural audio annotated in (44.1kHz)	Binaural mic.	P.
								sound classes
								2x 4-channels
	[26]	≈5h		Indoor rooms	360°RGB		spatial format audio (24kHz) annotated in 13	Spherical Mic. Array (SMA)	P.
								sound classes
	Real-World Audio-Visuals Datasets
	[23]	2,535h videos audio	of with	Wide range of activ-ities, cities and loca-tions	RGB, 3D scans and other	Person	?		Various	P.
	[5]	46,577		Indoor and outdoor scenes	RGB	Person	35s long audio	Mic. attached to camera	A. stuck drumstick) (object with
	[19]	411 recordings video	Urban environments RGB	Car	1s long audio (48kHz)	Mic. array	P.
						RGB,		RIR	and	A.	(chirp,
	[31]	?		Mock-up apartment	Depth from mono,	Rig	conversation	SMA	speaker
						Occupancy map		(16kHz)		conversation)
	[15]	54,250 video segments	Streets of a city cov-ering 165 locations	360°RGB	Rig	3s long 8-channels audio (96kHz)	4 hear-shaped binaural mic.	P.
	[24]	1,478		Near-rectangular re-verberant studio	RGB, Depth	Rig	1s long audio (16kHz)	4 omnidirec-tional mic.	A. (chirp)
	[17]	≈15h		Classroom and hall-ways	RGB, Depth	Robot	2 channels au-dio (16kHz)	Stereo mic.	P.

TABLE II :

 II Depth prediction results from Beyond Image to Depth[START_REF] Kumar Parida | Beyond image to depth: Improving depth prediction using echoes[END_REF], trained on BatVision and simulated data (Replica and Matterport). Similar V1 and Matterport results suggest comparable difficulty of tasks. Our simple U-Net baseline slightly outperforms[START_REF] Kumar Parida | Beyond image to depth: Improving depth prediction using echoes[END_REF] when using audio-only (AO).RMSE↓ REL↓ log10↓ δ 1.25 ↑ δ 1.25 2 ↑δ 1.25 3 ↑

	Replica [18]	0.249	0.118 0.046 0.869 0.943 0.970
	Matterport3D [18] 0.950	0.175 0.079 0.733 0.886 0.948
	BV1	0.901	0.234 0.097 0.688 0.888 0.942
	BV2	2.286	0.323 0.119 0.647 0.834 0.901
	BV1 AO	1.350	0.453 0.159 0.441 0.707 0.843
	BV2 AO	2.878	0.521 0.197 0.430 0.629 0.765
	U-Net BV1 AO	1.336	0.361 0.147 0.508 0.738 0.856
	U-Net BV2 AO	2.676	0.432 0.160 0.497 0.717 0.835
	simulated data can be adapted and works with the same
	hyperparameters on our real dataset.

Center for Robotics, MINES Paris, Université PSL, Paris, France

University of Michigan, Ann Arbor, United States of America * Equal Contribution

We thank all our collaborators, especially Daniel Lin for data collection, Jesper Haahr Christensen for insights into sonar and collaboration on the original idea, Karl Zipser for conception of the robot for BV1. Finally David Mazouz and Jacky Lech for advice and help building the robot for BV2. The authors acknowledges the support of the French Agence Nationale de la Recherche (ANR), under grant ANR-22-CE94-0003 (projet Omni-BatVision).