Cross-Modal Attention for Accurate Pedestrian Trajectory Prediction

Accurately predicting human behavior is essential for a variety of applications, including self-driving cars, surveillance systems, and social robots. However, predicting human movement is challenging due to the complexity of physical environments and social interactions. Most studies focus on static environmental information, while ignoring the dynamic visual information available in the scene. To address this issue, we propose a novel approach called Cross-Modal Attention Trajectory Prediction (CMATP) able to predict human paths based on observed trajectory and dynamic scene context. Our approach uses a bimodal transformer network to capture complex spatio-temporal interactions and incorporates both pedestrian trajectory data and contextual information. Our approach achieves state-of-the-art performance on five real-world pedestrian prediction datasets, making it a promising solution for improving the safety and reliability of pedestrian detection and tracking systems.

Introduction

Accurately predicting human movement has significant applications in various domains, including autonomous driving, surveillance systems, and wheelchair automation. It helps detect potential threats in security, ensures safe navigation in autonomous driving, and provides valuable insights into human-environment interactions for social and behavioral sciences. However, predicting human movement is a challenging task due to dynamic interactions between agents, complex environments, and long-term dependencies. The multimodality of human motion also presents a significant challenge.

Recent research has focused on leveraging the power of deep learning models to improve the accuracy of predicting human movement. Early models, such as Social Forces, had limitations in complex crowded environments. Researchers have since developed sequence prediction methods based on Recurrent Neural Networks (RNNs) [START_REF] Alahi | Social LSTM: Human Trajectory Prediction in Crowded Spaces[END_REF], which performed well for modeling nearby trajectories but could not capture the impact of further pedestrian motion. More recent works have combined temporal encoding of kinematics data using LSTM and spatial feature extraction through convolution networks on image inputs [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF], improving state-of-the-art results. However, these models have limitations in predicting unexpected scenarios, such as sudden changes in motion direction or avoidance of moving obstacles.

To overcome these limitations, we propose a novel approach that utilizes Transformer Networks, which we believe prioritize attentive focus as a crucial aspect in predicting trajectories. While most current methods treat trajectory prediction as time sequence generation using LSTMs or Transformers, our approach fully leverages both the set of coordinates and videos through multimodal transformers. However, despite increasing research in this area, most studies still overlook the dynamic visual information available in the scene, instead focusing on static environmental data. To address this gap, we introduce the Cross-Modal Attention Trajectory Prediction (CMATP) framework, which predicts human paths based on both the observed trajectory and dynamic scene context, leveraging a ResNet and attention mechanism on video input. By doing so, CMATP captures both environmental constraints and social interactions in dynamic scenes, without requiring communication with other humans.

Our approach includes a cross-attention module that integrates trajectory data with contextual information, allowing the network to capture the general temporal consistency of pedestrian movement. By using a convolutional model for feature extraction and a bimodal transformer, CMATP captures intricate spatio-temporal interactions, improving accuracy while maintaining the same computational complexity as using a single data type. The main contribution lies in the ability to leverage the benefits of tow input modalities while avoiding the computational overhead of incorporating additional data types.

Related Work

This paper discusses research trends in human trajectory forecasting, a topic that has garnered interest for over two decades. We identify three major research directions: improving sequence modeling, studying the impact of people's actions on each other, and modeling interactions between people and their environment.

Sequence modeling using RNNs. RNNs are often used to generate sequences, including kinematic trajectory information [START_REF] Alahi | Social LSTM: Human Trajectory Prediction in Crowded Spaces[END_REF][START_REF] Mangalam | It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction[END_REF][START_REF] Zhang | SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction[END_REF]. However, they struggle to capture spatiotemporal interactions among humans in a scene [START_REF] Gupta | Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks[END_REF][START_REF] Amir Sadeghian | SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints[END_REF]. To address this, researchers have proposed augmenting RNNs with pooling [START_REF] Alahi | Social LSTM: Human Trajectory Prediction in Crowded Spaces[END_REF][START_REF] Gupta | Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks[END_REF] or attention [START_REF] Amirian | Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs[END_REF][START_REF] Vemula | Social Attention: Modeling Attention in Human Crowds[END_REF] modules. Recent work [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF] leverages dynamic scene features via a conditional 3D visual encoder based on attention which captures complex interactions. However, RNNs and CNNs have limitations in modeling long-term dependencies and extracting local sequence patterns [START_REF] Wang | Disconnected recurrent neural networks for text categorization[END_REF]. Transformers are argued to be more suitable for sequence modeling and trajectory forecasting, especially with large amounts of data, due to their better capability of learning non-linear patterns.

Social aware models. Pedestrian trajectory prediction can be approached either by modeling pedestrians as a crowd or as individuals. Traditional crowd models [START_REF] Alahi | Socially-aware large-scale crowd forecasting[END_REF][START_REF] Helbing | Social Force Model for Pedestrian Dynamics[END_REF][START_REF] Robicquet | Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes[END_REF][START_REF] Yamaguchi | Who are you with and where are you going?[END_REF] rely on handcrafted kinetic forces and energy potentials to help pedestrians reach their goals while avoiding collisions. But, these methods cannot capture complex interactions in crowded environments. Recent works focus on RNN-based architectures to encode interactions between humans [START_REF] Alahi | Social LSTM: Human Trajectory Prediction in Crowded Spaces[END_REF][START_REF] Lee | DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents[END_REF][START_REF] Lerner | Crowds by Example[END_REF][START_REF] Zhang | SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction[END_REF]. However, RNNs struggle to capture spatio-temporal interactions among pedestrians. Graph representations have been used to capture social interactions [START_REF] Kosaraju | Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks[END_REF][START_REF] Ma | Trafficpredict: Trajectory prediction for heterogeneous traffic-agents[END_REF][START_REF] Mohamed | Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction[END_REF][START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF], but some suffer from limited understanding of the environmental context. Other approaches incorporate models of human interaction with the environment [START_REF] Ivanovic | The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs[END_REF][START_REF] Amir Sadeghian | SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints[END_REF][START_REF] Salzmann | Tra-jectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data[END_REF], such as visual features [START_REF] Choi | Looking to relations for future trajectory forecast[END_REF][START_REF] Hao Xue | SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction[END_REF] and dynamic 3D scene information [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF]. There is criticism of RNNs' ability to model human-human interaction [START_REF] Becker | Red: A simple but effective baseline predictor for the trajnet benchmark[END_REF][START_REF] Schöller | What the constant velocity model can teach us about pedestrian motion prediction[END_REF], with suggestions that it limits the model's generalization capability [START_REF] Schöller | What the constant velocity model can teach us about pedestrian motion prediction[END_REF]. Transformer-based methods [START_REF] Su | Pedestrian Trajectory Prediction via Spatial Interaction Transformer Network[END_REF][START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF][START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF] are seen as potentially more suitable for trajectory forecasting, but these methods make predictions based solely on past trajectories, which may fail to detect unpredictable sharp turns, suggesting that additional information, such as environmental configuration, should be incor-porated. Our work focuses on predicting individual pedestrian motion, sidestepping social and environmental interactions. Fascinatingly, our approach achieves the best performance on the toughest benchmark.

Context aware models. Context-aware trajectory prediction models aim to incorporate physical scene information, such as crosswalks and roads. Previous methods have been proposed to extract and integrate static scene information [START_REF] Lee | DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents[END_REF][START_REF] Amir Sadeghian | SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints[END_REF][START_REF] Salzmann | Tra-jectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data[END_REF]. Recent models used dynamic spatial and temporal context [START_REF] Cheng | Exploring Dynamic Context for Multi-path Trajectory Prediction[END_REF][START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF]. However, all these models suffer from limitations related to memory and computational complexity. For example, [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF] employs 3D-CNNs, but these models can be computationally expensive and require large amounts of memory because they process volumetric data, which can be larger than the 2D images used in traditional CNNs. Incorporating additional visual modalities can significantly improve performance compared to those only trajectory-based methods [START_REF] Alahi | Social LSTM: Human Trajectory Prediction in Crowded Spaces[END_REF]. However, existing networks often merge features from different modalities through a simple concatenation in the fusion mechanism. However, this approach lacks the ability to capture the interaction between various granular motion features and does not effectively mine the characteristics and relations of distinct modalities.

After reviewing existing research, we found that pedestrian behavior prediction can greatly benefit from the use of Transformer models and attention mechanisms, as well as the inclusion of contextual information and observed trajectory. To address these challenges, we propose a novel model that incorporates all of these features and utilizes a co-attentional mechanism for capturing dynamic motion information. Our model provides a solution to the limitations of existing methods and has the potential to significantly improve pedestrian behavior prediction.

Approach

Problem Formulation

The aim of this work is to predict the future positions of individuals in a scene using a transformer-based framework. During training, the method requires preprocessed videos with human detection and tracking algorithms, but during testing, it relies on the target human's trajectory.

At any time-instant t, the i th person in the scene is represented by his/her xy-coordinates (x

(i) t , y (i) t ).
We observe the positions of all individuals from time 1 to T obs , and predict their positions for time instants t obs + 1 to t pred . In formal terms, we denote the 2D position of human i at frame t by: u

(i) obs = (x (i) t , y (i) t ) ∈ R 2 .
Assume we observe trajectories and the scene from frame 1 to t obs . We represent the observed sequence for a person, denoted as i, using

T (i) obs = (u (i) 1 , ..., u (i)
t obs ), and future positions by T (p) pred = (u (i) t obs+1 , ..., u (i) t pred ).

Overview

In order to enhance the precision of pedestrian trajectory forecasting, the proposed model (CMATP) employs a bimodal encoder-decoder architecture with a cross-modal attention mechanism, which handles two modalities: kinematic and visual information. The (CMATP) model has two parallel encoder branches (Figure 1). The first branch utilizes self-encoding to transform the pedestrian trajectory τ into a latent vector X kin , while the second branch extracts visual information through a feature extraction process using a pre-trained convolutional neural network, specifically a ResNet50. The resulting feature vector ν is then passed through a fully connected layer and self-attention block to generate a latent vector X vis that encodes both visual and temporal information. A cross-attention block is introduced to capture the relationship between the kinematic X kin and visual latent X vis vectors outputted by the top and bottom self-attention modules, respectively. This cross-attention mechanism effectively improves the accuracy of future trajectory prediction. The proposed method introduces a novel approach to spatio-temporal attention modeling by decomposing it into two parts: kinematic modeling and contextual modeling. Kinematic modeling employs a temporal Transformer network, which is more effective than RNNs in capturing temporal dependencies from individual trajectory data. Contextual modeling, on the other hand, introduces a Transformer-based encoder module that encodes contextual information from video data to enhance the attention mechanism. To predict human trajectory, the method employs two encoder modules joined by a cross-modal attention mechanism, which is then used with a decoder transformer. The method argues that attention is a crucial component for effective and efficient trajectory prediction.

Positional encoding. The transformer architecture uses multi-head self-attention instead of the recurrence mechanism, resulting in faster training and the ability to capture longer dependencies. Positional encoding is a vital component of the transformer, which incorporates sequence order information by adding positional embeddings to input embeddings.

Attention module. The use of self-attention mechanisms improves the model's ability to capture long-term dependencies, model complex interactions. In particular, attention modules capture the context and the effect of past data on the current one by dividing the sequence entries into Query (Q), Keys (K), and Values (V). The Query is compared to all other Keys using a scaled dot product, and the output determines the weights assigned to the Values. Attention is therefore given by the equation:

Attention(Q, K,V ) = So f tmax( QK T √ d k ),
where d k is the embedding dimension which is equal for both the Q and K.

Cross-attention mechanism. Cross-attention mechanism allows the model to be more contextually aware, as it takes into account the effect of visual context on pedestrian trajec-tories. This is particularly important in crowded scenes where the visual environment plays a significant role in predicting pedestrian movements. Figure 1 illustrates the operation of cross-attention.

Training method/ Loss function. As prior work [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF], our loss function consists of two components -the mean-squared loss and a regularization term called Lreg, which regulates the smoothness of future trajectories. In training our network, we use the following loss function: Lmodel = Lmse + λ Lreg, where λ is a regularization parameter. We kept the value of λ fixed at 0.5 in our experiments to avoid restricting the model's ability to capture sudden changes in the target pedestrians' trajectory. Lmse is calculated as the average of the squared differences between predicted and observed values, while Lreg is calculated as the sum of Euclidean distances between each step of the predicted trajectory and a line fitted to the observed trajectory. In our experiments, we sample 20 future trajectories and select the top 5 trajectories that are closest to the ground-truth to calculate L mse . Through empirical observation, we have found that this approach enables our network to converge more quickly while producing more accurate predictions.

Experiments

Experimental Setup

Datasets. Our approach was evaluated on well-established public human-trajectory datasets, namely ETH [START_REF] Pellegrini | You'll never walk alone: Modeling social behavior for multi-target tracking[END_REF] and UCY [START_REF] Lerner | Crowds by Example[END_REF] datasets, which are widely-used benchmarks for pedestrian motion prediction. These datasets were acquired from surveillance videos of pedestrians walking on sidewalks and annotated with location coordinates. The datasets contain realworld pedestrian trajectories in top-view coordinates expressed in meters, with rich humanhuman and human-object interaction scenarios. The acquisition was done using a fixed camera on 5 different scenarios captured at 2.5 Hz, and an image is annotated with pedestrian positions every 0.4 seconds. ETH/UCY consists of five different scenes. The ETH dataset consists of two scenes (ETH and Hotel) taken from a bird's eye view, with hundreds of pedestrian trajectories engaged in walking activities. The UCY dataset provides three scenes (Zara1, Zara2, and Univ) taken from a bird's eye view with standing/walking activities.

Evaluation Metrics. Similar to existing works [START_REF] Alahi | Social LSTM: Human Trajectory Prediction in Crowded Spaces[END_REF][START_REF] Amirian | Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs[END_REF][START_REF] Gupta | Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks[END_REF][START_REF] Ivanovic | The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs[END_REF][START_REF] Kosaraju | Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks[END_REF][START_REF] Lee | DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents[END_REF][START_REF] Li | Conditional Generative Neural System for Probabilistic Trajectory Prediction[END_REF][START_REF] Amir Sadeghian | SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints[END_REF][START_REF] Salzmann | Tra-jectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data[END_REF][START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF][START_REF] Zhao | Multi-Agent Tensor Fusion for Contextual Trajectory Prediction[END_REF], our method is evaluated using two widely used metrics in the field, namely the Average Displacement Error (ADE) and the Final Displacement Error (FDE). ADE is defined as the average L2 distance (in meters) between the actual trajectory and the predicted trajectory at each time step of the trajectory from T obs+1 to T pred on average over all pedestrians. FDE is defined as the Euclidean distance (in meters) between the ground truth (actual position) and the prediction (predicted position) at the last time step of the prediction T pred , averaged over all pedestrians. Formally:

ADE = ∑ n i=1 ∑ T pred t=T obs+1 || Ŷ i t -Y i t || n * T ; FDE = ∑ n i=1 || Ŷ i T pred -Y i T pred || n
Where n represents the number of pedestrians, Ŷ i t are the predicted coordinates for pedestrian i at time t, Y i t are the real future positions, and || is the Euclidean distance. T pred is the final pedicted timestep. T is the prediction horizon.

Evaluation mehod. For benchmarking purposes, we follow a similar evaluation method as in prior works (See Table 1) When evaluating trajectory forecasting models, the time horizon of the forecast is crucial, as different objects move at different speeds. The choice of time horizon should depend on the class of objects being considered. To have a fair comparison with all the existing works, we observe each training trajectory for 8 times-steps (3.2 seconds) and evaluate the model's performance by measuring prediction errors for the next 12 time-steps (4.8 seconds). To fully utilize the datasets during model training, we adopt a leave-one-out approach for evaluation that has been commonly used in previous studies. We train our model on four sets of data and evaluate it on the remaining set. We repeat this process for all the 5 sets.

Implementation details. Our model is based on the original Transformer Networks architecture [START_REF] Vaswani | Attention is All you Need[END_REF] with a dimension of model of 512 and 6 layers with 8 heads. We trained the entire network end-to-end with a batch size of 40 for 400 epochs, using stochastic gradient descent (SGD) optimizer with a learning rate scheduler and two mean squared error (MSE) loss functions. The learning rate is adjusted every 40 steps with an initial learning rate of 0.01 and the maximum gradient value is clipped to 1 to prevent gradient explosion. We adopted the teacher force strategy and used our proposed loss function with a value of λ = 0.5 to achieve faster convergence as prior work [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF]. The model was implemented using PyTorch on an Ubuntu server equipped with an NVIDIA TITAN RTX GPU and 24 GB RAM.

Results

Baseline. For evaluation purposes, we generate 20 predictions for each observed trajectory and select the prediction closest to the ground truth. This evaluation technique enables us to examine the multi-modality and diversity of the predictions. We evaluate our approach against six deterministic baselines, which are linear regression, LSTM, Social-LSTM [START_REF] Alahi | Social LSTM: Human Trajectory Prediction in Crowded Spaces[END_REF], Social ATTN [START_REF] Vemula | Social Attention: Modeling Attention in Human Crowds[END_REF], TrafficPredict [START_REF] Ma | Trafficpredict: Trajectory prediction for heterogeneous traffic-agents[END_REF], and SR-LSTM [START_REF] Zhang | SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction[END_REF]. We also compare our approach against various generative baselines [START_REF] Amirian | Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs[END_REF][START_REF] Giuliari | Transformer Networks for Trajectory Forecasting[END_REF][START_REF] Gupta | Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks[END_REF][START_REF] Ivanovic | The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs[END_REF][START_REF] Kosaraju | Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks[END_REF][START_REF] Kothari | Human trajectory prediction using adversarial loss[END_REF][START_REF] Lee | DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents[END_REF][START_REF] Liang | Peeking into the future: Predicting future person activities and locations in videos[END_REF][START_REF] Mangalam | It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction[END_REF][START_REF] Mohamed | Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction[END_REF][START_REF] Amir Sadeghian | SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints[END_REF][START_REF] Salzmann | Tra-jectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data[END_REF][START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF][START_REF] Xu | GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning[END_REF][START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF][START_REF] Yuan | AgentFormer: Agent-Aware Transformers for Socio-Temporal[END_REF][START_REF] Zhao | Multi-Agent Tensor Fusion for Contextual Trajectory Prediction[END_REF] using various approaches such as LSTM, GAN, spatio-temporal GCNs, and transformers to predict human trajectories.

Quantitative Analysis. In Table 1, we reporte obtained results against state-of-the-art approaches as mentioned above, using the best-of-20 protocol, which involves sampling 20 possible future trajectories and selecting the one with the best test performance. Our proposed method achieves outstanding performance, ranking either first or second among state-of-the-art methods. In particular, on the FDE metric, our method significantly outperforms existing algorithms on 4 out of 5 datasets, achieving the best average error of 0.33. On the ADE metric, the proposed method outperforms existing algorithms on 3 out of 5 datasets and achieves an average ADE error of 0.22 across all 5 datasets. The University dataset has higher displacement errors compared to other datasets, making it challenging to predict future trajectories accurately. Our method remains comparable to other existing approaches but outperforms all the dense interaction-based methods like S-GAN, Sophie, S-BiGAT, S-STGCNN, and Social Ways. The Hotel dataset has many pedestrians waiting for trains, resulting in limited motion. Therefore, most methods, including ours, achieve relatively small displacement errors by predicting small motions accurately. Our proposed method achieves the lowest FDE (0.16) and ADE (0.11) errors on this dataset. The ETH dataset often produces larger displacement errors, which is a common occurrence among many models, due to lower frequency of video frames and kinematic data. However, our All the stochastic models sample 20 possible trajectories and report the best result using a best-of-20 protocol. All models observe 8 frames and forecast the subsequent 12 frames. method achieves the lowest ADE/FDE errors on the ETH dataset, showing the effectiveness of our approach, especially the cross-attention module, in capturing and incorporating information about the movements and behaviors of neighboring pedestrians. The inferior performance of our model without cross-attention in table 2 confirms this.

When comparing individual approaches, the transformer predictor outperforms any individual LSTM-based approach. Specifically, Transformer-TF performs better than Social-LSTM and has a significant advantage over Social-ATTN in terms of FDE. However, on the Zara1 dataset, which is the least structured dataset in the benchmark and mostly consists of straight lines, LSTM-based methods like Introvert perform better than transformer based methods, achieving the lowest ADE (0.16) compared to 0.19 achieved by our proposed TFbased method. Our approach shares similarities with Transformer-TF, which utilizes an encoder-decoder transformer architecture. However, we have enhanced our model by incorporating contextual information in addition to the pedestrian positions. As seen, our approach outperforms previous Transformer-based methods such as Transformer-TF, STAR, and AgentFormer on the ETH and UCY datasets. Our cross-attention + Transformer encoder/decoder structure explores better dynamic context between agents than Transformer encoder/decoder in terms of trajectory prediction. Overall, our model offers a competitive alternative to graph-based methods [START_REF] Kosaraju | Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks[END_REF][START_REF] Mohamed | Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction[END_REF] and has the potential to improve trajectory prediction accuracy.

Qualitative Analysis. We conducted a qualitative analysis of our approach's predictions to gain a more comprehensive understanding of its performance. Figure 2 showcases the qualitative outcomes of our trajectory prediction on multiple videos from the ETH and UCY datasets, providing visual evidence of its effectiveness in accurately predicting pedestrian trajectories. Each column contains two plots showcasing two different pedestrians from the same dataset. In most cases, our method is able to accurately predict the future positions of pedestrians in the scene. The examples in Figure 2 show different scenarios, such as human-human interaction, human-space interaction, and avoiding obstacles. For example, the bottom example in Zara1 demonstrates our model's success in predicting that the target pedestrian will go through the door of the store on the left side of the scene. In the top example in Zara2, our method correctly predicts that the target human entering the scene will avoid a car and turn left. Also, for the bottom example in Hotel, our method correctly predicts that the target person entering the scene will avoid a pole and will continue straight towards the train. In the two cases from ETH, our method correctly predicts that the target human entering the scene will avoid an obstacle and turn right/left. Finally, in the top example from the Univ, we see an instance of human-human interaction, where the target pedestrian slows down before reaching a group of standing people, bypasses them from the left side, and then speeds up. In such crowded scenes, our method is able to capture interactions and predict future positions effectively.

While our model's predictions closely matched the ground-truth data in most cases, there were scenarios where our predictions were not as precise as we had hoped, such as in the bottom example from the University. However, our approach still captured some of the essential features of the pedestrian's behavior, demonstrating its effectiveness in capturing the underlying dynamics of the scene. Ablation Study. Here, we investigate the effect of the Cross Attention module in the design of trajectory prediction models. We performed w/o cross attention, a variant test where we removed the cross attention and concatenated the encoder stream outputs. Results in Table 2 provide insight into the model design for trajectory prediction tasks. Based on analysis of 5 datasets, Cross Attention improved our approach's performance in predicting accurate trajectories in real-world traffic scenes, outperforming alternatives like concatenation. Results showed our approach with Cross Attention significantly reduced errors to 0.22/0.33 compared to 0.26/0.40 without Cross Attention across 4 out of 5 datasets. However, the Univ dataset presented a unique challenge due to higher crowd density and increased uncertainty of future predictions, resulting in comparable error rates between the two models. Further investigation is required to identify reasons behind this discrepancy. Overall, our transformer architecture with Cross Attention enabled smoother temporal predictions and learning of complex sequential patterns, outperforming the baseline model.

Discussion. According to our comparison, CMATP demonstrates the following key points. First, it predicts accurate trajectories in real-world traffic scenes, surpassing the state-of-the-art methods on 4 out of 5 datasets while achieving comparable performance on the remaining dataset. Second, it incorporates a transformer architecture with cross attention to learn interaction, which enables a smoother temporal prediction and outperforms other attention mechanisms, such as additive or multiplicative attention, allowing the model to selectively focus on the most relevant parts of the input sequence. Third, the transformer architecture allows for capturing long-term dependencies and modeling complex interactions between agents in the scene. Fourth, it takes advantage of the transformer's architecture and considers context, which is crucial for accurate trajectory prediction in real-world traffic scenes. Finally, CMATP demonstrates the effectiveness of incorporating a transformer architecture with cross attention in learning interaction and improving model performance.

Conclusion

In this paper, we have proposed a novel approach called CMATP, which utilizes an attentionbased Transformer Network for pedestrian trajectory prediction. Our framework employs attention mechanisms on dynamic scene context and a cross-attention mechanism to capture complex relationships between the inputs (positions and context), resulting in improved overall performance. The model can produce future-conditional predictions that respect dynamic constraints and full probability distributions, making it suitable for robotic tasks. Our study demonstrates the effectiveness of the Cross Attention mechanism in enhancing the model's performance. Despite discrepancies between predicted and ground-truth trajectories that may be attributed to the multi-modal nature of pedestrian paths in diverse environments, CMATP has significant potential to advance the field of pedestrian trajectory prediction and contribute to the development of safer and more efficient transportation systems. Future work will focus on exploring more sophisticated attention mechanisms, larger training datasets, multi-class settings, and additional contextual information (such as weather and time of day) to enhance our model's prediction capabilities. We also plan to leverage hierarchical modeling techniques to improve the model's accuracy further.

Figure 1 :

 1 Figure 1: Overview of CMATP approach.

Figure 2 :

 2 Figure 2: Illustration of the prediction trajectories. yellow dots represents the past observed while red & green dots represent our prediction and the ground truth.

Table 1 :

 1 The average/final displacement error (ADE/FDE) metrics for several methods compared to our model are shown. Lower is better. The models with * have deterministic outputs.

				Performance ADE/FDE ↓ (m)
	Method	Univ	Zara1	Zara2	Hotel	ETH	Avg
	Linear*	0.82/1.59 0.62/1.21 0.77/1.48 0.39/0.72 1.33/2.94 0.79/1.59
	LSTM*	0.61/1.31 0.41/0.88 0.52/1.11 0.86/1.91 1.09/2.41 0.70/1.52
	Social-LSTM* [2]	0.67/1.40 0.47/1.00 0.56/1.17 0.79/1.76 1.09/2.35 0.72/1.54
	Social-ATTN* [29]	0.33/3.92 0.20/0.52 0.30/2.13 0.29/2.64 0.39/3.74 0.30/2.59
	TrafficPredict* [17]	3.31/6.37 4.32/8.00 3.76/7.20 2.55/3.57 5.46/9.73 3.88/6.97
	SR-LSTM* [37]	0.51/1.10 0.41/0.90 0.32/0.70 0.37/0.74 0.63/1.25 0.45/0.94
	DESIRE [13]	0.59/1.27 0.41/0.86 0.33/0.72 0.52/1.03 0.93/1.94 0.53/1.11
	Social-GAN [8]	0.60/1.26 0.34/0.69 0.42/0.84 0.72/1.61 0.81/1.52 0.58/1.18
	FSGAN [12]	0.54/1.14 0.35/0.71 0.32/0.67 0.43/0.89 0.68/1.16 0.46/0.91
	SoPhie [23]	0.54/1.24 0.30/0.63 0.38/0.78 0.76/1.67 0.70/1.43 0.54/1.15
	Trajectron [10]	0.54/1.13 0.43/0.83 0.43/0.85 0.35/0.66 0.59/1.14 0.47/0.92
	MATF [38]	0.44/0.91 0.26/0.45 0.26/0.57 0.43/0.80 1.01/1.75 0.48/0.90
	Next [16]	0.60/1.27 0.38/0.81 0.31/0.60 0.30/0.59 0.73/1.65 0.46/1.00
	Social-BiGAT [11]	0.55/1.32 0.30/0.62 0.36/0.75 0.49/1.01 0.69/1.29 0.48/1.00
	Social-STGCNN [19]	0.44/0.79 0.34/0.53 0.30/0.48 0.49/0.85 0.64/1.11 0.44 / 0.75
	Social Ways [3]	0.55/1.31 0.44/0.64 0.51/0.92 0.39/0.66 0.39/0.64 0.46/0.83
	PECNet [18]	0.35/0.60 0.22/0.39 0.17/0.30 0.18/0.24 0.54/0.87 0.29/0.48
	M2P3 [21]	0.64/1.34 0.45/0.95 0.37/0.79 0.54/1.13 1.04/2.16 0.60/1.27
	Transformer-TF [7]	0.35/0.65 0.22/0.38 0.17/0.32 0.18/0.30 0.61/1.12 0.31/0.55
	STAR [34]	0.31/0.62 0.26/0.55 0.22/0.46 0.17/0.36 0.36/0.65 0.26/0.53
	AgentFormer [35]	0.25/0.45 0.18/0.30 0.14/0.24 0.14/0.22 0.45/0.75 0.23/0.39
	Trajectron++ [24]	0.30/0.54 0.25/0.41 0.18/0.32 0.18/0.28 0.67/1.18 0.32/0.55
	SGN LSTM [36]	0.48/1.08 0.30/0.65 0.26/0.57 0.63/1.01 0.75/1.63 0.48/0.99
	Introvert [26]	0.20/0.32 0.16/0.27 0.16/0.25 0.11/0.17 0.42/0.70 0.21/0.34
	GroupNet [31]	0.26/0.49 0.21/0.39 0.17/0.33 0.15/0.25 0.46/0.73 0.25/0.44
	Our model (CMATP) 0.37/0.52 0.19/0.27 0.14/0.21 0.11/0.16 0.32/0.51 0.22/0.33

Table 2 :

 2 Ablation study on the ETH/UCY datasets. CA denotes Cross Attention.

				Performance ADE/FDE ↓ (m)
	Method	Univ	Zara1	Zara2	Hotel	ETH	Avg
	Ours w/o CA (BTT) 0.36/0.52 0.19/0.29 0.15/0.23 0.12/0.17 0.48/0.81 0.26/0.40
	Ours (CMATP)	0.37/0.52 0.19/0.27 0.14/0.21 0.11/0.16 0.33/0.53 0.22/0.33


Mayssa Zaier

Hazem Wannous
email: hazem.wannous@imt-nord-europe.fr

Hassen Drira

Jacques Boonaert

A Dual Perspective of Human Motion Analysis -3D Pose Estimation and 2D Trajectory Prediction

Anticipating human motion based on given sequences is a challenging and crucial task in computer vision and machine learning, enabling machines to understand human behaviors effectively. Precise prediction of human pose and motion trajectory holds great significance for various applications, including autonomous driving, robotics, and virtual reality. This paper presents a novel approach to address the interconnected tasks of estimating human motion, represented as 3D poses or 2D trajectories, and predicting future motions using 2D images and human pose/position sequences jointly. We propose an encoder-decoder architecture that leverages Transformer networks with a selfattention mechanism, utilizing visual context features, combined with an LSTM to model human motion kinematics. Our approach demonstrates consistent and remarkable improvements over existing methods, both quantitatively and qualitatively. Extensive experiments conducted on diverse public datasets, such as GTA-IM and PROX for 3D human pose estimation, and ETH and UCY combined datasets for 2D trajectory prediction, showcase that our method substantially reduces prediction errors compared to the current state-of-the-art methods.

Introduction

Predicting human motion accurately plays a pivotal role in various fields, such as computer vision, autonomous driving [13,9], intelligent robots [START_REF] Hema | Anticipating human activities using object affordances for reactive robotic response[END_REF], human-robot collaboration [START_REF] Mao | Multi-level motion attention for human motion prediction[END_REF][START_REF] Liu | Human motion prediction using adaptable recurrent neural networks and inverse kinematics[END_REF], virtual reality [START_REF] Pohl | Neural network 3d body pose tracking and prediction for motion-to-photon latency compensation in distributed virtual reality[END_REF], and can occur in different environments, such as streets, airports and sports arenas. The task involves two main aspects: 3D poses estimation and 2D trajectories prediction. These tasks are interrelated and essential for understanding human movements and enabling machines to interact intelligently with humans. In the context of 3D poses estimation, the goal is to focus on reconstructing the three-dimensional positions and orientations of human joints from a given sequence of data, such as images or videos. On the other hand, 2D trajectories prediction aims to forecast future paths or movements of humans in a two-dimensional space based on their past behaviors.

Human motion can be interpreted and represented in various ways, including kinematics, joints graph, video frames, and moving from one point to another as a mass point, each reflecting distinct understandings of human motion and leading to diverse predictions of it. Three primary types of motion trajectory prediction are discussed in the literature: 2D motion trajectory prediction [2, [START_REF] Gupta | Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks[END_REF][START_REF] Amir Sadeghian | Sophie: An attentive gan for predicting paths compliant to social and physical constraints[END_REF], video prediction [START_REF] Zhang | Video-based reconstruction of smooth 3d human body motion[END_REF][START_REF] Shi | Motionet: 3d human motion reconstruction from monocular video with skeleton consistency[END_REF], and 3D poses sequence prediction [10,33,[START_REF] Mao | Learning trajectory dependencies for human motion prediction[END_REF]56,[START_REF] Cao | Long-term human motion prediction with scene context[END_REF]34,11].

To accurately predict human behavior, it is crucial to consider not only the movements of individuals but also the physical surroundings, including obstacles, trees, buildings, and any other scene elements. The combination of these factors makes forecasting human behavior in crowded settings a formidable undertaking. For instance, the complex nature of human intentions which act as internal stimuli that can influence behavior. Additionally, external factors in the physical world can also actively or passively impact human motion. Previous approaches have explored various methods for integrating scene information, including static scene information extraction [22,[START_REF] Amir Sadeghian | Sophie: An attentive gan for predicting paths compliant to social and physical constraints[END_REF][START_REF] Salzmann | Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data[END_REF] and dynamic spatial and temporal context utilization [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF]7]. However, these models often suffer from limitations concerning memory and computational complexity.

Remarkable progress has been made in predicting human pose sequences with the emergence of deep learning. While predicting human motion, many methods tend to overlook the fact that humans move around in three-dimensional spaces, which leads to ignoring the crucial interactions between humans and their surroundings. Zhang et al. [56], detect human bounding boxes across various time frames and use the changing appearance of the human form to generate their predictive signal, but they do not incorporate the background image. Mao et al. [START_REF] Mao | Learning trajectory dependencies for human motion prediction[END_REF] use the skeleton's spa-tial configuration explicitly in the deep architecture itself, which provides valuable information that leads to more accurate predictions. The joints of the skeleton are related to each other in terms of their angles and distances, and introducing this information to the deep model constrains its output from producing random points far from the ground truth. More other methods for estimating 3D human motion such as TR [47], VP [START_REF] Pavllo | 3d human pose estimation in video with temporal convolutions and semi-supervised training[END_REF], and LTD [START_REF] Mao | Learning trajectory dependencies for human motion prediction[END_REF] have also been evaluated for this task. Although significant advancements have been achieved, these approaches have certain limitations since they exclusively rely on kinematic data and neglect to consider information from alternative sources. Nevertheless, incorporating the scene context by including an embedding of a 2D scene image, a 3D scene, or a specific object as an additional input to the model, is considered to be an advantageous approach. This issue has recently addressed by introducing a novel long-term trajectory prediction problem that focuses on deriving 3D poses from 2D poses [START_REF] Cao | Long-term human motion prediction with scene context[END_REF] . To accomplish their task, they proposed a threestage deep framework named GPP-Net, which utilizes a single scene image and 2D pose histories, the framework sequentially predicts 3D human pose sequences along each path. Meanwhile, Zhe et al. [START_REF] Cao | Long-term human motion prediction with scene context[END_REF] utilize the visual signal of the observed sequence to incorporate information about the environment's objects and geometry, which helps to eliminate ambiguity in some prediction scenarios. Recently, Mohamed et al. [34] formulate the problem as an end-to-end spatio-temporal graph while learning a suitable graph adjacency kernel to capture the interaction among the joints of the skeleton by exploiting their spatial configuration. Additionally, the authors fuse the vision signal into the model. While previous methods have shown remarkable progress in predicting human pose sequences using deep learning, they often overlook the fact that humans move in threedimensional spaces, leading to limited consideration of interactions with their surroundings.

In this article, we propose a new approach which: first, addresses the two tasks: 3D poses estimation and 2D trajectories prediction. Second, it leverages the scene context to enhance both 2D trajectory and 3D human pose predictions. We argue that the necessary information for human motion and interaction, along with environmental constraints, can be effectively extracted from the video itself, rather than individual frames. This is because kinematic trajectories are derived from videos and cannot contain more information than the video. As a result, we adopt a self-attention mechanism to capture relevant features from the video context and investigate its interaction with human motion kinematics in spatial and temporal dimensions. The dynamic scene context is introduced in many existing trajectory prediction methods. However, complex architectures like 3DCNN [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF][START_REF] Zhao | Multi-Agent Tensor Fusion for Contextual Trajectory Prediction[END_REF] are often used to describe the context. Having a simple architecture using a 2D CNN combined with a trans-former, it allows to: (1) capture the dynamic context of the scene by taking into account the observed poses / trajectories and the video streams, (2) better understanding of the scene taking advantage of static elements and moving objects, (3) demonstrate improved performance in scenarios with rapid motion changes, predicting sharp turns and avoiding moving obstacles.

Related Work

This section provides an overview of the existing research in the field of estimating human motion, which encompasses two primary aspects: 3D poses estimation and 2D trajectories prediction.

2D trajectory prediction. Early methods for trajectory prediction [START_REF] Helbing | Social Force Model for Pedestrian Dynamics[END_REF][START_REF] Alahi | Socially-aware large-scale crowd forecasting[END_REF][START_REF] Yamaguchi | Who are you with and where are you going?[END_REF][START_REF] Robicquet | Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes[END_REF] used physical forces and Hidden Markov models to model social-scene interactions. However, data-driven approaches using neural networks that capture multi-modal interactions between the scene and agents have become dominant. Recent works focus on RNN-based architectures [START_REF] Lerner | Crowds by Example[END_REF]3,[START_REF] Zhang | SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction[END_REF]22] to encode interactions between humans, but RNNs struggle with spatio-temporal interactions.

Graph representations [29,[START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF]20,[START_REF] Mohamed | Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction[END_REF] capture social interactions, but some methods lack environmental context understanding. Other approaches incorporate human-environment interaction models [42,18,[START_REF] Salzmann | Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data[END_REF] with visual features [START_REF] Hao Xue | SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction[END_REF][START_REF] Choi | Looking to relations for future trajectory forecast[END_REF] and dynamic 3D scene information [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF]. Transformer-based methods [START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF][START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF]46] are seen as potentially more suitable for trajectory forecasting, but solely relying on past trajectories may fail to detect unpredictable sharp turns, suggesting the need to incorporate additional information, such as environmental configuration. Our work focuses on predicting individual pedestrian motion, achieving the best performance on challenging benchmarks. Context-aware trajectory prediction models aim to incorporate physical scene information, such as crosswalks and roads. Previous methods have proposed extracting and integrating static scene information [22,42,[START_REF] Salzmann | Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data[END_REF], while recent models have explored dynamic spatial and temporal context [7,[START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF]. However, these models suffer from limitations related to memory and computational complexity. For instance, employing 3D-CNNs can be computationally expensive and require large amounts of memory due to processing volumetric data [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF].

3D pose estimation. The challenge of predicting 3D human motion involves estimating both the pose and skeleton of the target subject, which is more complex than 2D pose estimation due to a larger 3D pose space and increased ambiguities. Various approaches have been proposed to address this challenge. Some methods directly infer 3D coordinates from 2D RGB images through joint points regression and point detection tasks [26], while others explore intermediate 2D pose approximations [START_REF] Chen | 3d human pose es-timation= 2d pose estimation+ matching[END_REF] instead of directly inferring the 3D pose from an image. However, these approaches may not fully consider the image context, leading to inconsistencies in predicted human motion with the scene. To address this limitation, some works utilize the static 3D layout of the scene to improve pose estimation from monocular images by considering environmental constraints [START_REF] Hassan | Resolving 3d human pose ambiguities with 3d scene constraints[END_REF]. Additionally, recent studies define tasks for predicting 3D human motion over extended periods while accounting for the surrounding environment and employ scene context for achieving goal-oriented motion prediction [START_REF] Cao | Long-term human motion prediction with scene context[END_REF]. CNN graphs and deep graph networks have also been effective in classifying human skeleton actions and predicting 3D human motion based on past movements [START_REF] Yan | Spatial temporal graph convolutional networks for skeleton-based action recognition[END_REF]17,[START_REF] Li | Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction[END_REF][START_REF] Zhong | Spatio-temporal gating-adjacency GCN for human motion prediction[END_REF].

State-of-the-art works address these two applications of human motion analysis and estimation separately. We propose in this work an approach which, on the one hand tackles the two problems, and on the other hand addresses the limitations of existing methods, in particular those relating to the modality of motion and context description.

Approach

Problem Formulation

The problem addressed in this study is the prediction of human motion using two main inputs: data and video context. Our model's estimation of movement depends on the specific target representation, which can be either 3D pose prediction or 2D trajectory prediction. The data can be in the form of 2D skeleton sequences or coordinate position sequences, based on the target representation. The challenge lies in effectively integrating the 2D pose information and video context to achieve accurate predictions for each task. The first input, denoted as u (p) t , defines the 2D position of human p at frame t following this formula: u

(p) t = [(x (p) t , y (p) t ), ..., (x (p) t , y (p)
t )] Given that we observe trajectories and scenes from frame 1 to t obs , the goal is to predict the paths of movements in frames ranging from the next frame (t obs +1) to the final frame (t f ). For a pedestrian p, the sequence of observed and future positions are respectively denoted by τ

(p) obs = (u (p) 1 , ... , u (p) t obs ) and τ (p) f = (u (p) t obs +1 , ... , u (p)
t f ). The sequence of observed frames is denoted by ν o = (I 1 , ..., I t obs ), which consists of video frames of the scene. Building such a model presents challenges due to the complexity and diversity of human motions and the scarcity of accurate data. To address these issues, we employ an LSTM-based 2D pose / position encoder, a transformerbased visual encoder, and a decoder for predicting either the 2D trajectory or the 3D human poses.

Architecture

Our architecture is expressed an encoder-decoder model for both 2D trajectory prediction and 3D pose prediction (Figure 1). The encoder part is composed of a landmark encoder-based LSTM responsible of encoding the motion from 2D skeleton joints matrices, and a context encoder which analyse the image through a block of ResNet followed by a Transformer encoder. Finally, a decoder take as input the concatenation of the two previous encoders, and predicts both 2D trajectory and/or 3D skeleton joints representation of human motion. 2D pose/position encoder. This encoder uses the observed trajectory of the target human as input, which includes the starting position and the relative displacements between consecutive frames. This format was chosen to better capture the similarities between similar trajectories that start at different points. The input vectors are transformed using a fully-connected network, which power onelayer LSTM, to model the dependencies between different coordinates of the observed trajectory. The outputs from the LSTM units capture the latent kinematic trajectory (2D human coordinates) or landmarks (2D human poses).

Context encoder. This encoder is used to extract the visual information of each human. In fact, the observed video contains information about the physical constraints of all humans present in the scene. This encoder is composed of a ResNet followed by a transformer network [47].

The ResNet encodes spatial information from the video frames while the Transformer encode the temporal relationship while focusing on regions relevant to each human using the multi-head attention mechanism. It is composed of a Multi-Head attention mechanism, a fully connected layer and a normalization layer.

M ultiAtt(x) = (Att(x) ⊕ ... ⊕ Att(x))W O , where Att(x) = σ( Q i K i d k )V i
where

Q i = xW Q i , K i = xW K i , V i = xW V
i are independent linear projections of x into a d-feature space, d is a scaling factor. Some missing information present here as no recurrence neither convolution in the encoder attention mechanism. To overcome that, a Positional Encoding information is added with the same shape as encoder input to inject information on the relative or absolute position of the frame in the set of videos. Thus, T (x) = AvgP ool(ET (x + P E)), where ET is the encoder transformer mechanism function which can be defined as: ET (x) = N orm(x + F C(M ultiAtt(x))), P E is the aforementioned Positional Encoding, F C is a fully connected layer, and N orm is a normalization layer.

Decoder. The decoder takes as input the combination of the two resulted previous vectors. After, it is passed to a maxpool layer followed by a linear layer to understand the relation between these. Then, the result will be feed to the LSTM as hidden vector, followed by a MLP. We aim to predict human motion using two main inputs: 2D human poses / position and video context. The desired output depends on the specific target problem. In the case of 3D pose prediction, our goal is to forecast the 3D poses of the human subjects. Conversely, for 2D trajectory prediction, the objective is to predict the trajectory of the target person in the 2D space. The challenge lies in effectively leveraging the 2D pose / position information and video context to achieve accurate and reliable predictions for the respective tasks.

Loss function. The loss function is defined depending on the form of data to be predicted. For 3D pose estimation, a sub-sequence of 2D body pose (21 2D joints) is observed in order to predict a sub-sequence of 3D body poses (21 3D joints). As in [34], the idea is to use the loss functions to fix the distance and the angle between two joints, in addition to the typical mean square error. For the angle, we use the cosine similarity between the joints. For the length between two joints, we use the l2-norm. In other words, the loss function have 3 components: MSE between the joints and the prediction, a consistency loss that make sure the intra-distance between joints are consistent and that the angles between the joints are consistent. Also known as, we don't want to end up with weird angles and/or weird skeleton bone lengths. The loss function can be defined as:

L model = λ 1 L cos + λ 2 L norm + L mse ,
where λ 1 and λ 2 are parameters to tune, so we set them to 0.01.

For the 2D trajectory prediction, our loss function consists of two components -the mean-squared loss and a regularization term called Lreg, which regulates the smoothness of future trajectories, as prior work [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF]. In training our network, we use the following loss function: Lmodel = Lmse + λLreg, where λ is a regularization parameter. We kept the value of λ fixed at 0.5 in our experiments to avoid restricting the model's ability to capture sudden changes in the target pedestrians' trajectory. Lmse is calculated as the average of the squared differences between predicted and observed values, while Lreg is calculated as the sum of Euclidean distances between each step of the predicted trajectory and a line fitted to the observed trajectory. In our experiments, we sample 20 future trajectories and select the top 5 trajectories that are closest to the ground-truth to calculate L mse . Through empirical observation, we have found that this approach enables our network to converge more quickly while producing more accurate predictions.

Experiments

Datasets

Two datasets are used to evaluate our approach for 3D pose prediction: GTA Indoor Motion (GTA-IM) [START_REF] Cao | Long-term human motion prediction with scene context[END_REF] and Proximal Relationships with Object eXclusion (PROX) [START_REF] Hassan | Resolving 3d human pose ambiguities with 3d scene constraints[END_REF].

GTA-IM. GTA-IM is collected in an indoor environments emphasizing human-scene interactions. It contains HD RGB-D image sequences of 3D human pose and camera pose annotations, and large diversity in human appearances, indoor environments, camera views, and human activities. Following the experimental protocol proposed by [START_REF] Cao | Long-term human motion prediction with scene context[END_REF]34], we spit the data into two subsets, with 80% (8 scenes) for training and 20% (2 scenes) for evaluation purposes.

PROX. PROX is collected with the Kinect-One sensor, and contains 12 different 3D scenes, including 3d human body skeleton, and RGB sequences of 20 subjects moving in and interacting with the scenes. Following [START_REF] Cao | Long-term human motion prediction with scene context[END_REF]34], we split this dataset into 52 training sequences for training and 8 sequences for test.

Regarding 2D trajectory prediction, our approach was evaluated on well-established public human-trajectory datasets, namely ETH [37] and UCY [START_REF] Lerner | Crowds by Example[END_REF] datasets, which are widelyused benchmarks for pedestrian motion prediction.

ETH and UCY combined. These datasets were acquired from surveillance videos of pedestrians walking on sidewalks and annotated with location coordinates. The datasets contain real-world pedestrian trajectories in top-view coordinates expressed in meters, with rich human-human and human-object interaction scenarios. The acquisition was done using a fixed camera on 5 different scenarios captured at 2.5 Hz, and an image is annotated with pedestrian positions every 0.4 seconds. ETH/UCY consists of five different scenes. The ETH dataset consists of two scenes (ETH and Hotel) taken from a bird's eye view, with hundreds of pedestrian trajectories engaged in walking activities. The UCY dataset provides three scenes (Zara1, Zara2, and Univ) taken from a bird's eye view with standing/walking activities.

Metrics

For evaluation, we choose several appropriate metrics to measure the error of the 3D path and pose human motion prediction. The main metric employed is Mean Per Joint Position Error (MPJPE) or 3D pose error. It is calculated for a frame t and a skeleton consisting of J joints by the following formula [START_REF] Alahi | Socially-aware large-scale crowd forecasting[END_REF], where P the prediction, and Y the ground truth.

M P JP E(t) = 1 J J j=1 ||P t j -Y t j || 2 (1)
For the 3D path error, we use the same formula but only for a fixed joint that is the center of the skeleton joints. In fact, these two metrics are used to compute the evaluation of the error with the time step, i.e. after the quarter, half, one half, and full 3D joint estimation. Similar to existing works, other metrics are also used for comparison, Average Displacement Error (ADE), Final Displacement Error (FDE), and Stability Error (STB). ADE is used to measure the overall performance of the pose and the path on the entire predicted sequence. On the other hand, FDE is an indicator of the final time step prediction for path and pose error. Finally, STB is used to detect the variation of the path and pose MPJPE error over different time steps. This metric is mainly used to check the divergence over the prediction time steps, not the accuracy of the model.

For the 2D trajectory prediction, similar to existing works [3, 14, 4, 42, 43, 53, 22, 18, 20, 59, 24], our method is evaluated using two widely used metrics in the field, namely the (ADE) and the (FDE). ADE is defined as the average L2 distance (in meters) between the actual trajectory and the predicted trajectory at each time step of the trajectory from T obs+1 to T pred on average over all pedestrians. FDE is defined as the Euclidean distance (in meters) between the ground truth (actual position) and the prediction (predicted position) at the last time step of the prediction T pred , averaged over all pedestrians. Formally, ADE =

n i=1 T pred t=T obs+1 || Ŷ i t -Y i t || n * T
, and

F DE = n i=1 || Ŷ i T pred -Y i T pred || n
Where n represents the number of pedestrians, Ŷ i t are the predicted coordinates for pedestrian i at time t, Y i t are the real future positions, and || is the Euclidean distance. T pred is the final predicted timestep. T is the prediction horizon.

Implementation details

In GTA-IM, the initial learning rate is set at 0.01, while in the PROX dataset, it is 0.03. The model is trained for 400 epochs, with the learning rate decreasing by 0.2 every 200 epochs. A batch size of 128 is utilized. We use 1 second of observation (5 frames) and 2 seconds for predictions (10 frames) following the settings of prior works for comparison. Table 1 shows the different model parameters chosen in the experiments. The model is trained for 400 epochs, a batch size of 128 and a learning rate of 0.01. Table 1 shows the different model parameters chosen in the experiments. For the 2D trajectory prediction, we trained the entire network with a batch size of 40 for 400 epochs, using stochastic gradient descent (SGD) optimizer with a learning rate scheduler and two mean squared error (MSE) loss functions. The learning rate is adjusted every 40 steps with an initial learning rate of 0.01 and the maximum gradient value is clipped to 1 to prevent gradient explosion. We adopted the teacher force strategy and used our proposed loss function with a value of λ = 0.5 to achieve faster convergence as prior work [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF]. During training, we generated 20 output trajectory samples and used the 5 samples with the lowest loss value to train the model. We use 3.2 second of observation (8 frames) and 4.8 seconds for predictions (12 frames) following the settings of prior works for comparison. Our model is implemented in Pytorch on Ubuntu with an NVIDIA TITAN RTX GPU and 24 GB RAM memory.

Results on 3D pose prediction

Baseline. There exists very few prior works that predicts 3D human pose with global movement using 2D pose sequence as input. Thus, as a reference, we use the results from methods reported in [START_REF] Cao | Long-term human motion prediction with scene context[END_REF]. (1) TR [47]: utilizes transformer architecture to predict the 3D human pose directly from 2D input data, treating the entire problem as a singlestage sequence to sequence task. (2) TR [47]+VP [START_REF] Pavllo | 3d human pose estimation in video with temporal convolutions and semi-supervised training[END_REF]: a combination of the transformer and a 2D-to-3D human pose estimation method. It is constructed by first predicting the future 2D human pose using [47] from inputs and then lifting the predicted pose into 3D using [START_REF] Pavllo | 3d human pose estimation in video with temporal convolutions and semi-supervised training[END_REF].

(3) VP [START_REF] Pavllo | 3d human pose estimation in video with temporal convolutions and semi-supervised training[END_REF]+LTD [START_REF] Mao | Learning trajectory dependencies for human motion prediction[END_REF]: a combination between a 2D-to-3D human pose estimation method [START_REF] Pavllo | 3d human pose estimation in video with temporal convolutions and semi-supervised training[END_REF], and a 3D human pose estimation based on graph [START_REF] Mao | Learning trajectory dependencies for human motion prediction[END_REF]. None of these three baselines take into account the context of the scene. ( 4) GPP-Net [START_REF] Cao | Long-term human motion prediction with scene context[END_REF]: is composed of three stages and incorporates various concepts, including VAEs, as well as customized stages for predicting both paths and poses. ( 5) SG [34]: concerns the prediction of the 3D human pose prediction based on Skeleton Graph, using graph CNNs with self learning adjacency matrix and formulating the problem as a spatio-temporal graph. Note that there are two variants of SG : with/without the context (designed as +/-C), in order to observe the impact of context, especially on the PROX dataset.

Quantitative Analysis. Tables 2345show the result obtained from both GTA-IM and PROX datasets, employing the provided metrics. Specifically, Table 2 and 3 display the assessment outcomes for the forecasted path and 3D pose on the GTA-IM and PROX datasets respectively. Table 4 and 5 illustrate the FDE, ADE, and STB outcomes obtained through various techniques on the GTA-IM and PROX datasets respectively. Our approach demonstrates better performance than previous methods across several metrics. At first sight, the STB metric of 1 and 4 for GTA-IM and PROX dataset respectively shows that our model is stable for long term prediction. This result could be due to the trick of predicting displacement instead of just the coordinates, in order to better capture the similarities between close trajectories starting at different positions. For the 3D path error and the 3D pose error in GTA-IM, only for long term, our approach starts to outperform the other methods. FDE is very closed to ADE in our case, which is due to the stability of the prediction over time. If we want to choose short-term prediction, GPP-Net [START_REF] Cao | Long-term human motion prediction with scene context[END_REF] is the best choice, and for a long term prediction, our model is the best. In the other hand, PROX dataset was primarily recorded in a vacant laboratory setting, resulting in limited diversity in terms of visual characteristics. As a result, it lacks the same level of richness as the GTA-IM dataset, which has implications for the diversity of backgrounds and camera angles. Due to these limitations, the visual signal is inherently less informative, which can result in increased ambiguity in predictions. These limitations of the PROX dataset will impact our results as seen it in the table 3 and 5 (without context, our results are better than those with context). A similar discussion was raised in [START_REF] Cao | Long-term human motion prediction with scene context[END_REF]34]. For the PROX dataset using the vision signal resulted in a divergence of the results. As seen in table 4, the errors in short-term prediction are even larger than in longterm prediction. Further investigation is needed to understand the reasons behind this behavior. Despite this, we are mostly better than the other methods and these results on the PROX dataset demonstrate the resilience of our approach in scenarios involving imprecise 2D pose estimations. Also, we can note that the 3D pose error was the lowest compared to prior works, suggesting our model (-C) effectiveness in predicting 3D poses, even without the context. However, it had the highest error for the 3D path indicating a tradeoff between the objectives of path and poses, which may be influenced by the lack of visual signal. Additionally, as seen in table 2 and 4, when the vision signal was used on the GTA-IM dataset, there was a significant improvement in both path and pose prediction performance mainly in long term. However, when the same signal was used on the PROX dataset, the results diverged. Additionally, even without context, the results for GTA were superior, emphasizing the importance of context in the prediction as we explained in the introduction.

Qualitative Analysis. 2, when the error decreases with time steps. In general, this prediction seems to be good, with the exception of the two last predicted joints, where the feet cross each other. For the sitting sequence in Figure 2-(down), we can observe a total lack of prediction, especially for the feet. This error may be due to the lack of training data on the sitting motion. This result is obtained despite that during training we use the consistency loss, which tunes the angle between two consecutive joints. Here for instance between two successive feet joint, the angle is tuned, but the problem comes from the angle between the feet and the back joint. For the evaluation, even if the prediction is not the best, the metric used will only compute the distance between the prediction and the target joins, so no loss related to angle is evaluated.

Results on 2D trajectory prediction

To address the problem of pedestrian trajectory forecasting (2D), we used the same encoder-decoder architecture. It is bimodal in the way that it is composed of two parallels encoder branches, see figure 1.

Baseline. We compare the effectiveness of our proposed approach with state-of-the-art models by conducting a comparative analysis. We evaluate our approach against six deterministic baselines, which are linear regression, LSTM, Social-LSTM [3], Social ATTN [START_REF] Vemula | Social Attention: Modeling Attention in Human Crowds[END_REF], TrafficPredict [29], and SR-LSTM [START_REF] Zhang | SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction[END_REF]. For evaluation, we generate 20 predictions for each observed trajectory and select the prediction closest to the ground truth. This evaluation technique enables us to examine the multi-modality and diversity of the predictions. We also compare our approach against various generative baselines, including Social-GAN [START_REF] Gupta | Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks[END_REF], DESIRE [22], FSGAN [START_REF] Kothari | Human trajectory prediction using adversarial loss[END_REF], SoPhie [42], Trajectron [18], MATF [START_REF] Zhao | Multi-Agent Tensor Fusion for Contextual Trajectory Prediction[END_REF], NEXT [START_REF] Liang | Peeking into the future: Predicting future person activities and locations in videos[END_REF], Social-BiGAT [20], Social-STGCNN [START_REF] Mohamed | Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction[END_REF], Social ways [START_REF] Amirian | Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs[END_REF], and PECNet [START_REF] Mangalam | It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction[END_REF], M2P3 [START_REF] Poibrenski | M2P3: multimodal multi-pedestrian path prediction by self-driving cars with egocentric vision[END_REF], SGN LSTM [57], Trajectron++ [START_REF] Salzmann | Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data[END_REF], Introvert [START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF], GroupNet [START_REF] Xu | GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning[END_REF], Transformer-TF [12], STAR [START_REF] Yu | Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction[END_REF],

and AgentFormer [START_REF] Yuan | AgentFormer: Agent-Aware Transformers for Socio-Temporal[END_REF]. These models use various approaches such as LSTM, GAN, spatio-temporal graph convolutional neural networks, and transformers to predict human trajectories.

Evaluation method. For benchmarking purposes, we follow a similar evaluation method as previous studies (See Table 6). When evaluating trajectory forecasting models, the chosen time horizon is crucial due to varying object speeds.

The choice of time horizon should depend on the class of objects being considered. To ensure fairness in comparisons, we observe each training trajectory for 8 times-steps (3.2 seconds) and then evaluate performance oven the next 12 time-steps (4.8 seconds). To fully utilize the datasets during training, we adopt a leave-one-out evaluation strategy, common in prior research. We train our model on four sets of data and evaluate it on the remaining set. We repeat this process for all the 5 sets.

Quantitative Analysis. The main results, shown in Table 6, highlight the superior performance of ours* over state-ofthe-art (SOTA) deterministic models in terms of overall performance, with ADE/FDE scores of 0.23/0.37. Moreover, our stochastic model exhibits a significant performance improvement over all SOTA models by a large margin. Our primary focus lies in the comparison of these stochastic models. The results are presented against state-of-the-art approaches as mentioned above, using the best-of-20 protocol, which involves sampling 20 possible future trajectories and selecting the one with the best test performance. Our proposed method achieves outstanding performance, ranking the first among state-of-the-art methods. In particular, on the FDE metric, our method significantly outperforms existing algorithms on 4 out of 5 datasets, achieving the best average error of 0.33. On the ADE metric, the proposed method outperforms existing algorithms on 3 out of 5 datasets and achieves an average ADE error of 0.19 across all 5 datasets. The University dataset has higher displacement errors compared to other datasets, making it challenging to predict future trajectories accurately. Our method remains comparable to other existing approaches but outperforms all the dense interaction-based methods like S-GAN, Sophie, S-BiGAT, S-STGCNN, and Social Ways. The Hotel dataset has many pedestrians waiting for trains, resulting in limited motion. Therefore, most methods, including ours, achieve relatively small displacement errors by predicting small motions accurately. Our proposed method achieves the lowest FDE (0.10) and ADE (0. placement errors, which is a common occurrence among many models, due to lower frequency of video frames and kinematic data. However, our method achieves the lowest ADE/FDE errors on the ETH dataset, showing the effectiveness of our approach in capturing and incorporating information about the movements and behaviors of neighboring pedestrians. Our proposed method outperforms transformer-based methods on the Zara1 dataset, which is the least structured dataset in the benchmark. The dataset mainly consists of straight lines. The proposed method achieves an ADE similar to the best method [START_REF] Yuan | AgentFormer: Agent-Aware Transformers for Socio-Temporal[END_REF] (0.14) and also achieves the lowest FDE among all the methods compared (0.23). As seen, our approach outperforms previous Transformer-based methods such as Transformer-TF, STAR, and AgentFormer on the ETH and UCY datasets. Overall, our model offers a competitive alternative to graphbased methods [20,[START_REF] Mohamed | Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction[END_REF] and has the potential to improve trajectory prediction accuracy.

Qualitative Analysis. The qualitative outcomes, depicted in Figure 3, display the accuracy of our trajectory prediction on multiple videos from the ETH and UCY datasets, providing visual evidence of its effectiveness in accurately predicting pedestrian trajectories. The examples presented illustrate various scenarios, including human-human interaction, human-space interaction, and avoiding obstacles.

For instance, in the top left example, our model successfully predicts the target pedestrian's trajectory through the store's left-side door. Additionally, in the bottom left example, our method accurately predicts the target person's avoidance of a tree and continues straight towards the train. In the top right example, our model captures a human-human interaction, wherein the target pedestrian slows down before a group of standing people, bypasses them from the left side. These instances demonstrate our method's ability to effectively predict future pedestrian positions, particularly in crowded scenes with complex interactions.

Conclusion

In this paper, we presented a novel approach for predicting future human motion, encompassing both 3D pose estimation and 2D trajectory prediction, depending on the specific target representation. Our general encoder-decoder architecture combined transformer networks with self-attention mechanisms and LSTM to jointly capture contextual information and motion sequences. By incorporating image video scene context and a sequence of 2D positions/body poses, our model demonstrated remarkable performance across diverse benchmark datasets, including GTA, PROX, and the combined ETH and UCY datasets. The extensive benchmarking underscored the superiority of our approach over existing state-of-the-art methods, particularly excelling in long-term prediction tasks essential for real-world applications. For future work, we propose exploring multi-modal data fusion, integrating additional modalities such as depth information or semantic segmentation maps, along with attention mechanisms and transfer learning.

Figure 1 :

 1 Figure 1: Our proposed approach.

  Figure 2 visualizes the predicted 3D human poses, presenting predictions for 10 human poses in two motion scenarios: walking and sitting. The joint estimation for walking sequence Figure 2-(top) seems very close to the ground truth and even the model becomes better with time steps, which is a concretization of the result obtaining in the Table

Figure 2 :

 2 Figure 2: Predicted 3D human pose compared to the ground truth. The arrow indicates the time direction. (top) Example of walking motion prediction. (down) Example of sitting motion prediction.

Figure 3 :

 3 Figure 3: Illustration of the prediction trajectories. Yellow dots represents the past observed while violet & green dots represent our prediction and the ground truth.

Table 1 :

 1 Model details.

	Input	Image size	Output	Hidden size	Embed Size	Visual out	Landmark out
	42	90*160	63	256	64	256	256

Table 2 :

 2 Evaluation results in GTA-IM dataset (errors are in mm).

		3D path error	3D pose error
	Time step (s)	0.5 1	1.5 2	0.5 1	1.5 2
	TR[47]	277 352 457 603 291 374 489 641
	TR[47]+VP[36]	157 240 358 494 174 267 388 526
	VP[36]+LTD[31] 124 194 276 367 121 180 249 330
	Ours (-C)	222 221 222 226 299 291 291 295
	GPP-Net (+C) [5] 104 163 219 297 91 158 237 328
	SG (+C) [34]	154 163 172 186 198 209 217 230
	Ours (+C)	167 168 169 167 208 211 212 213

Table 3 :

 3 Evaluation results in PROX dataset. Error are in mm. C: to indicate, using context or not.

		3D path error	3D pose error
	Time step (s)	0.5 1	1.5 2	0.5 1	1.5 2
	TR[47]	487 583 682 783 512 603 698 801
	TR[47]+VP[36]	262 358 461 548 297 398 502 590
	VP[36]+LTD[31] 194 263 332 394 216 274 335 394
	Ours (-C)	306 303 302 302 291 286 285 286
	GPP-Net (+C)[5] 189 245 317 389 190 264 335 406
	SG (+C) [34]	353 353 355 360 358 362 365 370
	Ours (+C)	350 346 342 343 362 351 350 352

Table 4 :

 4 Results of FDE, ADE and STB in GTA-IM dataset are reported in mm. C: to indicate, using context or not.

	Time step (s)	FDE	ADE	STB
	TR[47]	622	436	147
	TR[47]+VP[36]	510	326	150
	VP[36]+LTD [31]	349	230	98
	Ours (-C)	260	258	2
	GPP-Net (+C) [5]	313	200	93
	SG (+C) [34]	208	192	11
	Ours (+C)	190	189	1

Table 5 :

 5 Results of FDE, ADE and STB in PROX dataset are reported in mm. C: to indicate, using context or not.

	Time step (s)	FDE	ADE	STB
	TR[47]	792	644	126
	TR[47]+VP[36]	569	427	126
	VP[36]+LTD[31]	394	300	82
	Ours(-C)	294	295	2
	GPP-Net (+C) [5]	398	292	90
	SG (+C) [34]	365	359	3
	Ours(+C)	347	349	4

Table 6 :

 6 /1.59 0.62/1.21 0.77/1.48 0.39/0.72 1.33/2.94 0.79/1.59 LSTM* 0.61/1.31 0.41/0.88 0.52/1.11 0.86/1.91 1.09/2.41 0.70/1.52 Social-LSTM* [3] 0.67/1.40 0.47/1.00 0.56/1.17 0.79/1.76 1.09/2.35 0.72/1.54 Social-ATTN* [48] 0.33/3.92 0.20/0.52 0.30/2.13 0.29/2.64 0.39/3.74 0.30/2.59 TrafficPredict* [29] 3.31/6.37 4.32/8.00 3.76/7.20 2.55/3.57 5.46/9.73 3.88/6.97 SR-LSTM* [58] 0.51/1.10 0.41/0.90 0.32/0.70 0.37/0.74 0.63/1.25 0.45/0.94 .32 0.30/0.62 0.36/0.75 0.49/1.01 0.69/1.29 0.48/1.00 Social-STGCNN [35] 0.44/0.79 0.34/0.53 0.30/0.48 0.49/0.85 0.64/1.11 0.44 / 0.75 Social Ways [4] 0.55/1.31 0.44/0.64 0.51/0.92 0.39/0.66 0.39/0.64 0.46/0.83 PECNet [30] 0.35/0.60 0.22/0.39 0.17/0.30 0.18/0.24 0.54/0.87 0.29/0.48 M2P3 [39] 0.64/1.34 0.45/0.95 0.37/0.79 0.54/1.13 1.04/2.16 0.60/1.27 Transformer-TF [12] 0.35/0.65 0.22/0.38 0.17/0.32 0.18/0.30 0.61/1.12 0.31/0.55 STAR [53] 0.31/0.62 0.26/0.55 0.22/0.46 0.17/0.36 0.36/0.65 0.26/0.53 AgentFormer [54] 0.25/0.45 0.18/0.30 0.14/0.24 0.14/0.22 0.45/0.75 0.23/0.39 Trajectron++ [43] 0.30/0.54 0.25/0.41 0.18/0.32 0.18/0.28 0.67/1.18 0.32/0.55 SGN LSTM [57] 0.48/1.08 0.30/0.65 0.26/0.57 0.63/1.01 0.75/1.63 0.48/0.99 Introvert[START_REF] Shafiee | Introvert: Human Trajectory Prediction via Conditional 3D Attention[END_REF] The average/final displacement error (ADE/FDE) metrics for several methods compared to our model are shown. Lower is better. The models with * have deterministic outputs. All the stochastic models sample 20 possible trajectories and report the best result using a best-of-20 protocol. All models observe 8 frames and forecast the subsequent 12 frames.

				Performance ADE/FDE ↓ (m)
	Method	Univ	Zara1	Zara2	Hotel	ETH	Avg
	Linear* 0.82Ours* 0.25/0.41 0.16/0.27 0.17/0.26 0.10/0.14 0.47/0.76 0.23/0.37
	DESIRE [22]	0.59/1.27 0.41/0.86 0.33/0.72 0.52/1.03 0.93/1.94 0.53/1.11
	Social-GAN [14]	0.60/1.26 0.34/0.69 0.42/0.84 0.72/1.61 0.81/1.52 0.58/1.18
	FSGAN [21]	0.54/1.14 0.35/0.71 0.32/0.67 0.43/0.89 0.68/1.16 0.46/0.91
	SoPhie [42]	0.54/1.24 0.30/0.63 0.38/0.78 0.76/1.67 0.70/1.43 0.54/1.15
	Trajectron [18]	0.54/1.13 0.43/0.83 0.43/0.85 0.35/0.66 0.59/1.14 0.47/0.92
	MATF [59]	0.44/0.91 0.26/0.45 0.26/0.57 0.43/0.80 1.01/1.75 0.48/0.90
	Next [27]	0.60/1.27 0.38/0.81 0.31/0.60 0.30/0.59 0.73/1.65 0.46/1.00
	Social-BiGAT [20]	0.55/1

[START_REF] Hassan | Resolving 3d human pose ambiguities with 3d scene constraints[END_REF]

errors on this dataset. The ETH dataset often produces larger dis-0.20/0.32 0.16/0.27 0.16/0.25 0.11/0.17 0.42/0.70 0.21/0.34 GroupNet

[START_REF] Xu | GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning[END_REF]

0.26/0.49 0.21/0.39 0.17/0.33 0.15/0.25 0.46/0.73 0.25/0.44 Ours 0.23/0.40 0.15/0.26 0.14/0.23 0.10/0.15 0.35/0.62 0.19/0.33