Farhood Negin
email: farhood.negin@inria.fr

Mohsen Tabejamaat
email: mohsen.tabejamaat@inria.fr

Renaud Fraisse
email: renaud.fraisse@airbus.com

Francois Bremond
email: francois.bremond@inria.fr

Transforming Temporal Embeddings to Keypoint Heatmaps for Detection of Tiny Vehicles in Wide Area Motion Imagery (WAMI) Sequences

Nowadays, due to its many applications, objects detection in wide area motion imagery (WAMI) sequences has received a lot of attention. Unlike natural images, object detection in WAMI faces unique challenges. Lack of appearance information due to the small size of objects makes object detection difficult for conventional methods. In addition, pixel noise, registration errors, sparse or densely populated objects, brings on pronounced artifacts which amplifies the difficulty of detection. This paper aims to address object detection problem in the presence of these issues by considering objects as keypoints in the relevant background and proposes a spatiotemporal anchor-free detector for tiny vehicles in WAMI images. Instead of background subtraction, a region of interest network refines large search space of sequences to indicates object clusters. For further investigation, clusters are encoded by a codebook which is learned through an unsupervised encoder-decoder network. To accurately generate the detections, a Transformer network is trained on cluster embeddings using groundtruth heatmaps that are described by Gaussian distribution rather than hard label annotation. The network is trained with a redesigned version of Focal loss comprising a shape prior regularizer which help the generated heatmaps to conform to the shape of the keypoints. Extensive experiments on WPAFB dataset demonstrate the high capability of our method for the detection of small vehicles where it achieves competitive performance when compared to the state-ofthe-art.

INTRODUCTION

Today, with the advent of high-altitude airborne and space cameras, long-term WAMI video recording has become possible, which has many applications in the military and civilian fields [2, [START_REF] Bao | Single shot anchor refinement network for oriented object detection in optical remote sensing imagery[END_REF][START_REF] Liang | Vehicle detection in wide area aerial surveillance using temporal context[END_REF]21,[START_REF] Reilly | Detection and tracking of large number of targets in wide area surveillance[END_REF][START_REF] Saleemi | Multiframe many-many point correspondence for vehicle tracking in high density wide area aerial videos[END_REF][START_REF] Xiao | Vehicle detection and tracking in wide field-of-view aerial video[END_REF]35]. For example, in urban planning, it can help control traffic and analyze drivers' behavior, and in security monitoring scenarios, it can help to identify dangerous and abnormal behaviors. In all of these applications, object detection has been one of the major techniques for achieving quantitative surveillance from videos. Numerous detection methods have been extensively studied and applied to ground-based surveillance systems in which videos are usually recorded by a fixed camera with a relatively low height from the ground and usually of high quality. Satellite remote sensing videos, on the other hand, are often recorded with low resolution and a non-stationary camera platform, and the recorded data usually contains geometrical deformations. Therefore, surveillance systems for processing such data face different challenges than traditional methods applied on ground-based systems and natural images. At a ground sample distance of about 1 meter, the target moving vehicles in current satellite videos are spatially recorded at approximately 5 to 20 pixels. Therefore, a vehicle in these images, which are usually captured in gray-level, is consisted of only a few pixels without color, shape and distinctive texture information. This makes the detection task very challenging in particular for the anchor-based methods at which the predefined anchors need to cover the objects very well. For tiny objects, a few number of pixels can produce large errors as it is not simple to determine whether a pixel near an anchor belongs to an object or not.

Lack of appearance information makes motion the most powerful feature to identify these objects. Nonetheless this feature is often challenged in detection task due to background movement. In addition, since satellite camera systems often consist of several separate sensors, this can create artifacts such as lighting discontinuities and shearing. Accordingly, moving vehicles in the scene can be confused with pixel noise and, in general, with noise patterns from complex moving backgrounds, which creates problems for detection (Figure 1). Furthermore, objects in high resolution WAMI images are often sparse and located only in certain areas, such as roads and around buildings. This adds to the complexity of detecting objects in the whole image, which produces a lot of false alarms and thereby reduces the efficiency of the system.

In this paper, we propose a novel spatiotemporal anchorfree strategy without requiring any post-processing to address the challenges of object detection task that are specific to WAMI sequences. To avoid problems regarding anchor-based methods, such as designing anchors and tuning their hyperparameters, we consider tiny vehicle detection as a keypoint detection problem. At first, the framework learns to detect regions of images containing sparse or dense clusters of objects through a lightweight convolutional network and reduces the wide search space of the large-scale images using their corresponding Gaussian heatmaps. Varying from conventional CNN-based representation, the model exploits spatiotemporal information of the sequences using the Vector Quantized Variational Autoencoder (VQ-VAE) technique to achieve an efficient discrete video-level representation. Unlike most of the current solutions that benefit only from limited spatial information and ignore the temporal dependency of targets or utilize inefficient sliding windows, our vector quantized embedding allows an efficient use of both appearance and motion information to surpass this limitation in an unsupervised manner. This way, we encourage the framework to encode the spatial characteristic of the samples with motion dynamics which leads to a further rich representation describing spatiotemporal evolution of the data. For precise detection of the objects, the embeddings are then used to train a Transformer network that takes clip-level embeddings as input and generates frame-level detection by estimating center of the objects. Therefore, our contributions can be summarized as follows:

• A region of interest network based on Gaussian heatmap of sequences is trained that significantly reduces the large search space of WAMI images.

• knowing the inadequate efficiency of standard feature representations and considering the scarcity of appearance features in WAMI images, we introduce a novel strategy that benefits from a combined spatiotemporal feature in an unsupervised manner.

• Unlike the current strategies, our method does not require any inefficient and error-prone background subtraction operations, which allows for effective detection of moving as well as stationary objects in the scene.

• To generate precise detection masks, we adapt a loss function with a shape prior regularizer to compensate for the unbalanced distribution of the objects presented in sparse sequences and also to achieve pixel-level segmentation heatmaps, which in return helps in improving accuracy of the detection.

• The proposed method achieves competitive performance in the detection task compared to the state-of-theart.

Related work

It has been practically and theoretically proven that it is more difficult to detect smaller objects, especially in WAMI images. It has been shown in practice [START_REF] Jiao | A survey of deep learningbased object detection[END_REF] that there is a big discrepancy between the performance of models on small objects and their performance on large objects. Theoretically, it can be explained that since the models use the information that passes through the model, they gradually form and learn the features required for detection. The lower the information flow (in the case of WAMI, only a few pixels), the weaker and less distinctive the features learned. Therefore, the loss function, which works on the basis of pixel computations, will not have enough signal to learn during backpropagation. To address these issues, object detection methods generally use spatial information, temporal information, or both.

Background subtraction methods for WAMI

The most straightforward approach to use temporal information is to calculate frame differences between two or a higher number of consecutive frames [START_REF] Bao | Single shot anchor refinement network for oriented object detection in optical remote sensing imagery[END_REF][START_REF] Saleemi | Multiframe many-many point correspondence for vehicle tracking in high density wide area aerial videos[END_REF][START_REF] Xiao | Vehicle detection and tracking in wide field-of-view aerial video[END_REF]. Despite simplicity this method is associated with the problem of aperture for slow objects where the inner pixels of objects are recognized as the background, or the ghost effect where the pixels behind moving objects are recognized as the foreground. Background subtraction [START_REF] Kent | Robust background subtraction for automated detection and tracking of targets in wide area motion imagery[END_REF][START_REF] Liang | Vehicle detection in wide area aerial surveillance using temporal context[END_REF][START_REF] Sobral | A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos[END_REF] takes a more holistic view of the problem. Considering the detection of moving objects as a special case of foreground segmentation, it creates an explicit background model that recognizes foreground objects by subtraction of the current input image from this background. Parallax effect, difficulty in detection of the stationary targets, and change in global illumination that negatively affect modeling of the background are considered as the main issues of these methods.

Spatiotemporal networks

Region proposal networks (RPN) such as Faster-RCNN, which do not use temporal information and only detect objects in a single frame, have been shown [START_REF] Lalonde | Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information[END_REF] to fail in WAMI images. WAMI images are not only large and can not easily be used as input to such networks, but even splitting them into grids does not help because it does not conform to the principle of performance on which RPN networks are developed. It requires to design more anchors and tune their parameters that can not be generalized from other datasets. In addition, downsampling may not be a good choice as there is almost no reliable information left after downsampling due to the small size of the target objects (Which are usually annotated with only one point in the ground-truth). The benefits of using spatiotemporal information in various tasks have been exploited in recent years. Various network architectures [9, 10] have been tried in order to detect objects in videos. Some methods [START_REF] Lin | Focal loss for dense object detection[END_REF]30] rely on post-processing of temporal information to make object detection results more coherent and consistent. These methods first use an off-the-shelf detector to find objects in each of the frames and then try to associate the results together. Some other work [6, 9, 10, 29, 33] takes advantage of the aggregation of spatiotemporal features. In particular, by designing special operators, they mainly improve the features of the current frame by aggregation of the features of adjacent frames (or the whole clip) and ultimately increase the overall detection performance. Very recently, Transformers [4, 7, 11, 41] raised great attention on various vision tasks and some methods use them to design object detection systems that achieve on par performance with state-of-theart CNN-based detectors. However, most of these methods require complex post-processing stages to link the same object throughout the video to form tubelets that usually have a long convergence time. Consequently, only a handful of studies [START_REF] Lalonde | Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information[END_REF]40] have used these methods on WAMI sequences.

Method

Figure 2 shows the overall architecture of our method. It is comprised of three main modules. First, to limit the search space on wide input images, a CNN network identifies areas of interest that are most likely to contain objects. In the second module an unsupervised model of representative features is learned from the generated proposals. The feature embeddings are then used as input to a Transformer in order to generate the outputs of the framework which are the generated heatmaps and the detection centers.

Regions of interest (ROI) proposals by regression heatmaps

Many CNN-based methods directly detect keypoint coordinates through regression [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] Martin | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] or use anchors [3,34] to extract deep features of candidate regions. Regressionbased detectors require dense layers with large network parameters to learn image-to-coordinate mapping. This results in highly nonlinear models that are difficult to learn. Anchor-based methods, on the other hand, require anchor design in advance, and given the small number of object pixels in WAMI images, only the very precise part of the anchor can be used as a positive example. At this stage, finding the exact coordinates of objects is not our goal. The goal is to learn a lightweight network to find areas that po-tentially contain vehicles, thereby reducing the large search space of WAMI images. Some methods do this by using overlaid road maps [START_REF] Zhang | An onroad vehicle detection method for high-resolution aerial images based on local and global structure learning[END_REF] where in addition to requiring extra information about images, they are not effective in scenarios that require the study of off-road objects. Our model adopts an anchor-free approach using heatmap-based proposal generation. Similar to [START_REF] Lalonde | Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information[END_REF]27], proposals are generated by regression on heatmap images. The ground-truth information of objects in a specific pixel position are probabilistically encoded into heatmaps. The proposal generator takes a stack of five consecutive images as input and, after processing, generates a heatmap in which the approximate location of an object or group of objects is determined with Gaussian peaks. Input images are aligned based on a feature-based method. Detected FAST features [START_REF] Rublee | Orb: An efficient alternative to sift or surf[END_REF] are used to compute descriptors. By matching descriptors with their corresponding pairs between two consecutive frames, a homography is estimated using the RANSAC algorithm [8] to warp them into a common reference. By having N the total number of the object centers in the ground-truth O i , i = {1, . . . , N } with center coordinate of x i ∈ R, the d dimensional heatmap H(x, σ) : R d → R for all pixels is obtained by

H(x, σ) = N i=1 1 (2π) d/2 σ d i e - ||x-x i || 2 2 2σ 2 i (1)
with high values around x i which decreases as the pixel goes farther away from a ground-truth keypoint center.

Considering that the objective at this stage is to detect clusters of objects, rather than accurate detection, the Euclidean loss between the network outputs and the groundtruth heatmap is used for loss calculations.

Learning latent encodings with VQ-VAE

The previous step gives us the regions of the input frames that may contain one or multiple objects, exempting us from an exhaustive search of the entire image. These smaller regions (128 × 128) are then searched in more details to determine the exact location of the objects. VQ-VAE [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] is utilized to represent these regions. VQ-VAE is a model that learns to compress high-dimensional data into a discrete hidden space in an unsupervised manner, which is potentially more suitable for many complex reasoning and predictive learning methods. In addition, it avoids the problem of "posterior collapse", which is one of the main problems of many VAE models due to the creation of powerful decoders that ignore the latents. In the case of videos, the goal is to learn a set of discrete latent representations from the raw pixels of video frames. To learn these discrete representations, instead of a one by one mapping between input data and latent variables, VQ-VAE learns a codebook and then forces the latents to be mapped to the nearest vector of the learned codebook. Given a video sample V , the encoder E(V ): q θ (z|V ) by passing through 3D convolutional layers (parameterized by θ), encodes the video frames into a series of latent continuous representations ẑ ∈ {1, . . . , K}. Then, to obtain posterior distributions (q θ (z|V )), the continuous latents are discretized using a codebook C = [e 1 , e 2 , . . . , e k ] (which is a collection of K continuous vectors), by looking up the nearest element (L2 distance) in the codebook to the latent (ẑ):

q θ (z = c|V ) = 1 for c = argmin i ||ẑ -e i || 2 0 otherwise (2)
Then, the decoder D(e): q φ (V |e c ) uses the codebook embeddings of the input to generate the original input V . VQ-VAE is trained by the following objective function:

L = ∥V -D(e)∥ 2 2 + ∥sg[E(V )] -e∥ 2 2 + β∥sg[e] -E(V )∥ 2 2
(3) where sg(.) is the stop-gradient operator and the function can be summarized as a three-terms loss:

L = L reconstruct + L codebook + L commitment (4)
The reconstruction loss is a log-likelihood function that encourages the VQ-VAE to learn accurate representations by optimizing encoder and decoder parameters. Codebook loss's aim is to optimize codebook vectors in order to make embeddings closer to their latent encoder outputs. Third term with β scaling hyperparameter, prevents the encoder from fluctuating representations and makes it to bind to the representations which are closer to the codebook.

Heatmap generation and keypoint detection transformer

Transformer architectures have shown great ability in modeling discrete data as well as high-dimensional data such as images [4,[START_REF] Chen | Generative pretraining from pixels[END_REF]. The common architecture in these models usually comprise multi-head self-attention blocks which are followed by MLP feed-forward network (FFN) blocks. Our method here is inspired by [4,31]. However, instead, here we investigate the representative power of quantized representations rather than CNN and Transformer descriptors. After training, VQ-VAE provides a codebook (C) that can be used for representing the frames of a sequence. These representations will be used as input to our temporal transformer architecture. First, the VQ-VAE backbone produces the frame level embeddings f ∈ R t×d×H×W (t ∈ {1, . . . , T } and d is embedding dimension). Because the Transformer encoder expects the inputs as a sequence, the embeddings are flattened to create a two-dimensional feature map. Due to the importance of order of the frames and to avoid permutation invariance problem of the Transformer architecture, the input features of each attention layer are complemented by positional encodings. The encoder follows a standard architecture consisting of a multi-head self-attention module and a fully connected FFN that computes the encodings (E). The main purpose of the Transformer decoder is to decode pixel features that represent instances of objects in each frame. Like VisTR [31], we use a fixed number of input embeddings (instance queries) to query instance features, except that the number of our queries is much higher due to the larger number of instances per frame. Therefore, the decoder takes encoded VQ-VAE embedding from the encoder and n object instance queries and generates object instance level features. Similar to VisTR, the output of the decoder (instance level features) and encoded embeddings (E) are fed to an attention layer and concatenated with VQ-VAE embeddings. To produce the heatmaps, these features are then used to train a three layered 3D convolutional network followed by layer normalization and ReLU activation. In this way, the mask features of each object in different frames can get obtained. The network is optimized with a loss which is an extension of Focal loss [START_REF] Lin | Focal loss for dense object detection[END_REF] with a shape prior regularizer for encouraging the network to better learn the object shapes provided in the ground-truth:

L = αL f + βL sh (5)
where α and β are hyperparameters determining weights of each loss in the optimization function. L f is the extended Focal loss used for minimization of the estimates between the predicted heatmap Ĥx,y and its corresponding groundtruth heatmap H x,y and is defined as:

L f (H, Ĥ) = --1 N xy     
(1 -Ĥx,y ) γ log( Ĥx,y ) if H x,y = 1 (1 -H x,y ) λ ( Ĥx,y ) γ log(1 -Ĥx,y ) otherwise (6) where γ and λ represent hyperparameters of the loss function and N is the total number of ground-truth object centers in the input image. L f is our shape prior regularizer which helps in efficient generation of the heatmaps. By assuming that all the ground-truth objects are circular (Gaussians), the network is urged to adapt circular shapes in the heatmaps. Having c as the center of a circle-shaped object O, for any pixels p belonging to the object O, all the q pixels on the line l connecting p to the center c should be within the maximum range of r from c where r is the maximum diameter of the object annotations in the ground-truth. Therefore, the shape prior is defined as:

L sh (H, Ĥ) = p∈O q∈l B p,q ×D p,q ×|P ( Ĥp |H)-P ( Ĥq |H)| (7)
B p,q is 1 if both pixels are predicted as an object pixel and 0 if classified as background. D p,q is 1 while pixel q on line l is within range r from the center and 0 otherwise. P ( Ĥp |H) and P ( Ĥq |H) are the predicted probabilities of pixels p and q being an object in the predicted heatmap given the current heatmap ground-truth. Then, for inference, we take the local maximum values of the generated heatmaps as the center of the predicted keypoint coordinates of the available objects.

Imbalance problem and shape regularization in WAMI

Imbalance problem in object detection occurs when distribution of an input property affects the performance of the detector. In natural images, and in the most common form, the imbalance of the foreground-to-background class creates a severe disparity between the number of positive and negative samples. In this case, thousands of negative samples can be extracted against a few positive samples where ignoring this imbalance in the distribution of samples can greatly affect the detector and disrupt its performance. In WAMI, although there is a relatively higher number of objects per frame compared to natural images (∼ 2k), the imbalance problem is more pronounced as these objects occupy only less than 7 × 10 -6 pixels of the total pixels in a frame [START_REF] Lalonde | Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information[END_REF]. Most regions are easy negatives without useful information that can overwhelm the training and there are only a few positive regions that are hard to differentiate (Fig. 1). While introducing a weight factor can balance the importance of positive versus negative samples, it has no effect on emphasizing differences between easy and hard examples (examples with large errors). Focal loss resolves this problem by reshaping the cross-entropy loss to down-weight easy samples and focus more on training hard negatives. This feature can also be valuable when it comes to WAMI images since the distribution of Gaussian peaks in heatmaps is unbalanced and covers a very small part of them. The scaling factor can be regularized by dynamically scaling the loss values. Therefore, by increasing confidence in the correct class, it can quickly decay to zero hence, automatically down-weight the effect of easy samples.

Furthermore, usually during training, optimization is performed only locally and in pixel-level of the input and the target, ignoring the global context and prior information. A shape prior, especially in the case of keypoint detection, provides global information on geometry of the target and guides the output mask to get closer to the shape of the class in the ground-truth. Given the similar shapes of the tiny targets occupying very small regions in the context of WAMI, this promotes the networks to learn easier and converge even faster.

Experiments

Dataset

In this work, we used the WPAFB 2009 dataset [START_REF]Wright-patterson air force base (wpafb) dataset[END_REF] to train and evaluate the proposed method. The images of the dataset were taken through six optical sensors and then the images were stitched together to cover a wide area of about 35 square kilometers. The dataset (1025 frames) is divided into two categories: training and testing, with 512 frames are dedicated for training and 513 frames for testing. Vehicles and their trajectory are manually annotated in each frame. Five different resolutions of the videos are included in the dataset, which we use the second largest image resol-ution (∼ 13K ×11K pixels). Our framework is capable of detecting moving as well as stationary vehicles. In order to have a fair comparison with state-of-the-art, the evaluation protocols in [40] are followed.

Evaluations

All the proposal regions are divided to images of size 128 × 128 and normalized between [-0.5, +0.5] prior to the training. For training 60k ground-truth heatmaps are randomly generated ensuring that there is at least one object inside the boundary of the selected patch. The best performance achieved when the frame length is 5. All the models are developed in PyTorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF] and optimized with Adam [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with learning rate of 10 -2 and 10 -4 for proposal network and VQ-VAE with Transformer networks respectively. For VQ-VAE the Codebook size is 1024 with vector dimension of 256 and the dimension of latents are set to 8 × 32 × 32. For an area of 128 × 128 the model can predicts up to 30 vehicles. Therefore, having a 5 frame sequence 150 object query is sent to the Transformer which comprises of 6 encoder and decoder layers with 4 attention heads. Transformer's hyperparameters are adjusted following VisTR. Batch size is set to 8 for all models where training and testing is done using two RTX 6000 GPUs with 192GB of RAM. A detected point is considered true positive (TP) when there is at least one ground-truth point in the 10-pixel range around that point. Detections outside of this range are considered as false positive (FP) and groundtruth annotation without any detection within its 10 pixels vicinity is considered as false negative (FN). Each annotated point in the ground-truth can only be assigned to one detection at inference. In case of multiple detections within the range only the closest one is considered as TP and the rest as FP (unless another annotation of the ground-truth is found that satisfies the conditions of a TP). The results are reported and compared in terms of precision, recall and F 1 measure. Table 1 shows the precision and recall of our proposed detector framework when applied to each AOI. For the most part, the detector achieves similar precision in different AOIs. However, for areas 03 and 41, the precision rate is a bit lower due to the detection of several non-vehicle moving objects which were not annotated in the ground-truth. Lower recall rates compared to precision is also the result of the incomplete ground-truth annotation where only some stationary vehicles are arbitrarily annotated in the presence of many others in the surrounding. The visualization of the detection results are shown in figure 3, with each row containing examples of each region in the WPAFB dataset. It can be seen that the detection is performed well even in challenging situations where the object instances are dense or sparse. Figure 4 visualizes the output attention map of the Transformer which shows how it acquires more accurate neuron activations in the feature map.

The maps may contain some local details which can cause noisy detections as some weak detection can get obscured by the neighbor activations. Retaining the local maximum value of the generated heatmaps and assigning zeros to the other positions prevents confusions during detection. We compare our framework against state-of-the-art object detection methods in remote sensing which we are aware of. Table 2 shows the results in terms of detection accuracy using F 1 measure. The compared methods are from both background-subtraction-based [START_REF] Bao | Single shot anchor refinement network for oriented object detection in optical remote sensing imagery[END_REF][START_REF] Saleemi | Multiframe many-many point correspondence for vehicle tracking in high density wide area aerial videos[END_REF][START_REF] Shi | Context-driven moving vehicle detection in wide area motion imagery[END_REF]40] and spatiotemporal-based [START_REF] Lalonde | Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information[END_REF]39] categories where some use an extra post-processing step. Our proposed framework achieves competitive results compared to the others (highest accuracy in 4 out of 6 areas) given that it is anchor-free and does not use background subtraction or any post-processing technique.

Ablation study

This section provides extensive ablation experiments which are conducted to evaluate the key specifications of our framework. Sequence length. The main difference between video and image stems from the temporal information contained in the videos. In spatiotemporal methods, the effective use of temporal information by the model is the most important factor in understanding videos. To show that, we train the model with different amount of temporal information. Table 3 demonstrates the effect of sequence lengths on the detector performance and their comparison. Based on the results, as the length varies from 3 to 16, first there is an increase in the F 1 score and then, there is a monotone decrease. This is due to small input patch size and low frame rate of the videos. When the sequence length exceeds 5, almost none of the objects searched in the first frame are available, and stacking more frames does not help with better detection. Codebook length. Table 4 shows experimental results of training the VQ-VAE codebook with different number of codes. It can be seen that increasing the number of codes improves the performance of the detector, however, further increase of the latent size does not affect the quality of detections in a positive way. This indicates that for this dataset, a medium size codebook is enough for representation of spatiotemporal information and a longer latent size surpasses the required base threshold for learning the representations. Effect of feature encoding. To illustrate the effect of feature encoding, the detection module is trained by three types of input features: VQ-VAE encodings from the trained codebook, CNN-encoded, and Transformer-encoded features. As reported in Table 5 VQ-VAE encoded features achieve superior performance compared to both CNNencoded and Transformer-encoded features in all AOIs. It is also interesting to see the superiority of the Transformerencoded features to the CNN-encoded features. This might be related to the Transformer updates based on the pairwise similarities of its self-attention module which helps to learn spatial and temporal features together. The results show the power of discreet representations in modeling spatiotemporal information that achieve even better performance than the Transformer encoder which learns spatiotemporal information as a whole. Loss function with shape prior. As discussed, imbalance problem poses a significant challenge for detector when the target object occupies considerably smaller amount of spatial space compared to the background. While crossentropy based losses are being more affected by this problem, Focal loss solutions alleviate the issue by downweighting the easy samples. In case of object detection formulated as a keypoint detection problem, adding a shape prior can be an effective solution. Table 6 compares the effect of different loss functions on the performance of the detection framework: cross-entropy loss, Focal loss, and Focal loss with a shape prior. Adding the shape prior significantly improves the performance and outperforms the detection in all regions when it is compared to the other loss function.

Conclusion

In this work, we proposed a novel framework for detecting tiny vehicles in WAMI sequences. We considered vehicles as keypoints in the relative background which are described by Gaussian distributions at the object position annotated in the ground-truth. To exploit the spatiotemporal information we trained a VQ-VAE network that learns a codebook of latent representations. To generate the detection masks, a Transformer network with a revamped loss function was trained which not only addresses the foreground-background imbalance problem but also tries to conform to the shape of the objects in the ground-truth benefiting from a shape regularizer. Extensive experiments demonstrated the performance of the proposed framework with ablation studies investigating different design choices. Compared to the existed methods that we are aware of, our detector achieves competitive results in the detection task. As the framework utilizes spatiotemporal information, we will explore joint detection and tracking in future research.

Figure 1 .

 1 Figure 1. Sometimes small vehicles are indistinguishable from noise: in the green box, the top row shows real vehicles. The bottom row is just noise. Effective modeling of temporal features is critical for more accurate detection of moving objects (yellow boxes).

Figure 2 .

 2 Figure 2. shows the overall architecture of our method which is comprised of three main modules: ROI proposal generator which is a CNN network that identifies areas that are most likely to contain objects. Backbone module that computes representative features of the generated proposals by training a VQ-VAE model. The detection module that uses feature embeddings to generate the detection heatmaps.

Figure 3 .

 3 Figure 3. Visualization of qualitative detection results on different regions (Regions AOI 01, AOI 02, AOI 34, and AOI 40 from first to forth rows repectively). In the figures, True Positives (TP) are indicated in green, False Positives (FP) in red and False Negatives (FN) in blue.

Figure 4 .

 4 Figure 4. Illustrates attention maps of Transformer before 3DConv layers.

Table 1 .

 1 The precisions, recalls, and F1 measures on six AOI using the proposed detection method.

Table 2 .

 2 Comparison with state-of-the-art for six AOI in the WPAFB dataset in terms of F1 measure. The top performance in each area is indicated in bold.

	Method	01	02	03	34	40	41
	[25]	0.645 0.760 0.861	-	-	-
	[13]	0.743 0.825 0.876 0.763 0.737 0.708
	[24]	0.783 0.793 0.876 0.755 0.749 0.762
	[32]	0.738 0.820 0.868 0.761 0.733 0.700
	[21]	0.816 0.868 0.892	-	-	-
	[22]	0.850 0.876 0.889 0.826 0.817 0.799
	[28]	0.866 0.890 0.900	-	-	-
	[40]	0.935 0.947 0.945 0.953 0.935 0.934
	[16]	0.947 0.951 0.942 0.933 0.983 0.928
	[39]	0.944 0.967 0.964 0.967 0.938 0.948
	Ours	0.943 0.970 0.973 0.968 0.944 0.950

Table 3 .

 3 Sequence length. Due to low frame rate, the performance (F1 measure) drops as sequence length increases.

	length	01		02	03	34	40	41
	3	0.924 0.917 0.937 0.908 0.925 0.894
	5	0.943 0.970 0.973 0.968 0.944 0.950
	8	0.897 0.884 0.895 0.872 0.891 0.864
	16	0.852 0.869 0.846 0.832 0.858 0.817
	# of codes	01	02	03	34	40	41
	256	0.864 0.886 0.904 0.891 0.874 0.919
	1024	0.943 0.970 0.973 0.968 0.944 0.950
	4096	0.938 0.953 0.959 0.927 0.946 0.922

Table 4 .

 4 Codebook length. While increasing codebook size for embeddings improves the performance, further increase of code size does not help it.

	Backbone	01	02	03	34	40	41
	VQ-VAE	0.943 0.970 0.973 0.968 0.944 0.950
	CNN	0.868 0.821 0.847 0.886 0.844 0.871
	Transformer 0.907 0.885 0.894 0.865 0.883 0.898

Table 5 .

 5 VQ-VAE vs CNN vs. Transformer backbones. VQ-VAE provides the best feature quality.

	Loss	01	02	03	34	40	41
	CE	0.883 0.913 0.904 0.892 0.914 0.921
	FL	0.908 0.921 0.937 0.916 0.934 0.924
	FL+SP 0.943 0.970 0.973 0.968 0.944 0.950

Table 6 .

 6 Loss function. Cross-entropy vs Focal loss vs improved version of Focal loss with shape prior.

Acknowledgements This work has been supported by the BPI France funding under the LiChiE project. The authors are also grateful to the OPAL infrastructure provided by Université Côte d'Azur.