Eduardo Hugo Sanchez
email: eduardo.sanchez@irt-saintexupery.com

Read, look and detect: Bounding box annotation from image-caption pairs

Various methods have been proposed to detect objects while reducing the cost of data annotation. For instance, weakly supervised object detection (WSOD) methods rely only on image-level annotations during training. Unfortunately, data annotation remains expensive since annotators must provide the categories describing the content of each image and labeling is restricted to a fixed set of categories. In this paper, we propose a method to locate and label objects in an image by using a form of weaker supervision: image-caption pairs. By leveraging recent advances in vision-language (VL) models and self-supervised vision transformers (ViTs), our method is able to perform phrase grounding and object detection in a weakly supervised manner. Our experiments demonstrate the effectiveness of our approach by achieving a 47.51% recall@1 score in phrase grounding on Flickr30k Entities and establishing a new state-of-the-art in object detection by achieving 21.1 mAP 50 and 10.5 mAP 50:95 on MS COCO when exclusively relying on image-caption pairs.

Introduction

Locating and classifying objects within an image is a fundamental task in computer vision that enables the development of more complex tasks such as image captioning [START_REF] Yang | Image captioning with object detection and localization[END_REF], visual reasoning [START_REF] Harold | Grounded language-image pre-training[END_REF], among others. Nevertheless, the success of object detection models [START_REF] Jocher | ultralytics/yolov5: v7.0 -YOLOv5 SOTA Realtime Instance Segmentation[END_REF][START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] typically relies on human supervision in the form of bounding box annotations. In particular, data annotation is a time-consuming and arduous task that requires annotators to draw bounding boxes around objects and label each bounding box with a category from a fixed set of categories. Furthermore, modifying the number of categories may require annotators to relabel or add new bounding boxes.

Several approaches have been proposed to reduce the cost of data annotation in object detection by using image-level labels [START_REF] Bilen | Weakly supervised deep detection networks[END_REF][START_REF] Diba | Weakly supervised cascaded convolutional networks[END_REF][START_REF] Tang | Pcl: Proposal cluster learning for weakly supervised object detection[END_REF][START_REF] Zhang | W2f: A weaklysupervised to fully-supervised framework for object detection[END_REF][START_REF] Gao | C-midn: Coupled multiple instance detection network with segmentation guidance for weakly supervised object detection[END_REF][START_REF] Zeng | Wsod2: Learning bottom-up and top-down objectness distillation for weakly-supervised object detection[END_REF][START_REF] Ren | Instance-aware, context-focused, and memory-efficient weakly supervised object detection[END_REF][START_REF] Huang | Comprehensive attention self-distillation for weakly-supervised object detection[END_REF], a dataset containing both labeled and unlabeled data [START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF][START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF][START_REF] Liu | Unbiased teacher v2: Semi-supervised object detection for anchor-free and anchor-based detectors[END_REF] or sparsely-annotated data [START_REF] Xu | Missing labels in object detection[END_REF][START_REF] Zhang | Solving missing-annotation object detection with background recalibration loss[END_REF][START_REF] Wang | Co-mining: Self-supervised learning for sparsely annotated object detection[END_REF][START_REF] Li | Siod: single instance annotated per category per image for object detection[END_REF]. For instance, WSOD methods only use image-level annotations along with the multiple instance learning (MIL) [START_REF] Maron | A framework for multiple-instance learning[END_REF] approach. However, the annotation effort is still significant and similar to that required for supervised classification.

In this paper, we take a step forward by learning to locate and label objects within an image from image-caption pairs. Not only captions provide a more natural description of the image content than image-level labels but also constitute a form of weaker supervision since image-caption pairs are easier to collect in vast amounts (e.g. from the Web [START_REF] Sharma | Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[END_REF]). Our approach combines recent advances in vision-language (VL) models [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF] and self-supervised vision transformers (ViTs) [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF].

VL models leverage large-scale image-caption datasets and have strong performance on zero-shot image classification, image-text retrieval, and visual reasoning tasks. These models align images with their corresponding captions via contrastive learning. Notably, models that include a cross-modality

LOST

Ours LOST Ours

Figure 1: Our approach leverages language from captions via ALBEF to annotate multiple objects per image (e.g. we detect a dog and a frisbee while LOST generates a single categoryless bounding box).

encoder seem to implicitly learn a more fine-grained word-region alignment without using additional supervision [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF]. We propose to use the location ability of VL models to automatically annotate objects of interest mentioned in captions. Moreover, VL models do not require retraining when the number of categories to annotate changes as they are already included in the VL model's vocabulary.

Despite the strong ability of VL models to locate objects, they aim at the most distinctive part of the object rather than the whole object. For example, ALBEF [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF] and VilBERT [START_REF] Lu | Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF] perform phrase grounding by ranking the object proposals provided by the supervised detector MattNet [START_REF] Yu | Mattnet: Modular attention network for referring expression comprehension[END_REF]. On the other hand, recent work has shown that representations from self-supervised ViTs contain explicit information about the scene layout of images and produce heatmaps that highlight salient objects [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF]. LOST [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF] and TokenCut [START_REF] Wang | Selfsupervised transformers for unsupervised object discovery using normalized cut[END_REF] show the effectiveness of self-supervised ViT representations to perform unsupervised object discovery and detection without any labels.

We make the following contributions in this work. First, we propose a novel method to locate and label objects in images by combining the ability of VL models to point at objects and the ability of self-supervised ViTs to extract whole objects in Section 3. By building upon ALBEF [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF] and LOST [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF], our method is able to locate multiple objects and generate accurate bounding boxes without human supervision. Figure 1 illustrates the improved ability of our model over LOST. Second, we use our approach to perform phrase grounding and object detection in a weakly supervised fashion.

In Section 4, we demonstrate that our method achieves competitive performance in phrase grounding on Flickr30k Entities [START_REF] Bryan A Plummer | Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models[END_REF] and establish a new state-of-the-art in object detection on MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] when exclusively relying on image-caption pairs as unique source of supervision. Additionally, we perform ablation experiments to investigate the key components of our approach, transfer learning experiments on PASCAL VOC2007 [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] and pseudo-labeling experiments to improve the performance in WSOD. In Section 5, we discuss the limitations, future work and conclusions of our work.

Related work

Weakly supervised object detection: To reduce the cost of data annotation, several methods propose to train object detectors using only image-level annotations without the need for bounding box annotations. WSDDN [START_REF] Bilen | Weakly supervised deep detection networks[END_REF] introduces the first end-to-end WSOD framework that adopts MIL [START_REF] Maron | A framework for multiple-instance learning[END_REF]. Since then, several improvements have been proposed: PCL [START_REF] Tang | Pcl: Proposal cluster learning for weakly supervised object detection[END_REF] performs clustering to improve object proposals and W2F [START_REF] Zhang | W2f: A weaklysupervised to fully-supervised framework for object detection[END_REF] leverages pseudo-label mining from a WSOD model to train a supervised object detector. C-MIDN [START_REF] Gao | C-midn: Coupled multiple instance detection network with segmentation guidance for weakly supervised object detection[END_REF] introduces a method for coupling proposals to prevent the detector from capturing the most discriminative object part rather than the whole object. WSOD 2 [START_REF] Zeng | Wsod2: Learning bottom-up and top-down objectness distillation for weakly-supervised object detection[END_REF] performs pseudo-label mining and incorporates a bounding box regressor to fine-tune the location of each proposal. Likely, MIST [START_REF] Ren | Instance-aware, context-focused, and memory-efficient weakly supervised object detection[END_REF] performs pseudo-label mining where highly overlapping proposals are assigned to the same label. CASD [START_REF] Huang | Comprehensive attention self-distillation for weakly-supervised object detection[END_REF] combines self-distillation with multiple proposal attention maps generated via data augmentation. Closely related to our work, Cap2Det [START_REF] Ye | Cap2det: Learning to amplify weak caption supervision for object detection[END_REF] learns from image-caption pairs by extracting image-level annotations from captions using a supervised text classifier. These predicted image-level annotations are subsequently used to train a WSOD model based on MIL. Additionally, Cap2Det [START_REF] Ye | Cap2det: Learning to amplify weak caption supervision for object detection[END_REF] refines the WSOD model by retraining on instance-level pseudo-labels multiple times. Most of the existing WSOD methods rely on object proposal algorithms (e.g. Selective Search [START_REF] Jasper Rr Uijlings | Selective search for object recognition[END_REF] or Edge Boxes [START_REF] Zitnick | Edge boxes: Locating object proposals from edges[END_REF]). By exclusively leveraging self-supervision on imagecaption pairs, our approach outperforms the state-of-the-art model Cap2Det [START_REF] Ye | Cap2det: Learning to amplify weak caption supervision for object detection[END_REF] without the need for a supervised text classifier. Additionally, our approach outperforms relevant WSOD baselines [START_REF] Tang | Pcl: Proposal cluster learning for weakly supervised object detection[END_REF][START_REF] Gao | C-midn: Coupled multiple instance detection network with segmentation guidance for weakly supervised object detection[END_REF] that use a form of stronger supervision (image-level annotations) and object proposal algorithms.

Learning from unlabeled or partially labeled data: Some approaches alleviate the lack of bounding box annotations by leveraging a small labeled dataset and a large unlabeled dataset via semi-supervised learning [START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF][START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF][START_REF] Liu | Unbiased teacher v2: Semi-supervised object detection for anchor-free and anchor-based detectors[END_REF] and active learning [START_REF] Wang | Towards human-machine cooperation: Self-supervised sample mining for object detection[END_REF][START_REF] Huy | Active learning strategies for weakly-supervised object detection[END_REF]. Li et al. [START_REF] Li | Siod: single instance annotated per category per image for object detection[END_REF] propose to train an object detector using only a single instance annotation per category per image. Other methods combine image-level and instance-level pseudo-annotations during training [START_REF] Xu | Missing labels in object detection[END_REF][START_REF] Ren | Instance-aware, context-focused, and memory-efficient weakly supervised object detection[END_REF]. Sohn et al. [START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF] propose a two-stage training in which an object detector is trained on available labeled data. This model is subsequently used to select high-confidence bounding boxes on unlabeled data as pseudolabels. Wang et al. [START_REF] Wang | Co-mining: Self-supervised learning for sparsely annotated object detection[END_REF] address the missing annotation problem by introducing a siamese network where each branch is used to generate pseudo-labels for each other. Likely, recent work [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF][START_REF] Liu | Unbiased teacher v2: Semi-supervised object detection for anchor-free and anchor-based detectors[END_REF] leverage the teacher-student framework in object detection. In this work, we also explore the use of pseudo-labels to improve WSOD performance.

VL models: Learning joint VL representations from image-caption pairs in a self-supervised fashion has proven to be effective to perform multiple downstream tasks [START_REF] Lu | Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Lu | 12-in-1: Multi-task vision and language representation learning[END_REF] such as visual question answering, image retrieval, image captioning, zero-shot classification, etc. VL models [START_REF] Tan | Lxmert: Learning cross-modality encoder representations from transformers[END_REF][START_REF] Harold | Visualbert: A simple and performant baseline for vision and language[END_REF][START_REF] Su | Vl-bert: Pre-training of generic visual-linguistic representations[END_REF][START_REF] Kim | Vilt: Vision-and-language transformer without convolution or region supervision[END_REF][START_REF] Gan | Large-scale adversarial training for vision-and-language representation learning[END_REF][START_REF] Li | Oscar: Object-semantics aligned pre-training for vision-language tasks[END_REF][START_REF] Chen | Uniter: Universal image-text representation learning[END_REF][START_REF] Jia | Scaling up visual and vision-language representation learning with noisy text supervision[END_REF][START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF][START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF] are generally trained on a combination of loss functions: masked language modelling (MLM), where a masked word token is predicted; masked image modelling (MIM), where a masked image region feature or object category is predicted, image-text contrastive learning (ITC), where positive/negative image-caption pairs are assigned to high/low similarity scores, respectively; and image-text matching (ITM), that predicts whether an image and a caption match. Many strategies have been proposed to achieve improved VL representations. VisualBERT [START_REF] Harold | Visualbert: A simple and performant baseline for vision and language[END_REF] uses a supervised object detector to extract visual embeddings. VILLA [START_REF] Gan | Large-scale adversarial training for vision-and-language representation learning[END_REF] performs adversarial training in the representations space. OSCAR [START_REF] Li | Oscar: Object-semantics aligned pre-training for vision-language tasks[END_REF] uses object tags to ease VL alignment. UNITER [START_REF] Chen | Uniter: Universal image-text representation learning[END_REF] encourages alignment between words and image regions extracted by an object detector. More recently, CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] leverages a massive amount of image-caption pairs and achieves impressive performance at zero-shot classification. However, CLIP underperforms at other VL tasks as the interaction between vision and language is very shallow (i.e. a simple dot product). Li et al. [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF] propose a new model called ALBEF, which builds upon previous models [START_REF] Tan | Lxmert: Learning cross-modality encoder representations from transformers[END_REF][START_REF] Lu | Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF][START_REF] Kim | Vilt: Vision-and-language transformer without convolution or region supervision[END_REF][START_REF] Jia | Scaling up visual and vision-language representation learning with noisy text supervision[END_REF] and is composed of a vision encoder, a language encoder, and a cross-modality encoder for deeper VL interaction. By leveraging a large image-caption dataset [START_REF] Sharma | Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[END_REF], ALBEF outperforms previous models at many VL tasks without the need for a supervised object detector to extract region-based image representations. Our approach leverages a pre-trained ALBEF model to locate the image region that corresponds to a word or a textual description.

Open vocabulary detection: Classifying an object or image has been traditionally limited to a small set of fixed categories. Zhang et al. [START_REF] Zhang | Online collaborative learning for open-vocabulary visual classifiers[END_REF] leverages the vocabulary from image-caption datasets to perform image classification across more than 30k classes. The recent success of VL models [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF][START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF] has motivated other methods to leverage image-caption pairs to perform object detection on a larger number of categories. Zareian et al. [START_REF] Zareian | Open-vocabulary object detection using captions[END_REF] use bounding box annotations from base classes to perform correctly in target classes mentioned in captions. Gao et al. [START_REF] Gao | Open vocabulary object detection with pseudo bounding-box labels[END_REF] use a supervised object detector trained on MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] to generate pseudo-bounding box annotations for categories mentioned in captions. Similar approaches [START_REF] Zhong | Regionclip: Region-based language-image pretraining[END_REF][START_REF] Shi | Proposalclip: unsupervised open-category object proposal generation via exploiting clip cues[END_REF] have been proposed by extending CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF]. We also leverage VL models to annotate objects using an arbitrary number of categories in a self-supervised manner without relying on bounding box annotations like previous methods.

Object discovery: Recently, several studies explore methods for object localization that rely solely on visual cues. LOST [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF] extracts image representations via a self-supervised ViT [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF] which are "A woman is steering a boat with a pole" subsequently used to identify the image patches corresponding to an object based on their correlation. Wang et al. [START_REF] Wang | Selfsupervised transformers for unsupervised object discovery using normalized cut[END_REF] also leverage DINO representations which are used to build a graph. A normalized graph-cut is used to split the foreground object from the background. Both methods can only locate a single object per image without providing its category. Our approach builds upon LOST by integrating the language modality, enabling it to locate and label multiple objects per image.

Caption C L IT M (X, C) Φ c = ∂L IT M (X,C)

Method

To annotate objects from image-caption pairs, our approach consists of two main stages. First, we leverage the cross-modality encoder from a pre-trained VL model to automatically select the image patches (or seeds) that may belong to a given object (defined by a word token or a set of word tokens).

The seed selection process is described in Section 3.1. Second, we use a self-supervised ViT to compute the similarity between image patches. Intra-image similarity is used to filter out image patches selected in the first stage and generate a heatmap corresponding to the object. This process is known as seed expansion. Then, a heatmap threshold is computed via a Gaussian mixture model (GMM) to separate the object patches from the background ones. Finally, a bounding box enclosing the object patches is generated. Section 3.2 describes the process to generate a heatmap and extract an object from it. Figure 2 shows an overview of our approach.

Pointing at objects with VL models

Our proposed method is motivated by the observation that VL models implicitly learn to align words in the captions with patches in the images even though these models are only trained to align images with their corresponding captions [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF]. Furthermore, we can annotate a large amount of objects since the number of objects categories is as large as the vocabulary used in the captions during training of VL models. We leverage the ability of VL models to point at objects and the fact that most of the salient objects in an image are mentioned in its respective caption [START_REF] Li | Oscar: Object-semantics aligned pre-training for vision-language tasks[END_REF].

In this section, we explain how the fine-grained alignment between words and patches is computed in VL models implementing a cross-modality encoder (e.g. ALBEF [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF]) and how we leverage it to point at objects in an image. Let X = {p 1 , p 2 , . . . , p N P } be an image composed of N P patches and C = {w 1 , w 2 , . . . , w N T } be its corresponding caption composed of N T word tokens. An image encoder and text encoder are used to extract image and text representations which are both fed into the cross-modality encoder. In the l vl -th cross-attention layer of this encoder, we compute the value and key representations for each image patch, i.e. V = {v 0 , v 1 , . . . , v N P } and K = {k 0 , k 1 , . . . , k N P }, respectively, where v 0 and k 0 are the representations of the classification token [CLS].

Given a word token of interest w c (e.g. 'person', 'dog', etc.), we compute its query representation q c . The relation between the word token w c and the image patches {p i } N P i=1 is given by the hidden representation h c as shown in Equation 1where d is the dimension of the query representations.

h c = N P i=0 a c,i • v i where a c,i = exp(q ⊺ c k i / √ d) N P j=0 exp(q ⊺ c k j / √ d) (1)
As observed, the hidden representation of w c is a linear combination of the value representations corresponding to the image patches. Furthermore, these representations are weighted according to the attention scores a c,i that implicitly provide the similarity between w c and p i via the product q ⊺ c k i . Through the use of the cross-modality encoder, one can identify the image regions that are most closely aligned with a particular word token. We use Grad-CAM [START_REF] Ramprasaath R Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF] to rank the image patches in order of importance. Equation 2 displays the importance score of the image patch p i with respect to the word token w c where L IT M (X, C) is the binary cross-entropy loss that measures whether the image X and the caption C match or not. When ranking image patches, we do not take into account the attention score corresponding to the classification token [CLS], a c,0 .

Φ c,i = ∂L IT M (X, C) ∂a c,i • a c,i (2)
Unfortunately, Grad-CAM scores are insufficient to generate an accurate bounding box by themselves (see Section 4). For example, Gao et al. [START_REF] Gao | Open vocabulary object detection with pseudo bounding-box labels[END_REF] use a supervised Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] to generate bounding boxes that cover the activated image patches by the word token w c for object detection. Similarly, Li et al. [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF] rank MattNet [START_REF] Yu | Mattnet: Modular attention network for referring expression comprehension[END_REF] proposals based on Grad-CAM maps for phrase grounding.

However, we observe that while Grad-CAM scores do not highlight the image patches corresponding to the whole object, they are useful to point at the most discriminative parts of it. Therefore, we propose to use a set D = {f i } M i=1 of M image patches with the highest score Φ c,i for a given word token of interest w c to point at the object. The image patches in D are referred to as potential seeds and this process is referred to as seed selection. Pointing is a natural way for humans to refer to an object [START_REF] Bearman | What's the point: Semantic segmentation with point supervision[END_REF] and constitutes the first stage of our proposed approach.

Extracting objects with self-supervised ViTs

We make use of the self-supervised ViT capability [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF] to measure the similarity between image patches. Using the location information provided in the previous stage, our approach takes advantage of the fact that object patches correlates positively with each other but negatively with background patches. This idea is successfully applied in LOST [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF] to perform object discovery. Our work is inspired by LOST and extends its capabilities by incorporating the language modality. Assuming that the object area is smaller than the background area, LOST uses the patch with the smallest number of positive correlations with other patches in order to point at an object. However, this assumption may not always hold in practice (e.g. an object covering more area than the background, multiple objects, etc.). Compared to LOST, our method is able to generate multiple bounding boxes per image (as many objects as mentioned in the caption). Furthermore, our method can annotate each object with a label while LOST can only retrieve a single object without specifying its category. Figure 1 displays the differences between our approach and LOST.

In this work, we average the first N patch locations with the highest value of Φ c,i in D to compute the initial seed s for a given w c . Following LOST, we extract the key representations of the initial and potential seeds, i.e. k s and {k fi } M i=1 , respectively, from the l vit -th self-attention layer of a self-supervised ViT [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF]. Then, the similarity between the initial seed and potential seeds is computed via the dot product of their respective representations to determine the image patches belonging to the object. We assume that potential seeds that are positively correlated to the initial seed belong to the object while potential seeds that are negatively correlated to the initial seed belong to the background. Thus, patches belonging to the object are defined by the set O = {s} ∪ {f i | f i ∈ D and k ⊺ s k fi ≥ 0}. Each patch p ∈ O generates a heatmap Ψ p ∈ R N P , where the i-th dimension Ψ p i is computed via the dot product between its key representation k p and the key representation of the patch p i (also extracted by the ViT), i.e. k pi ∀i ∈ {1, . . . , N P } as shown in Equation 3.

Ψ p i = k ⊺ p k pi (3
) Finally, the heatmap of the object w c is defined by the sum of the heatmaps corresponding to the patches in O as shown in Equation 4. This process is referred to as seed expansion.

Ψ c = p∈O Ψ p (4)
To extract the object from the heatmap Ψ c , we define a threshold t. While LOST sets t=0, we assume that patches belonging to the object and background are defined by two normal distributions

p o =N (µ o , σ 2 o ) and p b =N (µ b , σ 2 b )
, respectively. The parameters µ o , σ o , µ b , σ b ∈ R are estimated via a GMM per heatmap with k=2 components. Then, the threshold is calculated by solving p o (t)=p b (t) such that µ b < t < µ o . For small objects, p o is barely noticeable and hard to estimate via GMM since only the background component is recognizable. We assume only one component is distinguishable when the overlapping between the estimated distributions p o and p b is significant (i.e. µ b + 1.5σ b < µ o -1.5σ o ). In such a case, we use the threshold t = µ + γσ where γ is a constant and µ and σ are the mean and the standard deviation of Ψ c , respectively. Supplementary material provides bounding box examples using multiple t values. To generate a bounding box, a mask m c is obtained by thresholding the heatmap Ψ c as shown Equation 5where Ψ c i is the i-th dimension of the heatmap Ψ c . Later, a bounding box is drawn by enclosing the segment that includes the initial seed s.

m c i = 1 Ψ c i ≥t (5)
4 Experiments and results

Setup details

Tasks and datasets: We perform weakly supervised phrase grounding and object detection to demonstrate the effectiveness of our method to annotate objects. In Section 4.2, we present our experimental results for phrase grounding on Flickr30k Entities [START_REF] Bryan A Plummer | Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models[END_REF], an extension of Flickr30k [START_REF] Young | From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions[END_REF] which consists of ≈ 32k images collected from Flickr each of which is described with 5 captions. Image-caption samples are split into ≈ 30k training, 1k validation, and 1k test samples. Flickr30k Entities includes manually-annotated bounding boxes that are linked with entities mentioned in captions. Results are reported in terms of recall@1 on the test set. In Section 4.3, we perform WSOD on MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] which contains 113k training and 5k validation images. Each image is described with 5 captions. Additionally, the dataset provides bounding box annotations covering 80 object categories such as person, bicycle, car, plane, etc. We also conduct transfer learning experiments using samples from MS COCO to train an object detector that predicts PASCAL VOC2007 [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] categories since this dataset does not provide captions. Model architecture: To point at objects, we use ALBEF pre-trained on 14M image-caption pairs [START_REF] Li | Align before fuse: Vision and language representation learning with momentum distillation[END_REF] and fine-tuned on 20k image-caption pairs [START_REF] Yu | Modeling context in referring expressions[END_REF]. It is worth mentioning that any VL model that includes a cross-modality encoder can be used. To perform seed expansion, we use the self-supervised ViT from DINO (i.e. ViT-S/16 [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF]). For comparative purposes, we also use the image encoder from ALBEF (i.e. ViT-B/16 [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF]) to compute the similarity between image patches. In WSOD, our approach generates bounding box annotations to train a YOLOv5 object detector [START_REF] Jocher | ultralytics/yolov5: v7.0 -YOLOv5 SOTA Realtime Instance Segmentation[END_REF] in a supervised manner.

Hyperparameters:

We set the VL cross-attention layer to l vl =8 and the ViT self-attention layer to l vit =11. To compute the initial seed, we average the first N =3 patch locations from D and set the number of potential seeds to M =10. To compute the threshold, we use γ = 1.75. Our experiments are executed on a NVIDIA GeForce RTX 3090. [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF] InfoGround [START_REF] Gupta | Contrastive learning for weakly supervised phrase grounding[END_REF] Flickr30k Entities [START_REF] Bryan A Plummer | Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models[END_REF] Yes, Faster R-CNN [START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] 47.88 InfoGround [START_REF] Gupta | Contrastive learning for weakly supervised phrase grounding[END_REF] MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] Yes, Faster R-CNN [START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] 51.67

Weakly supervised phrase grounding

We conduct experiments on Flickr30k Entities to evaluate the ability of our approach to associate phrases describing objects to image regions. While a single word can define the category of an object, a phrase provides additional attributes (e.g. color, size, position, etc.). Our method processes phrases by simply adding up the heatmaps of each word w ci in the phrase P , i.e. Ψ phrase = ci∈P Ψ ci .

In Table 1, we report our results in terms of recall@1 that represents the ratio of the number of phrases whose ground truth bounding boxes have significant overlap with the generated bounding boxes by our model (i.e. IoU ≥ 0.5) to the total number of phrases.

Our baseline model (referred to as ALBEF C-A maps) uses the cross-modality encoder to produce heatmaps Φ phrase , which are then thresholded to generate bounding boxes. As shown in Section 3.1, our approach builds upon Φ phrase via a self-supervised ViT to generate the expanded heatmaps Ψ phrase . We evaluate two variants of our approach by using the ViT from ALBEF and DINO to generate the object heatmaps (referred to as ALBEF ViT maps and DINO ViT maps, respectively).

As observed, the variants ALBEF ViT maps and DINO ViT maps achieve higher performance compared to the baseline (improvements of 7.11% and 10.65%, respectively). As hypothesized, the baseline model exhibits limitations in accurately capturing the spatial extent of objects despite its ability to point at them in the image as shown in Figure 3. Moreover, DINO ViT maps outperform ALBEF ViT maps by a margin of 3.54%. This difference suggests that DINO's loss function is more effective to capture the underlying relationships between image patches.

For the sake of comparison, we also report the performance of the state-of-the art model for weakly supervised phrase grounding, i.e. InfoGround [START_REF] Gupta | Contrastive learning for weakly supervised phrase grounding[END_REF]. Our approach achieves a competitive score of 47.51% comparable to InfoGround performance (47.88% and 51.67% when trained on Flickr30k Entities and MS COCO, respectively). Nevertheless, InfoGround uses a Faster R-CNN [START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] pretrained on Visual Genome [START_REF] Krishna | Visual genome: Connecting language and vision using crowdsourced dense image annotations[END_REF] to generate object proposals and extract object features. Thus, our approach offers an efficient solution for phrase grounding without the need for an object detector. Our approach represents a promising alternative to InfoGround, particularly in scenarios where the object detector does not include some categories or where obtaining bounding box annotations is difficult.

Weakly supervised object detection

We investigate the ability of our approach to perform WSOD. Our methodology involves defining a set of object categories and searching through captions to identify if any of these categories are mentioned. If a category is found, our approach generates a corresponding bounding box as described in Section 3. Then, we train an object detector (i.e. Yolov5 [START_REF] Jocher | ultralytics/yolov5: v7.0 -YOLOv5 SOTA Realtime Instance Segmentation[END_REF]) from scratch in a supervised manner using the generated bounding box annotations. We evaluate our approach on MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] and PASCAL VOC 2012 [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF]. While our method is capable of labeling a large number of object categories, we use these datasets as they provide bounding box annotations for evaluation purposes.

Comparison with WSOD methods: We compare our approach with state-of-the-art WSOD methods to demonstrate its effectiveness in annotations. Our approach demonstrates better performance without the need for an object proposal algorithm, a supervised text classifier or using refinement. Compared to methods that learn from image-level annotations [START_REF] Gao | C-midn: Coupled multiple instance detection network with segmentation guidance for weakly supervised object detection[END_REF][START_REF] Zeng | Wsod2: Learning bottom-up and top-down objectness distillation for weakly-supervised object detection[END_REF][START_REF] Ren | Instance-aware, context-focused, and memory-efficient weakly supervised object detection[END_REF][START_REF] Huang | Comprehensive attention self-distillation for weakly-supervised object detection[END_REF], our approach demonstrates competitive performance and outperforms relevant baselines such as PCL [START_REF] Tang | Pcl: Proposal cluster learning for weakly supervised object detection[END_REF] and C-MIDN [10] (8.5 mAP 50:95 and 9.6 mAP 50:95 , respectively) by achieving 10.5 mAP 50:95 . It is worth noting that these WSOD methods rely on pseudo-labeling techniques and image-level annotations that constitute a form of stronger supervision.

For the sake of comparison, we also report the results of Yolov5 trained in a fully-supervised manner.

Transfer learning and pseudo-labeling (P-L): Due to the lack of captions in PASCAL VOC2007, our approach generates annotations by searching PASCAL VOC2007 object categories from MS COCO image-caption pairs. Results in terms of mAP 50 per category are reported in Table 3 where best results are highlighted in bold. Our approach achieves 40.9 mAP 50 outperforming Cap2Det EM (39.9 mAP 50 ) while being behind Cap2Det CLSF (43.1 mAP 50 ). To further improve performance, we propose a simple pseudo-labeling (P-L) technique. First, we use the trained object detector to generate predictions on the training images of PASCAL VOC2007. Pseudo-labels are selected by setting the confidence and IoU thresholds to 0.2 and 0.5, respectively in the NMS algorithm. Then, we fine-tune our trained object detector on these pseudo-labels. We report an improvement of 1.6 mAP 50 and 1.1 mAP 50:95 . Despite the global mAP 50 being inferior to that of Cap2Det CLSF , it is worth noting that our approach implementing P-L outperforms Cap2Det CLSF in many categories.

Ablation experiments: We also perform ablation experiments to identify the key components of our approach in WSOD. To annotate objects, we employ the variants of our approach presented

Conclusion

In this paper, we present a two-stage method to locate and label objects by leveraging image-caption pairs without additional supervision. We demonstrate the effectiveness of our approach by performing two tasks in a weakly supervised setting: phrase grounding and object detection. We have performed extensive experiments on Flickr30k Entities, MS COCO and PASCAL VOC2007 achieving stateof-the-art results without the need for supervised object proposal algorithms or text classifiers to process captions. Despite the remarkable performance of our approach, we acknowledge some limitations. Our approach produces a single bounding box per object mentioned in the caption. An interesting direction for further investigation is to extend our method to produce multiple bounding boxes for words representing more than one object instance in the image (e.g. "people", "group of animals", etc.). This is particularly challenging, especially when object instances are overlapping in the image. Also, our approach does not generate bounding boxes for objects present in the image but not mentioned in the caption (or due to spelling mistakes). We believe that an important direction for future work is to extend our approach to explicitly take into account missing annotations. Improved performance could be achieved using a more sophisticated pseudo-labeling framework [START_REF] Xu | Missing labels in object detection[END_REF][START_REF] Wang | Co-mining: Self-supervised learning for sparsely annotated object detection[END_REF][START_REF] Li | Siod: single instance annotated per category per image for object detection[END_REF].

∂ac • a c Figure 2 :

 c2 Figure2: Bounding box generation using the caption "A woman is steering a boat with a pole". First, we select the initial and potential seeds (red and gray patches, respectively) via a VL model for each category identified in the caption. Second, we perform seed expansion by measuring similarity between patches via a ViT. Finally, each heatmap is thresholded and a bounding box is drawn on top.

Figure 3 :

 3 Figure 3: Heatmaps and generated bounding boxes corresponding to 'man', 'ball' and 'racquet'. ALBEF C-A maps point successfully at objects while struggle to get the object extent. ALBEF ViT maps tend to be noisier than DINO ViT maps which generate high-quality bounding boxes.

  PASCAL VOC2007 is an object detection dataset that contains 2501 training, 2510 validation, and 4952 test images. Objects are labeled into 20 classes (e.g. person, bird, cat, cow, dog, etc.). Results are reported in terms of mean average precision at IoU=0.5, i.e. mAP 50 , and average mAP over multiple IoU values ranging from 0.5 to 0.95 with a step of 0.05, i.e. mAP 50:95 . Results are reported on the MS COCO validation set and the PASCAL VOC2007 test set. In all cases, bounding box annotations are only used during evaluation.

Table 1 :

 1 Weakly supervised phrase grounding performance on Flickr30k Entities.

	Method	Training data	Supervised object proposal generator?	Recall@1
	ALBEF C-A maps 14M image-caption pairs [19]	No	36.86
	ALBEF ViT maps 14M image-caption pairs [19]	No	43.97
	DINO ViT maps	14M image-caption pairs [19] + ImageNet images	No	47.51

Table 2 .

 2 Our approach achieves 21.1 mAP 50 and 10.5 mAP 50:95 on MS COCO outperforming the variants of Cap2Det[START_REF] Ye | Cap2det: Learning to amplify weak caption supervision for object detection[END_REF] that learn from image-caption pairs: Cap2Det EM that generates image-level annotations from captions via lexical matching and Cap2Det CLSF that employs a supervised text classifier to process captions and extract image-level

	'man'	'ball'	'racquet'	Bounding boxes
	ALBEF C-A maps
	ALBEF ViT maps
	DINO ViT maps
	(Ours)

Table 2 :

 2 Comparison with WSOD models on MS COCO. ALBEF ViT maps and DINO ViT maps. Tables4 and 5display the results of our experiments on MS COCO and PASCAL VOC2007, respectively. As observed, ALBEF C-A maps perform poorly at object detection achieving the lowest scores mAP 50 and mAP 50:95 . While ALBEF C-A maps are able to accurately point at objects, they fail to correctly detect their extent. On the other hand, self-supervised ViTs (ALBEF ViT maps and DINO ViT maps) are effective to capture the extent of objects through the seed expansion. In MS COCO, DINO ViT maps outperform ALBEF ViT maps as expected since DINO ViT maps are less noisy and generates visually more accurate bounding boxes as shown in Figure3. Surprisingly, ALBEF ViT maps achieve slightly better results than DINO ViT maps in PASCAL VOC2007.

	Model	Supervision source	mAP 50 mAP 50:95
	Cap2Det EM [48]	image-captions pairs	19.7	8.9
	Cap2Det CLSF [48]	image-captions pairs	20.2	9.1
	Ours	image-captions pairs	21.1	10.5
	PCL [40]	image-level annotations	19.4	8.5
	C-MIDN [10]	image-level annotations	21.4	9.6
	WSOD 2 [53]	image-level annotations	22.7	10.8
	MIST [32]	image-level annotations	25.8	12.4
	CASD [13]	image-level annotations	26.4	12.8
	Fully supervised [15] bounding box annotations 66.2	46.7
	in Section 4.2, i.e. ALBEF C-A maps,

Table 3 :

 3 Comparison with WSOD models on PASCAL VOC2007. .3 50.7 25.9 14.1 64.5 50.8 33.4 17.2 49.0 48.2 46.7 44.2 59.2 10.4 14.3 49.8 37.7 21.5 47.6 39.9 Cap2Det CLSF [48] 63.8 42.6 50.4 29.9 12.1 61.2 46.1 41.6 16.6 61.2 48.3 55.1 51.5 59.7 16.9 15.2 50.5 53.2 38.2 48.2 43.1 Ours 58.8 64.6 52.3 28.9 10.0 57.2 42.2 50.7 12.8 54.3 32.4 38.8 37.4 61.9 24.2 17.6 47.3 39.0 52.3 34.4 40.9 Ours + P-L 56.1 68.5 55.6 31.1 12.3 64.8 48.6 48.8 15.5 57.8 22.9 34.8 42.3 59.1 23.2 19.1 51.8 42.8 54.8 41.0 42.5 Supervised [15] 70.2 74.3 42.8 40.4 40.8 73.6 83.3 62.0 37.7 61.3 58.3 56.1 77.5 71.2 78.0 35.5 50.5 55.0 75.1 60.2 60.2

	Model	aero	bike	bird	boat	bottle	bus	car	cat	chair	cow	table	dog	horse	mbike	person	plant	sheep	sofa	train	tv	mAP50
	Cap2Det EM [48]	63.0 50

Table 4 :

 4 Ablation experiments on MS COCO.

	Method	mAP 50 mAP 50:95
	ALBEF C-A maps 9.4	3.7
	ALBEF ViT maps 18.4	9.0
	DINO ViT maps	21.1	10.5

Table 5 :

 5 Ablation experiments on VOC2007.

	Method	mAP 50 mAP 50:95
	ALBEF C-A maps 9.2	3.3
	ALBEF ViT maps 42.9	20.8
	DINO ViT maps	40.9	18.0

Acknowledgments

This work was conducted as part of the MINDS project of IRT Saint Exupéry. We would like to thank Michelle Aubrun, Ahmad Berjaoui, David Bertoin and Franck Mamalet for useful feedback and suggestions and Jérôme Mathieu for invaluable technical support.


Zhiwen Chen

Zhigang Lv

Ruohai Di

Peng Wang

Xiaoyan Li

Xiaojing Sun

Yuntao Xu

A novel virtual sample generation method to improve the quality of data and the accuracy of data-driven models

Keywords: small sample, data-driven model, virtual sample generation (VSG), acceptable area, joint probability distribution sampling

Small data volume and data imbalance often lead to statistical failure and seriously restrict the accuracy of data-driven models, which has become a bottleneck problem, needing to be solved, in small sample modeling. The data expansion method has become the main way to solve small sample modeling. However, the randomness in the process of virtual sample generation and combination leads to many invalid data, resulting in poor consistency between the expanded data and the original data. For this reason, this paper proposes a virtual sample generation method based on acceptable area and joint probability distribution sampling (APS-VSG) to limit the randomness in the data expansion method, reduce the proportion of invalid data, improve data consistency after expansion, and improve the accuracy of the data-driven model under the condition of small samples. Firstly, the concept of "compact range of interaction (CRI)" was proposed, which further limits the domain estimation range of data to approximate the valid area of the data. Secondly, the prior knowledge was used to improve mega-trend-diffusion (MTD), and the CRI is delineated according to the trend dispersion to obtain the acceptable area of the virtual data. Finally, a joint probability distribution was constructed based on the true values of small samples in the acceptable area, and data sampling was conducted based on the probability distribution to generate virtual data. The experimental results of standard function datasets show that the virtual samples generated by the proposed method can ensure validity of more than 85%.

The experimental results of the NASA li-ion battery dataset show that, compared with Interpolation, Noise, MD-MTD, GAN, and GMM-VSG methods, the error of the data-driven model trained with virtual data generated by the proposed method is significantly reduced.

Compared with GAN and GMM-VSG, MSE, RMSE, MAE, and MAPE are reduced by at least 19.3%, 10.6%, 15.4%, and 16.7%, respectively.

Introduction

Machine learning is the focus in the field of artificial intelligence and pattern recognition.

Various novel machine-learning algorithms emerge in endlessly. In the age of big data, data-driven machine learning algorithms have gradually become research focus, which are widely used to solve complex problems in scientific fields and engineering applications. However, there are still many fields, which cannot obtain a large amount of data due to factors, such as high experiment cost and long test cycle. Therefore, it is difficult to apply advanced deep learning algorithm to solve problems [START_REF] Lu Yiyong | Researches on Few-shot Learning Based on Deep Learning:an Overview[END_REF], such as voice print recognition [START_REF] Nasef | Voice gender recognition under unconstrained environments using self-attention[END_REF] in multimedia field, disease diagnosis [START_REF] Qezelbash-Chamak | A survey of machine learning in kidney disease diagnosis[END_REF] [START_REF] Mirzaei | Machine learning techniques for diagnosis of alzheimer disease, mild cognitive disorder, and other types of dementia[END_REF] and water analysis [START_REF] Wang | Moisture quantitative analysis with a small sample set of maize grain in filling stage based on near infrared spectroscopy[END_REF] in biological and medical field, product sales prediction [START_REF] Li | Lifecycle Forecast for Technology Products with Limited Sales Data[END_REF] in the economic field and life prediction of fuel cells [START_REF] Wang | Data-driven prognostics based on time-frequency analysis and symbolic recurrent neural network for fuel cells under dynamic load[END_REF] in the industrial and military fields. In the process of using machine learning to deal with the above problems, there are problems such as small data volume and data imbalance, etc. All the above belong to small sample problems, because the sample size of the target object is too small to train a model that meets the accuracy requirements [START_REF] Zhao Kai-Lin | Survey on Few-shot Learning[END_REF][START_REF] Xulong | Virtual sample generation method and its application in reforming data modeling[END_REF].

For the small sample problem, scholars have carried out data-centric research. Data expansion is the main method used in data-centric research. It is the most direct method to deal with small sample problems by constructing virtual samples to increase sample size and balance data sets. The main ideas for constructing virtual samples are knowledge-based, disturbance-based, and distribution-based [START_REF] Xu | Research on Virtual Sample Generation Technology[END_REF].

The knowledge-based idea is mainly to artificially generate virtual samples based on expert knowledge in the research field [START_REF] Jing | A novel virtual sample generation method based on Gaussian distribution[END_REF]. Xu [START_REF] Xu Rong-Wu | RESEARCH ON VIRUTAL SAMPLE BASED IDENTIFICATION OF NOISE SOURCES IN RIBBED CYLINDRICAL DOUBLE-SHELLS[END_REF] found, in the engineering problems of the double-cylindrical shell structure, that the signals of the double-cylindrical shell structure collected by sensors are often mixed with the noise from the exciter and the seawater pump.

According to the frequency characteristics of these noises, he proposed a method, based on the frequency response function and Fourier transform, to highly simulate the sampling signals of the double-cylindrical shell structure with the noise signals of the exciter and the seawater pump.

These simulated virtual signal used to train the model can effectively improve the noise recognition rate of the model for the double cylindrical shell structure. The threshold for creating virtual samples based on knowledge is very high, which requires rich expert knowledge and experience, and is difficult for nonprofessional personnel to use. In various methods of constructing virtual samples based on the idea of disturbance, Chris M. Bishop [START_REF] Bishop | Training with Noise is Equivalent to Tikhonov Regularization[END_REF] found that a model with higher generalization performance was obtained by adding a certain amount of noise to the input data and inputting it into the neural network for training. Guozhong An [START_REF]The Effects of Adding Noise During Backpropagation Training on a Generalization Performance[END_REF] further confirmed that adding noise to input samples can effectively improve the generalization performance of classification and regression problems. Wang [START_REF] Weidong | Quadratic Discriminant Analysis Using Virtual Training Samples [J][END_REF] added disturbance based on training samples to obtain new virtual samples and used these virtual samples to make the model have a better recognition rate. The idea based on distribution is mainly to determine the range and probability of virtual sample generation according to the domain distribution of small sample data [START_REF] Li | Using mega-trend-diffusion and artificial samples in small data set learning for early flexible manufacturing system scheduling knowledge[END_REF][START_REF] Huang | A diffusion-neural-network for learning from small samples[END_REF][START_REF] Li | data-fuzzification technology in small data set learning to improve FMS scheduling accuracy[END_REF][START_REF] Li | Using mega-fuzzification and data trend estimation in small data set learning for early FMS scheduling knowledge[END_REF][START_REF] Zhu Bao | A novel mega-trend-diffusion for small sample[END_REF][START_REF] Zhu | Research on Virtual Sample Generation Technologies and Their Modeling Application[END_REF][START_REF] Junfei | Virtual Sample Generation Method Based on improved General Trend Diffusion and Hidden layer interpolation and its application[END_REF][START_REF] Li | A Gaussian mixture model based virtual sample generation approach for small datasets in industrial processes[END_REF][START_REF] Li | A genetic algorithm-based virtual sample generation technique to improve small data set learning[END_REF][START_REF] Zhu | Novel SVD integrated with GBDT based Virtual Sample Generation and Its Application in Soft Sensor[END_REF][START_REF] He | Enhanced virtual sample generation based on manifold features: Applications to developing soft sensor using small data[END_REF]. Der Chiang Li [START_REF] Li | Using mega-trend-diffusion and artificial samples in small data set learning for early flexible manufacturing system scheduling knowledge[END_REF] proposed mega-trend-diffusion (MTD), which uses a common diffusion function to spread a group of data and determine the possible coverage of the data set based on group consideration to generate reasonable virtual data. Chongfu Huang [START_REF] Huang | A diffusion-neural-network for learning from small samples[END_REF] and Der Chiang Li [START_REF] Li | data-fuzzification technology in small data set learning to improve FMS scheduling accuracy[END_REF] [19] used information diffusion, MTD, and other methods to turn a clear value sample point into a fuzzy set, turning a small number of single value sample points into a large number of set-valued sample points. This method can make the generated virtual samples have more information. Zhu Bao [START_REF] Zhu Bao | A novel mega-trend-diffusion for small sample[END_REF] [START_REF] Zhu | Research on Virtual Sample Generation Technologies and Their Modeling Application[END_REF] proposed multi-distribution mega-trend-diffusion (MD-MTD) to generate virtual samples, in which uniform distribution and triangular distribution are added to describe the characteristics of small sample data. Compared with MTD, the virtual samples generated by MD-MTD are more realistic. Ling Li [START_REF] Li | A Gaussian mixture model based virtual sample generation approach for small datasets in industrial processes[END_REF] proposed a Gaussian mixture model-based virtual sample generation (GMM-VSG) method to generate virtual samples under multiple working conditions. Der-Chiang Li [START_REF] Li | A genetic algorithm-based virtual sample generation technique to improve small data set learning[END_REF] proposed a genetic algorithm-based virtual sample generation (GABVSG), which generated more valid virtual samples by considering the overall integrated effects of the attributes. Qun Xiong Zhu [START_REF] Zhu | Novel SVD integrated with GBDT based Virtual Sample Generation and Its Application in Soft Sensor[END_REF] proposed a VSG method based on singular value decomposition (SVD) feature decomposition and gradient boosting decision tree (GBDT) prediction model. Yan Lin He [START_REF] He | Enhanced virtual sample generation based on manifold features: Applications to developing soft sensor using small data[END_REF] generated virtual samples by using the proposed t-SNE-based virtual sample generation (t-SNE-VSG). SVD-VSG and t-SNE-VSG both use the distribution of the original data as a reference to generate virtual samples, which is difficult to use for small sample data with unknown distribution. In addition, Der Chiang Li [START_REF] Li | A non-linearly virtual sample generation technique using group discovery and parametric equations of hypersphere[END_REF] proposed a nonlinear virtual sample generation technology based on a hypersphere parameter equation in combination with group discovery technology, which can effectively improve the learning accuracy of the model. He also used the interval kernel density estimator to generate more similar virtual samples, overcoming the problem of learning difficulties when data is insufficient [START_REF] Li | Using virtual sample generation to build up management knowledge in the early manufacturing stages[J][END_REF]. Chen Zhongsheng [START_REF] Chen Zhongsheng | Quantile regression CGAN based virtual samples generation and its applications to process modeling[END_REF] proposed a new virtual sample generation method QRCGAN, which embedded quantile regression into a conditional generation countermeasure network. Yan LinHe [START_REF] He | A novel virtual sample generation method based on a modified conditional Wasserstein GAN to address the small sample size problem in soft sensing[J][END_REF] proposed a novel virtual sample generation method embedding a deep neural network as a regression into conditional Wasserstein generative adversarial networks with gradient penalty (rCWGAN).

Embedding networks can improve the validity of virtual samples to some extent, but it takes a long time and is expensive to calculate.

Comparing virtual data generated by different methods, knowledge-based virtual samples are limited by the accuracy and diversity of expert knowledge, obtained in the research field. Manual sample production is expensive and time-consuming, which is difficult to apply to most scenarios.

Disturbance-based virtual samples mainly expand the boundary area of small samples by adding disturbances to improve the generalization of the model. However, there is no unified answer to how much disturbance is appropriate for different problems, which needs to be determined through a large number of experiments. For this reason, we proposed an improved acceptable area estimation method in this paper. This method delimited a more compact range of interaction (CRI) of multi variable within the theoretical upper limit and lower limit of each variable which is called the wide range of permissible (WRP) to ensure that the generated multidimensional data is acceptable in this area. This method can avoid the problem that the generated virtual samples deviate from the real samples too much. The basis for creating virtual samples based on distribution is statistics, but there is still doubt whether the conclusions obtained by using statistics for small samples that do not meet the large sample theorem can express the properties of large samples. For this reason, this paper reduced statistical methods for estimating data distribution through small samples and artificially constructed a probability distribution function with universality to describe the distribution of data according to small samples in the acceptable area, from which the problem that the sample size is too small to accurately estimate the sample distribution was solved.

A large number of scholars have found that in the process of generating multidimensional virtual data, the method of first generating single-dimensional virtual data and then reconstructing high-dimensional data often makes combination errors, resulting in a large number of invalid virtual samples [START_REF] Yao-San Lin | A Virtual Sample Screening Mechanism[END_REF]. This part of invalid virtual samples will greatly reduce the accuracy of the model and it is difficult to build an appropriate screening mechanism to screen out invalid data.

For this reason, this paper analyzes the probability distribution between multiple variables and target variables and then determines the joint distribution between multiple variables by target variables to solve the problem of data combination. The long-standing problem with virtual samples is that it is difficult to consider both validity and robustness. In this paper, edge virtual samples were added during sampling to explore the edge information inconsistent with small samples to improve the robustness of virtual samples in the data-driven model.

The main contributions of this paper are shown as the following:

(1) A virtual sample generation method, based on CRI and joint probability distribution sampling, was proposed in this paper. This method can expand the data for small samples with unknown data characteristics, limit the randomness in the process of virtual sample generation, and greatly improve the validity of virtual samples;

(2) An estimation method of acceptable area was proposed, which uses prior knowledge to improve MTD and estimates CRI according to trend dispersion to comprehensively estimate the generation space of virtual samples, to provide a guarantee for generating high-quality virtual samples;

(3) The conditional distribution of each variable under specific conditions was constructed to ensure a high sampling probability near the real sample and increase the overall sampling probability in the acceptable area to include part of the edge virtual samples to improve the robustness;

(4) Joint probability sampling was used to generate high-dimensional data as a whole to avoid the problem of data combination.

The rest organization of this paper is shown: In section 2, the basic theory of this paper is introduced. In section 3, the proposed method is described in detail. In section 4, the experimental verification is finished. In section 5, the conclusions are drawn.

Basic theory

In this section, the idea source and basic theory of the method in this paper is briefly introduced, including MTD, confidence interval, and sampling methods.

Mega-trend-diffusion

MTD [START_REF] Li | Using mega-trend-diffusion and artificial samples in small data set learning for early flexible manufacturing system scheduling knowledge[END_REF] was proposed by Der Chiang Li to determine the data coverage, which uses a common diffusion function to spread a group of data and determine the possible coverage of the data set based on group consideration. Zhu added uniform distribution and triangular distribution to describe the characteristics of small samples. The new method was called MD-MTD. The core idea of MTD and MD-MTD is to determine the floating range of data according to the number and distribution density of data. Fig. 1 shows that the extrapolation boundary of virtual samples was estimated based on small samples under the triangular diffusion function.

n i L L U L xx n L CL n n n n           (1) 2 20 1 () 2 ln(10 ) ( 1) n Ui L U U xx n U CL n n n n           (2) min max 2 CL   (3)
where, min and max are the minimum and maximum values in a small sample set, It is meaningful to use the MTD to estimate the acceptable area of data, and also effective to generate one-dimensional virtual data according to the extension field. However, it will cause a combination error if aligning multi-dimensional data after generating each one-dimensional virtual data. Especially, when there is a large distribution difference between each one-dimensional data, most of the virtual data generated is even invalid. For this reason, this paper optimizes MTD based on prior knowledge, proposes CRI to limit the excessive acceptable area, and uses joint probability sampling to avoid combination error.

Confidence interval

Confidence interval is a common interval estimation method. It is an estimation interval of population parameters constructed by sample statistics, with upper and lower confidence limits of statistics as upper and lower bounds respectively. It refers to the range where the true value appears with the measured value as the center under a certain confidence, which is the probability that the true value will occur within a certain range. The real data are often not known, and we can only estimate. For example, in[ , ] ab, the probability that the real value appears between a and b is 95% (there is also a 5% probability that it appears outside this interval). The smaller the interval is, the lower the confidence is, and the higher the accuracy is. The 95% confidence level is general. [32] [33] In regression analysis, confidence intervals are usually used to judge the reliability of data and increase the robustness of the analysis. This paper uses this idea for reference and sets a certain confidence level to expand the range of small samples according to the data distribution statistical chart of small samples, which is regarded as one of the criteria for defining the acceptable area.

Sampling

Stratified sampling

Hierarchical sampling is a method of randomly selecting samples from different layers according to the specified proportion of a sample population that can be divided into multiple layers. The advantage of this method is that the sample is representative and the sampling error is small. For data X , if n samples were obtained by stratified sampling, it is divided into k layers according to certain rules, then n k samples are sampled from each layer, and finally, all samples are integrated.

Acceptance-Rejection Sampling

Acceptance () px.

In the case of high-dimensional, there are two problems in ARS. First, it is difficult to obtain the general form of multidimensional distribution when there is only conditional distribution

1 ( | ) p x y , 2 ( | ) p x y , … , ( | ) n p x y .
The second is that it is difficult to find a suitable distribution () qx .

Proposed method

In this section, the methods proposed in this paper will be described in detail. The object of this algorithm is () mn X  and ( 1)

n Y  . When the target object is (( 1) )
mn Z  , try to divide a vector with a strong correlation with other vectors by correlation analysis, and take this vector as [START_REF] Lu Yiyong | Researches on Few-shot Learning Based on Deep Learning:an Overview[END_REF] n Y  , the remaining m-dimensional vector is taken as () mn X  . In this paper, we first use prior knowledge and trend dispersion to delineate a more reasonable acceptable area where generating virtual samples can greatly improve the validity of virtual samples. Then, two-stage sampling is conducted on the joint distribution constructed in this paper to obtain virtual samples, to ensure that a large amount of valid virtual samples is generated, and a certain number of edge virtual samples can also be generated to improve the robustness. 1 , 1 , , 1 1 2 , 2 , ,

, 1 , , 1 1 2 , 2 , , 2 2 ( 1) , ( 1) , , ( 1) ( 1) 1
n n n n n Z Z Z Z Z Z Z Z Z Z m Z m Z m Zm X X X X X X X partition X X Xm Xm Xm Xm Y Y Y Y                                          (4) 2 2 , , , [ , , , ] n n

Algorithm framework

Different from MTD, the proposed method estimates not only the domain range of small samples but also the compact range of interaction (CRI) between multiple variables to limit the acceptable area of data. To avoid the problem of data combination and generate high-dimensional data as a whole, the proposed method constructs the conditional distribution of each variable and multidimensional probability distribution according to the true value of small sample, which provides technical support for the generation of virtual data. The algorithm framework is shown in [ , ]

X X Q Buttom Q Top   2 2 [ , ] X X Q Buttom Q Top   [ , ] Y Y Q Buttom Q Top   X p Y XpY Q Top  XpY Q Buttom  Q  Compact range of interaction 1 1 [ , ] X Y X Y Q Buttom Q Top   1 X Y  : 2 2 [ , ] X Y X Y Q Buttom Q Top   2 X Y  : : 95% 99% Acceptable area Q Q Q    
, 1 , , 1 1 2 , 2 , , 2 2 ( 1) , ( 1) , , ( 1) ( 1) 1
, , [ ,,, ]

n n n n n n n Z Z Z Z Z Z Z Z Z Z m Z m Z m Z m X X X X X X X X X Xm Xm Xm Xm Y Y Y Y                                                                    [ , ] Xm Xm Q Buttom Q Top   Xm [ , ] XmY XmY Q Buttom Q Top   Xm Y  2 ( | ) P Xp y k  X p 1 ( | ) P Xp y k  X p X p 2 i X p 3 i X p ( | ) P Xp y l  1 i X p X p Y X p Q Buttom  Xp Q Top  Y Q Buttom  Y Q Top  XpY Q Top  XpY Q Buttom  2 y k  1 y k  y l  Xp Q Top  X p 2 ( | ) P Xp y k  3 i X p 1 2 2 ( ) ( ) ( , ) XpY y k XpY y k U Q Buttom Q Top   3 2 ( , ) i N Xp  2 ( ) XpY y k Q Buttom  2 ( ) XpY y k Q Top  1 1 1 1 min( 1 , 1 ) 3 n X Y i i X Y i Q Top x x Q Buttom n       2 ( 1, 2 | ) P X X y k  1 ( 1, 2 | ) P X X y k  1 X 2 X 1 X 2 X

Acceptable area construction based on a priori knowledge and trend dispersion

All virtual sample generation methods need to estimate the distribution range of data before generating virtual data. In this paper, this range is referred to as the acceptable area. This paper proposed an estimation method of acceptable area. Firstly, the upper and lower limits of each variable are determined according to prior knowledge, and a large acceptable area of data is determined according to the upper and lower limits of each variable. This large acceptable area of data is referred to as the wide range of permissible (WRP). Then, determine the trend of each X and Y , and fit the distribution curve according to the distribution histogram of small samples in the data trend direction. Take the 99% or 100% confidence interval of the distribution curve as CRI. Finally, final the acceptable area will be obtained by using CRI to limit this large acceptable area. Therefore, an acceptable area Q is the intersection of WRP Q  and CRI Q  .

Q Q Q    (5)

Estimation method of WRP

In practical application, the measured values collected have practical meanings. Generally, these measured values have critical values, which are objective limitations and are usually determined by physical factors such as material and shape. Especially for some products produced by assembly lines, each parameter has a rated working range and a maximum fluctuation range. In this paper, the interval bounded by the theoretical upper limit and lower limit is called the wide range of permissible (WRP), which is the large acceptable area mentioned above.

Fig. 3. The estimation method for WRP

As shown in Fig. 3, is a diagram of the estimation method for WRP, where,

1 () 2 1 22 1 () 2 n nn Z if n is odd CL Z Z if n is even          (7) [ ] 1 A CL  (8)
For the case that only the theoretical lower limit

[ ] max( ) [max( )] [ ] [max( )] [ ] max( ) (1 [min( )]) [min( )] min( ) [ ] max( ) (1 [ ] ) min( ) [] [max( ) min( )] max( A CL Zp A Zp CL Zp A CL A Zp A CL Zp A Zp CL A Zp Zp Zp A CL Zp A CL CL CL Zp Zp Zp A CL CL Zp CL Zp Zp CL Zp                            min min ) min( ) Zp Zp Zp   (12)
Similarly, when only the theoretical upper limit max Zp of p Z is known but the theoretical lower limit min Zp is not known. [max( ) min( )] min( ) max( )

A CL Zp A Zp CL Zp A CL A Zp CL Zp Zp CL Zp Zp Zp Zp             (13)

Estimation method of CRI

It is useful that MTD generates virtual samples

' N X  on the estimated acceptable area of ( 1) n X  and uses the hyperplane to give the corresponding [START_REF] Lu Yiyong | Researches on Few-shot Learning Based on Deep Learning:an Overview[END_REF] '

N Y  . When the dimension of X increases, many of () mn X 
obtained by data combinations after the expansion of

(1 ) (1 ) (1 )
1 , 2 , ,

n n n X X Xm   
will deviate from the real samples. The reason is that there may be some correlation factor between

(1 ) (1 ) (1 )
1 , 2 , ,

n n n X X Xm  
 , which makes 2 n X  . This change in coupling can greatly narrow the acceptable area of multivariate data. Therefore, this paper further reduces the acceptable area by estimating the area of such influence. This paper will refer to this area as a compact range of interaction (CRI).

As is known to all, in the feature selection of the neural network model, input features with a strong correlation with output features will be selected. Based on this, this paper analyzes the relationship between each input feature

1 , 2 , ,

n n n X X Xm   
range corresponding to (1 ) n Y  , to limit the acceptable area of the input feature and prevent the phenomenon of combination error.

Firstly,

1 , 2 , ,

n n n X X Xm   
is extracted from X in order, and takes vector

min( )

XpY V   of ( ) (2 1) XpY XpY XpY V    , then (2 2) XpY O  is: (2 2) (2 1) (2 1) (1 2) (2 1) (2 1) [max( ), min( )] * T XpY XpY XpY XpY O XpY V V V             ( 16
)
where  is 1.05.

In the direction of (2 2) XpY O  , the distribution curve is fitted according to the distribution histogram of small samples, and the 99% or 100% confidence interval of the distribution curve is taken as the CRI [ , ]

XpY XpY XpY Q Q Buttom Q Top     . Small samples Tendency Statistical Area Statistical Chart Xp Y XpY Q Top  XpY Q Buttom  95% 99%
Fig. 4. The estimation method for CRI

Joint probability distribution sampling

Tested by the author, it will cause a combination error if aligning multi-dimensional data after generating each one-dimensional virtual data. Especially when there is a large distribution difference between each one-dimensional data, most of the virtual data generated is even invalid.

To solve this problem, this paper constructed the artificial multi-variable joint distribution based on the acceptable area, to simultaneously generate 1, 2, , X X Xm as a whole to avoid combination error.

Artificial joint probability distribution

In small-sample modeling, virtual samples combined with small sample sets are used for model training. The basic reason why the accuracy is not improved or decreased is the low quality of the data. To improve the data quality, it is necessary to start from the data validity and generate virtual samples conforming to the characteristics of small samples. Secondly, to improve the fault tolerance ability of the model, it is necessary to generate a small amount of differentiated virtual samples to improve the robustness.

To make the generated virtual data conform to the characteristics of small samples, the conditional probability distribution ( | obtained by integration. Fig. 5 shows the diagram of the process of constructing a joint probability distribution.

) P Xp y k  of * ( [ 1
1 min( 1 , 1 ) 3 n X Y i i X Y i Q Top x x Q Buttom n       1 , , X X m 1 ,
Assume that sample

i is [ 1 , 2 , , ] i i i i X x x xm  , [] i Yy  , ij yk  ,
1 min(| 1 |,| 1 |) 3 n X Y i i X Y i Q Top x x Q Buttom n       (18)
The normal distribution constructed according to sample truth value ( 1 , ) ii xy and acceptable area boundary

j yk  , 22 2 1 1 1 ( 1 , ) ( 1 , ) ( , ) i i X Y X Y X N x N x U Q Buttom Q Top     (19)
From this, the conditional probability distribution ( 1| ) . The more number l is calculated, the denser the distribution, and the smoother the transition.

Then, the joint probability distribution ( | )

P X y is constructed. ( | ( ))= ( 1 2 | ( )) ( 1| ( )) ( 2 | ( )) ( | ( )) P X y k l P X X Xm y k l P X y k l P X y k l P Xm y k l         (20)
Finally, the joint probability distribution ( | ) P X y of all values in [ , ] YY Q Buttom Q Top is integrated to get the final joint probability distribution ( | ) P X Y .

Two-stage sampling

Since all the methods used above construct the joint probability distribution of 1, 2, , X X Xm on the premise of determining the value of y . Therefore, it is necessary to take The process of ARS is equivalent to using the CRI and distribution of real samples as a filter to remove invalid samples from the virtual samples obtained by the WRP.

N samples from [ , ] YY Q Buttom Q Top to get (1 ) ' N Y  ,

Algorithm description

In summary, the structure and details of the algorithm in this paper are expanded, and then the pseudo-code of the algorithm is given.

Algorithm: APS-VSG

Input: Small samples ( ) (1 )
,

m
( | ) ( 1| ) ( 2 | ) ( | ) P X Y P X Y P X Y P Xm Y     13: Get 1 2 (1 )
'

= ' , ' , , ' N N N mN x x x x x x X xm xm xm        18: Get a virtual sample set ( ) ( ) { ' , ' } m N m N D X Y   12 12 () 1' , 1' , , 1' 2' , 2' , , 2' '

Experiments

The method presented in this paper was tested on three standard functions and NASA lithium battery data sets to analyze the validity of the acceptable area and the virtual samples. Compared with five other methods, including Interpolation, Noise, MTD, GAN and GMM-VSG, the advantages and disadvantages of the proposed method are discussed in the experimental analysis.

Datasets

Standard function datasets

The

y x x x     [1,5] [1,10] [1,20] [-2,160] Oscillation 1 2 3 1 2 1 sin( ) sin( ) 3 3 3 y x x x        [1.7,4.3] [1.7,9.3] [1.7,19.3] [0,7]

NASA li-ion battery dataset

The ultimate purpose of generating virtual samples is to use the right amount of data to train a data-driven model that meets the target performance. Therefore, to verify the validity of the method proposed in this paper, from the performance degradation data of NASA li-ion battery [START_REF] Chen | Combining empirical mode decomposition and deep recurrent neural networks for predictive maintenance of lithium-ion battery[END_REF],

data of two batteries of the same model were selected to build a real sample set with five inputs and one output, and take 34 samples from the real sample set as the small sample set. 100%

va ca a V V   (24)
100%

va cv v V V   (25)
100%

va v a v va V V V V    (26)
Where, va V is the size of the valid area within the acceptable area, a V is the size of the acceptable area, v V is the size of the valid area.

Direct evaluation of virtual sample

It is judged whether the generated virtual samples are in the valid area, if yes, they are considered as valid virtual samples, if not, they are considered as invalid virtual samples. Count the number of valid virtual samples and the number of invalid virtual samples. The ratio of the number of valid virtual samples to the total number of virtual samples, v  , is used to evaluate the validity of the virtual samples. The expression of v  is as follows:

v v vi n nn    (27)
Where, v n is the number of valid virtual samples, i n is the number of invalid virtual samples.

Indirect evaluation of virtual sample

In

Results and Analysis

Experiment on the standard function datasets

The method proposed in this paper is used to estimate the acceptable area and generate 200 virtual samples through the small sample set of standard functions to evaluate the validity of the acceptable area and the validity of the virtual samples. The experimental results are shown in Fig. 6, 7 and 8. Linear is the classical linear model, which can be perfectly described by linearity. However, the data trends determined based on the small sample set differ slightly from the real sample trends. There is an overall tilt in the acceptable area, which leads to the loss of part of the valid area and the inclusion of part of the invalid area. This leads to the subsequent generation of partially invalid data. The virtual samples generated by APS-VSG mainly surround small samples, which avoids invalid virtual data from being generated in large quantities in sparse areas.

Nonlinear is the classical nonlinear model, which is difficult to describe by linearity. Therefore, the acceptable area is perceptible as too large. Although the acceptable area includes nearly all of the valid areas, most of them are invalid. In particular, the invalid area is as much as 6 times more than the valid area within the acceptable area of 2 X and Y , which greatly increases the difficulty of generating valid virtual samples. However, most of the virtual data generated by APS-VSG surrounds small samples, which greatly improves the validity of virtual samples, because of its sampling based on the distribution of small samples. Oscillation is a model with sinusoidal oscillations and is one of the most complex models. And the general VSG method tends to have huge problems with this type of model. Due to the presence of sinusoidal oscillations, the acceptable area inevitably includes invalid area of peaks and valleys, which results in the invalid area that accounts for almost half of the acceptable area. However, the acceptable area obtained by APS-VSG includes almost all the valid area. APS-VSG generates a large number of valid virtual samples around sinusoidal oscillations by its property of sampling based on small samples.

Although the superimposed uniform distribution-based sampling leads to a small amount of virtual samples out of the valid area.

From the final evaluation results of the experiment, metric ca  is not satisfactory. Using ca  as a criterion to evaluate the acceptable area, the best is Linear, followed by Oscillation, and finally Nonlinear. However, it cannot be ignored that cv  of the acceptable area are high enough to include as many valid areas as possible, which provides strong support for the subsequent sampling. Therefore, it is not feasible to blindly pursue ca  while ignoring cv  . Using cv  as a criterion to evaluate the acceptable area, the best is Oscillation, followed by Linear, and finally Nonlinear. The results of the follow-up the direct evaluation of virtual samples also illustrate exactly this.

In summary, the results of virtual data evaluation on three standard function datasets show that APS-VSG can guarantee the high validity of the virtual samples. This proves that the method proposed in this paper can improve the quality of data.

Experiment on NASA Li-ion battery dataset

Generate 50 virtual samples based on the constructed NASA li-ion battery small sample set, and train BP neural networks by small sample sets, real sample sets and virtual samples generated by the six methods to verify whether the virtual samples generated by various methods can effectively improve the accuracy of the model. A 3-layer BP neural network model with the structure of 5-10-1, 1000 iterations, a learning rate of 10 -3 , a momentum factor of 0.9, and a target error of 4 × 10 -3 is constructed. The training data is the set of small samples and virtual samples, and the test data is Real samples. The average prediction results and errors of multiple runs are shown in Fig. 9. The performance criteria of the model are shown in Table 5. trained with these virtual samples can make accurate predictions for both batteries. Benefiting from both the limits of the acceptable area and the probability distribution based on small samples, the resulting virtual samples are both highly valid and robust.

.96×10 -4 1.40×10 -2 0.91×10 -2 0.56×10 -2 Noise 3.60×10 -4 1.89×10 -2 1.49×10 -2 0.93×10 -2 MD-MTD 3.72×10 -4 1.93×10 -2 1.25×10 -2 0.75×10 -2 GAN 2.40×10 -4 1.54×10 -2 1.05×10 -2 0.66×10 -2 GMM-VSG 2.02×10 -4 1.42×10 -2 1.04×10 -2 0.66×10 -2 APS-VSG 1.63×10 -4
In summary, the small sample modeling test results on NASA li-ion battery dataset show that APS-VSG can improve the accuracy of the data-driven model, and the method proposed in this paper is progressiveness compared with other methods.

Conclusion

This paper proposes a virtual sample generation method based on acceptable area and joint probability distribution sampling (APS-VSG) for the expansion of small samples. Break the bottleneck of low validity of MTD in generating high-dimensional virtual samples. The proposed CRI estimation method and the improved acceptable area estimation method are used to limit the acceptable area of data generation and improve the validity of virtual samples. This paper constructs the reasonable joint probability distribution to ensure a high sampling probability near real samples and increase the overall sampling probability in the acceptable area to include part of the edge virtual samples to improve the robustness. This paper uses two-stage sampling to generate high-dimensional data as a whole to avoid the problem of data combination. Experiments on standard function datasets and NASA li-ion battery datasets show that APS-VSG can improve the quality of data and the accuracy of data-driven models and has better performance than MD-MTD, GAN and GMM-VSG.

  Fig. 1. Diagram of MTD

n

  are the number of samples in [min, ] CL and [ , max] CL in a small sample set.

Fig

  Fig. 2.

  Acceptable area construction based on a priori knowledge and trend dispersion (b)Joint probability distribution sampling

Fig. 2 .

 2 Fig. 2. The algorithm framework

  ( ) [min( )] [ ] [min( )]

Fig. 5 .

 5 Fig. 5. The process of constructing a joint probability distribution

  sample truth value. To include part of the edge virtual samples to improve the robustness, a uniform distribution is superimposed based on normal distribution. So when

  distribution is difficult to describe the probability distribution of continuous variables. Therefore, the conditional probability distribution ( | ) P Xp l corresponding to the value l in the middle vacancy of adjacent k values should be calculated,

  three artificial standard function datasets used in the experiment have different data characteristics. The first standard function is a classical linear model, called Linear. The second standard function is a classical nonlinear model called Nonlinear. The third standard function is the model with sinusoidal oscillation, called Oscillation. The three models have three inputs and one output and contain a certain amount of noise to simulate the data collected in the real environment. The applicability of the proposed method to various types of data can be explored and the factors that affect the performance of the method can be analyzed by testing on different standard function datasets. Evenly take 2000 samples from the standard function as the real sample set, and take 50 samples from the real sample set as the small sample set.

Fig. 7 .

 7 Fig. 6. Experimental results of Linear

Fig. 8 .

 8 Fig. 8. Experimental results of Oscillation

Fig. 9 .

 9 Fig. 9. Prediction results and errors of the BP model trained with different training dataTable 5 Performance criteria of the BP model trained with different training data Virtual Data Sources MSE RMSE MAE MAPE Small samples 8.29×10 -4 2.88×10 -22.09×10 -2 1.30×10 -2 Interpolation 1.96×10 -4 1.40×10 -2 0.91×10 -2 0.56×10 -2 Noise 3.60×10 -4 1.89×10 -2 1.49×10 -2 0.93×10 -2 MD-MTD 3.72×10 -4 1.93×10 -2 1.25×10 -2 0.75×10 -2 GAN 2.40×10 -4 1.54×10 -2 1.05×10 -2 0.66×10 -2 GMM-VSG 2.02×10 -4 1.42×10 -2 1.04×10 -2 0.66×10 -2 APS-VSG 1.63×10 -41.27×10 -2 0.88×10 -2 0.55×10 -2 Real samples 1.02×10 -4 1.01×10 -2 0.69×10 -2 0.43×10 -2

Table 1

 1 Definition of standard function

				Define interval
	Dataset	Standard function
		1 x	x	2	x	3	y

Table 2 NASA

 2 Li-ion Battery Dataset (Real sample set)

	Data source	Feature	Meaning	Length	Remarks
		F1	Time for the voltage to rise to 4.2V during the constant current charging process
	B0005 B0007	F2 F3 F4 F5	Time for the current to drop to 20mA during the constant voltage charging process Time for the voltage to drop to 2.7V during the constant current discharge process Total time spent in the discharge process The average temperature during the discharge process	326	the theoretical upper limit of the capacity of the battery is 2Ahr
		Capacity	Capacity of battery
	4

.2 Evaluation 4.2.1 Evaluation of Acceptable area Compare

  the degree of overlap between the acceptable area and the valid area. If the acceptable area overlaps highly with the valid area, the valid coverage level is high. If the acceptable area does not completely cover the valid area and contains lots of invalid areas, the valid coverage level is low.

		 is the ratio of the
		cv
	valid area covered by the acceptable area to the total valid area	 is the ratio of the valid area
		v
	covered by the acceptable area to the total space of the valid area and acceptable area. The
	expressions are as follows:

ca  , cv  , and v  are used to evaluate the acceptable area. ca  is the ratio of the valid area covered by the acceptable area to the acceptable area

  this paper, we use different virtual sample generation methods to obtain the same number of virtual samples, integrate the virtual samples with small samples as the training set of the neural network model, and evaluate the virtual samples by the prediction results of the model. The evaluation criteria used are MAE, MSE, RMSE, and MAPE. MAE is the average of absolute errors, which can better reflect the actual situation of prediction value errors. MSE is the summation average of the square of the difference between the true and predicted values, and RMSE is the root of the MSE. MSE and RMSE are used to detect the deviation between the predicted and true values of the model. MAPE is the mean of the absolute percentage error between the predicted and actual values. The four evaluation criteria are described as follows:

	MAE	1 n 1 || ii i y y n   	(28)
	MSE	1 n 1 () ii i y y n   	2	(29)
	RMSE	1 n 1 () ii i y y n   	2	(30)
	MAPE	1 1 | n i ny  ii i yy    | 100%	(31)

Table 3

 3 Evaluation of the acceptable area

				Evaluation criteria
	Dataset	Acceptable area
		1 xy Q 		0.9962	0.9623	0.9588
	Linear	2 xy Q 		0.9666	0.9062	0.8787
		3 xy Q 		0.9980	0.8653	0.8638
		1 xy Q 		0.9427	0.8586	0.8160
	Nonlinear	2 xy Q 		0.1548	0.9797	0.1543
		3 xy Q 		0.6742	0.9609	0.6562
		1 xy Q 		0.7502	0.9945	0.7471
	Oscillation	2 xy Q 		0.4104	0.9731	0.4058
		3 xy Q 		0.5392	0.9760	0.5321
		Table 4 Direct evaluation of virtual samples
			Evaluation criteria
		Dataset	n	n	
			v	i	v
		Linear	286	14	95.33%
		Nonlinear	258	42	86.00%
		Oscillation	281	19	93.67%

ca  ncv  v 

Table 5

 5 Performance criteria of the BP model trained with different training data

	Virtual Data Sources	MSE	RMSE	MAE	MAPE
	Small samples	8.29×10 -4	2.88×10 -2	2.09×10 -2	1.30×10 -2
	Interpolation	1

  Data expansion algorithms have their own advantages and disadvantages, but they can improve the accuracy of the model under small sample conditions to a certain extent by analyzing the result of the BP model trained with virtual samples generated by five methods. Through quantitative analysis of experimental results, it can be found that the accuracy of the model trained by the proposed method in this paper is improved more compared with the data-driven model under small sample condition. MSE, RMSE, MAE, and MAPE are decreased by 80.3%, 55.9%, 57.9%, and 57.7%, respectively. Compared with Noise and MD-MTD, MSE, RMSE, MAE, and MAPE are decreased by at least 54.7%, 32.8%, 29.6%, and 26.7%, respectively. Compared with GAN and GMM-VSG, MSE, RMSE, MAE, and MAPE are decreased by at least 19.3%, 10.6%, 15.4%, and 16.7%, respectively.The error criteria of Interpolation and APS-VSG are very close. However, Interpolation has a fatal flaw in that the BP model trained with virtual samples generated by Interpolation has difficulty in accurately predicting the interval during which some of the battery capacity picks up.The underlying reason is that the virtual samples generated by Interpolation are basically linear, and when the small sample set is not able to characterize the capacity rebound process, then the virtual data generated by Interpolation also does not have the battery capacity rebound property.Both Noise and MD-MTD have relatively high error criteria. The higher deviation of the result of the BP model trained with virtual samples generated by MD-MTD proves that there are more singular values with large or small values in the prediction results of MD-MTD. If the invalid virtual samples generated by MD-MTD can be further sieved, the accuracy of the model can be greatly improved. In theory, GAN can generate virtual samples that are infinitely close to small sample sets through continuous training. However, due to the small number of samples, the generated virtual samples cannot completely reproduce the real samples in a large number of tests, which leads to an unbreakable bottleneck in the model trained with virtual samples generated by GAN. Due to the stable performance of the Gaussian model, GMM-VSG performs much better than Noise and MD-MTD. However, in localized areas, excessive deviations lead to a slight decrease in the overall performance of the result of the BP model trained with virtual samples generated by GMM-VSG. In many areas where small samples are not well characterized, APS-VSG, like other methods, suffers from large biases. Relatively speaking, APS-VSG delivers excellent performance. The reason is that most of the virtual samples generated by APS-VSG are valid and retain the distribution characteristics of the small sample set well, and the BP model

			1.27×10 -2	0.88×10 -2	0.55×10 -2
	Real samples	1.02×10 -4	1.01×10 -2	0.69×10 -2	0.43×10 -2

Acknowledgment

This work was supported by National Natural Science Foundation of China (62171360), Shaanxi Science and Technology Department (2022GY-110), Xi'an Key Laboratory of Intelligence (2019220514SYS020CG042), National key research and development program (2022YFF0604900), 2022 Shaanxi University Youth Innovation Team Project, Shandong Key Laboratory of Smart Transportation (Preparation), 2023 Shaanxi Provincial University Engineering Research Center.