Mitja Nikolaus
email: mitja.nikolaus@univ-amu.fr

Emmanuelle Salin

Stephane Ayache

Abdellah Fourtassi

Benoit Favre

Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?

Recent advances in vision-and-language modeling have seen the development of Transformer architectures that achieve remarkable performance on multimodal reasoning tasks. Yet, the exact capabilities of these black-box models are still poorly understood. While much of previous work has focused on studying their ability to learn meaning at the word-level, their ability to track syntactic dependencies between words has received less attention. We take a first step in closing this gap by creating a new multimodal task targeted at evaluating understanding of predicate-noun dependencies in a controlled setup. We evaluate a range of state-of-the-art models and find that their performance on the task varies considerably, with some models performing relatively well and others at chance level. In an effort to explain this variability, our analyses indicate that the quality (and not only sheer quantity) of pretraining data is essential. Additionally, the best performing models leverage fine-grained multimodal pretraining objectives in addition to the standard image-text matching objectives. This study highlights that targeted and controlled evaluations are a crucial step for a precise and rigorous test of the multimodal knowledge of vision-and-language models.

Introduction

Vision-and-language (V&L) models have recently shown substantial improvement on a range of multimodal reasoning tasks. Taking inspiration from successes in text-only Natural Language Processing [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF][START_REF] Brown | Language Models are Few-Shot Learners[END_REF], stateof-the-art V&L models are usually composed of a Transformer-based architecture pre-trained in a self-supervised manner on large-scale data, and then fine-tuned on downstream tasks.

While these models show remarkable performance on a range of tasks, more controlled and systematic analyses are necessary in order to obtain Target sentence: A man is wearing a hat.

Distractor Sentence:

A man is wearing glasses.

Figure 1: We evaluate V&L models on their ability to track predicate-noun dependencies that require a joint understanding of the linguistic and visual modalities. The task is to find the correct sentence (choosing between the target and distractor) that corresponds to the scene in the image. In this example, the models should connect the predicate "is wearing a hat" to "man". A model that does not track dependencies would judge the distractor sentence "A man is wearing glasses" as equally likely, as there is a man is the image, as well as a person that is wearing glasses.

a better understanding of their exact multimodal knowledge.

A range of studies has investigated their ability to map words to their visual referents for nouns [START_REF] Kazemzadeh | ReferItGame: Referring to Objects in Photographs of Natural Scenes[END_REF][START_REF] Mao | Generation and Comprehension of Unambiguous Object Descriptions[END_REF][START_REF] Shekhar | FOIL it! Find One mismatch between Image and Language caption[END_REF] and verbs [START_REF] Ruggero | Describing Common Human Visual Actions in Images[END_REF][START_REF] Yatskar | Situation Recognition: Visual Semantic Role Labeling for Image Understanding[END_REF][START_REF] Pratt | Grounded Situation Recognition[END_REF][START_REF] Hendricks | Probing Image-Language Transformers for Verb Understanding[END_REF], but there are only a few studies on whether recent V&L models can capture multimodal syntactic dependencies between words and concepts.

In this paper, we explore how well V&L models learn predicate-noun dependencies across modalities (see example in Figure 1). To this end, we create an evaluation set that contains carefully selected images and pairs of sentences with minimal differences. Given an image and two predicatenoun sentences, the models need to find the correct sentence corresponding to the image. Crucially, they can only succeed by taking into account the dependencies between the visual concepts in the image corresponding to the noun and predicate in the sentence.

As it has been shown that visual reasoning performance in several tasks can be spuriously augmented by capitalizing on textual biases in the training data [START_REF] Goyal | Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering[END_REF][START_REF] Agrawal | Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering[END_REF][START_REF] Lisa | Women also Snowboard: Overcoming Bias in Captioning Models[END_REF][START_REF] Cao | Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models[END_REF], we counter-balance our evaluation dataset in a way that controls for such exploitation of linguistic biases.

We evaluate pre-trained state-of-the-art V&L models in a zero-shot setting and find that the ability to track predicate-noun dependencies varies considerably from model to model. Of all models tested, UNITER [START_REF] Chen | UNITER: Learning Universal Image-Text Representations[END_REF] and LXMERT [START_REF] Tan | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF] show the highest scores, but their performance is still far from optimal. Other models such as ViLBERT [START_REF] Lu | ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks[END_REF] and CLIP [START_REF] Radford | Learning Transferable Visual Models From Natural Language Supervision[END_REF] perform at chance level. We discuss how differences in the models could explain their performance variability, highlighting the role of pretraining data quality and fine-grained multimodal pretraining objectives.

Code to reproduce the analyses and run the evaluation on new models is publicly available at https://github.com/mitjanikolaus/ multimodal-predicate-noun-dependencies.

Related Work

Targeted evaluation of V&L models Recently, a growing number of tasks have been created for targeted evaluation of V&L models' abilities to perform various multimodal reasoning. [START_REF] Shekhar | FOIL it! Find One mismatch between Image and Language caption[END_REF] create sets of distractor captions to analyze whether V&L models are sensitive to single word replacements (with a focus on nouns). Similar targeted evaluation datasets have also been proposed for referring expressions [START_REF] Chen | Cops-Ref: A New Dataset and Task on Compositional Referring Expression Comprehension[END_REF], image-sentence matching (Hu et al., 2019), and Visual Question Answering (VQA;[START_REF] Bogin | COVR: A Test-Bed for Visually Grounded Compositional Generalization with Real Images[END_REF], with a focus on compositional reasoning.

Tasks such as visual semantic role labeling or situation recognition, typically involve classifying the primary activity depicted in an image, as well as the semantic roles of involved entities [START_REF] Ruggero | Describing Common Human Visual Actions in Images[END_REF][START_REF] Lu | Visual Relationship Detection with Language Priors[END_REF][START_REF] Chao | HICO: A Benchmark for Recognizing Human-Object Interactions in Images[END_REF][START_REF] Gupta | Visual Semantic Role Labeling[END_REF][START_REF] Yatskar | Situation Recognition: Visual Semantic Role Labeling for Image Understanding[END_REF][START_REF] Pratt | Grounded Situation Recognition[END_REF]. While these studies demonstrate that V&L models can learn semantic roles to some degree in a supervised learning setup, such tasks do not allow for a controlled evaluation of models in a zero-shot setting.

In Hendricks and [START_REF] Hendricks | Probing Image-Language Transformers for Verb Understanding[END_REF], the authors evaluate state-of-the-art V&L models in a controlled zero-shot setup and find that they still have more trouble understanding verbs compared to subjects or objects. They also observe that models trained on larger datasets with less descriptive captions perform worse than models trained on smaller, manually-annotated datasets.

Several works have also tried to shed more light on the precise multimodal semantic capabilities of V&L models using probing techniques. [START_REF] Salin | Are Vision-Language Transformers Learning Multimodal Representations? A Probing Perspective[END_REF] show that although state-of-the-art V&L models can grasp some multimodal concepts such as color, they still do not fully understand more difficult concepts such as object size and position in the image. [START_REF] Parcalabescu | Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks[END_REF] use probing to demonstrate that such models still lack the capability to correctly count entities in an image. [START_REF] Akula | Words Aren't Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions[END_REF] tests for sensitivity to word order in referring expressions. Similarly, [START_REF] Thrush | Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality[END_REF] studies the ability of V&L models to take word order into account by designing adversarial examples that require differentiating between similar image and text pairs, while the text pairs only differ in their word order. Their results suggest that stateof-the art models still lack precise compositional reasoning abilities. Li et al. (2020a) studies so-called syntactic grounding of VisualBERT. They show that certain attention heads of the transformer architecture attend to entities that are connected via syntactic dependency relationships. However, such probing experiments do not necessarily indicate to what degree a model is actually using the encoded information when making predictions.

Evaluation of grounded syntax

In our work, we test a range of state-of-theart models specifically on their ability to track predicate-noun dependencies. Crucially, we test the models in a much more controlled setting compared to previous work: Our setup involves visual distractors as well as control task, disentangling the challenge of understanding syntactic dependencies from more simple object and predicate recognition. Additionally, we strictly control for any possible linguistic bias by counter-balancing all evaluation examples.

Methods

Evaluation Dataset

We construct an evaluation dataset that is suited for evaluating the sensitivity to visually grounded predicate-noun dependencies in a zero-shot setup.

The data consists of pairs of triplets, and each triplet consists of an Image I, a target sentence S 1 , and a distractor sentence S 2 . Target and distractor sentences are minimal pairs, i.e. one sentence differs from the other only with regard to either the noun (e.g., "A girl is sitting." vs. "A man is sitting.", Figure 2) or the predicate (e.g., "A man is wearing a hat." vs. "A man is wearing glasses.", Figure 1).

Crucially, the images always contain visual distractors, meaning that both the noun and the predicate of the distractor sentence are present in the image, but they do not have a noun-predicate relationship (e.g., for the distractor sentence "A man is wearing glasses", there is a man in the image, who is not wearing glasses, and a person wearing glasses, who is not a man). Thus, it is necessary to take into account the dependency between noun and predicate to distinguish between the target and distractor sentence (Figure 1).

Controlling for linguistic biases V&L models have shown to rely sometimes on textual bias instead of using visual information [START_REF] Goyal | Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering[END_REF][START_REF] Agrawal | Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering[END_REF][START_REF] Lisa | Women also Snowboard: Overcoming Bias in Captioning Models[END_REF][START_REF] Cao | Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models[END_REF]. For example, if a training dataset contains more often the phrase "a girl is sitting" than "a man is sitting", a model might prefer the caption "a girl is sitting" during evaluation only based on linguistic co-occurrence heuristics, irrespective of the visual content. In our evaluation dataset, we control for potential linguistic biases in the training datasets by pairing every triplet with a corresponding counter-balanced example where target and distractor sentence are flipped. More specifically, for every triplet (I 1 , S 1 , S 2 ), there exists a corresponding triplet (I 2 , S 2 , S 1 ), as depicted in Figure 2. In that way, a model that does not take into account the visual modality cannot succeed in the task (see also [START_REF] Nikolaus | Evaluating the Acquisition of Semantic Knowledge from Cross-situational Learning in Artificial Neural Networks[END_REF].

Automatic pre-filtering Our evaluation dataset is based on Open Images [START_REF] Kuznetsova | The Open Images Dataset V4[END_REF]. We pre-filter the images based on existing humanannotated object and relationship labels and bounding boxes. The objects refer to persons, animals, as well as inanimate objects. The relationships can either describe an action that an object is engaged in (e.g., WOMAN -SIT), or an action linking two objects (e.g., MAN -WEAR -GLASSES). All nouns in the selected relationships for our dataset refer to persons, due to lack of sufficient annotations for other kinds of agents.

We look for images that contain a target objectrelationship pair as well as a distractor objectrelationship pair for which either the target and distractor object are the same, but the relationships differ, or vice versa (as in the example in Figure 1). Additional details on the pre-filtering can be found in Appendix A.1.

Manual selection

We manually select suitable images after the automated pre-filtering, in order to ensure high quality of each example and in particular to verify that the distractor sentences are indeed incorrect given the images. This step is crucial, because many of the annotations in Open Images are incomplete, and an image may contain, for example, a woman that is sitting but not annotated as such (in this case, we disregard the image for our evaluation set).

We select pairs of examples and counterexamples and ensure that there are no duplicate images within the set of images for each objectrelationship pair.

Sentence generation We generate target and distractor sentences based on the verified object and relationship annotations from Open Images.

We construct English sentences using a templatebased approach. Given an object and a relationship, we add the indefinite article (a/an) in front of each noun and use all verbs in present progressive tense as this is most frequent in image-text datasets. 1 For example, from WOMAN -IS -SIT we generate "a woman is sitting."; and from MAN -HOLD -CAM-ERA "a man is holding a camera.".

This template-based approach is necessary for our controlled evaluation. As the choice of the exact template for the construction of the sentences may influence the results 2 , we evaluate the models, additionally, using a slightly different template, and we show that the overall result patterns remain largely similar (see Appendix A.4.2).

Final evaluation set

The final evaluation set contains 2584 triplets. For 1486 of these triplets, the distractor sentence contains an incorrect predicate and for the other 1098 triplets, the distractor contains an incorrect noun. More detailed statistics regarding the number of triplets concerning specific concepts are provided in Appendix A.2.

A note on perceived gender annotations Our evaluation dataset uses annotations from the Open Images dataset, which rely on the physical appearance of persons to annotate their perceived gender. We use the provided annotations, and the resulting biases are unfortunately reproduced in our evaluation set. We discuss this issue in further detail in the Ethics Statement (Section 8).

In [START_REF] Salminen | Inter-Rater Agreement for Social Computing Studies[END_REF] gender classification from face pictures by human annotators shows an inter annotator agreement greater than 95%. True gender cannot be classified, and high interannotator agreement does not imply a correct gender choice, but we expect the gender annotations of Open Images to be reliable enough to be used as a basis for our analyses.

Metric

We evaluate pre-trained models on their image-text matching performance in a zero-shot setting, i.e. without any further training. For each triplet, we test whether the models give a higher similarity score for the correct sentence than for the distractor 1 In cases where multiple connecting predicates between a verb and a noun are plausible (e.g. "a man wearing glasses" vs. "a man with glasses"), we choose the construction that occurs most frequently in the Conceptual Captions training data [START_REF] Sharma | Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning[END_REF]. This dataset is most commonly used for training V&L transformers.

2 For example, [START_REF] Ravichander | On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT[END_REF] found that results of some probing experiments can vary substantially with slight changes in wording.

sentence. We calculate accuracy for each pair, i.e. the model needs to succeed for both the example and the counter-balanced example triplet.

For each pair of triplets (t 1 , t 2 ) = ([I 1 , S 1 , S 2 ], [I 2 , S 2 , S 1 ]), we calculate the following score:

f (t 1 , t 2 ) =    1, if s(I 1 , S 1 ) > s(I 1 , S 2 )
and s(I 2 , S 2 ) > s(I 2 , S 1 ) 0, otherwise where s(I, S) denotes the similarity between an image I and a sentence S. To obtain the similarity score, we use the softmaxed output of the imagetext matching pretraining heads of the models. 3The final accuracy is the average score over all pairs in the evaluation set. Chance performance is at 25%. 4As the dataset was manually filtered and requires only rather simple understanding of the images, we assume human performance to be close to 100%. To verify this claim, we had a one person annotate a randomly sampled subset of 500 triplets. For each triplet, the annotator was asked to judge which of the two sentences describes the image better. The resulting performance was at 100%.

A topline: the cropped task In order to explore the effect of the visual distractors on this nounpredicate dependency task, we additionally evaluate all models in a cropped task: We reduce the image to the bounding box of the target object. Thus, the cropped image usually 5 only contains the target object, and no more visual distractors (i.e., the referent of the noun or the predicate in the distractor sentence is no longer present in the cropped image). To succeed at this (simpler) task, the model no longer needs to capture the predicatenoun dependency, it just needs to ground the single words correctly. We use this task to estimate how much the performance of the models is affected by the ability to ground nouns and predicates in our evaluation dataset, in comparison to the (more sophisticated) ability of understanding predicatenoun dependencies.

Models

We consider a range of state-of-the-art V&L models that are pre-trained using text, image, and multimodal pretraining objectives on corpora of parallel image and text data. All models use the transformer architecture [START_REF] Vaswani | Attention is All you Need[END_REF], but vary in terms of pretraining data and objectives, image encoders, and multimodal fusion approaches.

In addition to their image and text pretraining objectives, the models commonly make use of an image-text matching objective, where the models are asked to predict whether a given sentence describes an image or not. We leverage the output of the corresponding pretraining head for calculating image-text similarities for our task. 6 We evaluate LXMERT [START_REF] Tan | LXMERT: Learning Cross-Modality Encoder Representations from Transformers[END_REF], UNITER [START_REF] Chen | UNITER: Learning Universal Image-Text Representations[END_REF], ViLBERT [START_REF] Lu | ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks[END_REF], Oscar (Li et al., 2020b), VinVL [START_REF] Zhang | VinVL: Revisiting Visual Representations in Vision-Language Models[END_REF], ViLT [START_REF] Kim | ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision[END_REF], and CLIP [START_REF] Radford | Learning Transferable Visual Models From Natural Language Supervision[END_REF]. We could not evaluate the original VL-BERT [START_REF] Su | Language (Technology) is Power: A Critical Survey of "Bias" in NLP[END_REF], because it was not pre-trained using an image-text matching loss. We also did not evaluate the original VisualBERT [START_REF] Harold | VisualBERT: A Simple and Performant Baseline for Vision and Language[END_REF], as their implementation of the image-text matching loss requires one correct caption (in addition to a possibly faulty caption), which is not available for the Open Images dataset. Both models were however evaluated in the controlled conditions using VOLTA (see Section 4.2).

Pretraining datasets ViLBERT and VL-BERT are pretrained on Conceptual Captions [START_REF] Sharma | Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning[END_REF]. UNITER, LXMERT, ViLT, and VinVL 7 make use of additional publicly available datasets such as COCO [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF], SBU captions [START_REF] Ordonez | Im2Text: Describing Images Using 1 Million Captioned Photographs[END_REF], Flickr30K [START_REF] Young | From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions[END_REF], VisualGenome [START_REF] Krishna | Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations[END_REF], 6 The multimodal pretraining objective of Oscar does not involve matching and mismatching images and descriptions, but only matching and mismatching sequences of object tags. Therefore, we evaluate the checkpoint that has been fine-tuned for image-text retrieval. 7 VinVL is pre-trained on parts of the Open Images dataset, and we found that most images used in our evaluation set are indeed part of the VinVL pretraining dataset. Because of this confound, results of VinVL are not directly comparable to the other models (even though we test the model here with novel sentences with respect to the images). Our results show however that VinVL compares very closely to Oscar, which has the same architecture, suggesting that the access to the images during training does not substantially affect the model's performance. The same confound is possibly present for CLIP, for which the training data is not public and VQA datasets [START_REF] Goyal | Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering[END_REF][START_REF] Zhu | Visual7W: Grounded Question Answering in Images[END_REF][START_REF] Hudson | GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering[END_REF]. The original VisualBERT is only pre-trained on COCO. Appendix A.3 details the pretraining datasets for all models as well as their sizes. The pretraining data for CLIP has not been publicly released, the authors state that it consists of 400M image-text pairs from the internet (an order of magnitude more data than for most of the other models, which do not surpass 10M image-text pairs in size).

Results

Original Implementations

We test all models using the evaluation methods and data described above. We make use of pretrained models made publicly available by the authors.

Resulting accuracies are shown in Table 1. We find that only some models perform substantially above chance, notably ViLT, UNITER and LXMERT. In the cropped task, performance is much higher for all models, with VinVL and ViLT reaching the highest performance. This gap in performance between the full and cropped tasks indicates that while those models can match nouns and predicates in the image with the corresponding words rather well, they struggle to take into account the dependencies between them.

Accuracy

Controlled Training Conditions

We additionally evaluate models that are trained in controlled (and therefore more directly comparable) conditions as proposed in the VOLTA framework [START_REF] Bugliarello | Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs[END_REF]. In this setup, all models are trained on Conceptual Captions using the same pretraining objectives (masked language modeling, masked object classification, and imagetext matching) and use the same image features, extracted from a Faster R-CNN. We evaluate all models for which pretrained weights are available. Resulting accuracy scores are presented in We find that under these controlled conditions, all models perform comparably and generally around chance level. It is therefore not straightforward to draw any conclusions regarding the effect of model architecture from these results.

In the cropped task, performance is much higher, with ViLBERT and VL-BERT reaching the highest performance. The performance gap between the two tasks (i.e., full vs. cropped) is substantially larger than for the original implementations, suggesting that the models are even less sensitive to predicate-noun dependencies under these controlled training conditions.

Analyses and Discussion

Comparing Model Performances

The role of pretraining data Within the set of the evaluated models, we do not find evidence for a correlation between the size of the pretraining dataset and the model's ability to capture predicatenoun dependencies (see also Appendix A.3). Despite being trained on comparable or even larger amounts of data, ViLT, Oscar and VinVL perform substantially worse than LXMERT and UNITER. CLIP performs below chance level, despite having by far the largest pretraining dataset.

The pretraining data of CLIP is not publicly available, but as it was automatically scraped from the internet we believe the quality (i.e descriptiveness) of its captions to be comparable to that of Conceptual Captions. In additional experiments (see Appendix A.4.1), we study the performance of CLIP models trained on different datasets using a range of publicly available model checkpoints. The performance of CLIP remains below chance level for all tested checkpoints. This might be because all available checkpoints are all trained on rather noisy data, or because the architecture and pretraining objectives of CLIP don't allow it to learn grounded predicate-noun dependencies.

Datasets that are composed of highly descriptive captions seem to be advantageous for the learning of noun-predicate dependencies. Indeed, for datasets such as COCO [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF] or VQA [START_REF] Antol | VQA: Visual Question Answering[END_REF], the images are not only strongly associated with the captions or question-answer pairs (as they were crowdsourced specifically for the tasks), but also precise and detailed in nature. In contrast, Conceptual Captions [START_REF] Sharma | Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning[END_REF] is composed of images with captions that were automatically collected from web pages, and therefore generally rather broad descriptions of the image content.

ViLBERT and models trained in the controlled conditions are only trained using Conceptual Captions, and the resulting performances are around chance level. UNITER and LXMERT perform much worse compared to their original training setups. One main difference for these two models in their original implementation compared to the controlled condition is that they are trained on richer datasets with respect to the language modality, leveraging more descriptive captions. 8This observation is coherent with what Hendricks and Nematzadeh (2021) found when studying verb understanding of V&L models: They compare performance of the same model when trained on Conceptual Captions or COCO, and find that the model trained on COCO performs better, despite Conceptual Captions being bigger and closer to the task in terms of image and language distribution. These results suggest that, when considering multimodal dependencies, having a high quality pretraining dataset with less noise and more de-scriptive textual data could be more important than having a larger dataset. Highly descriptive textual data is essential to learn precise predicate-noun dependencies.

The role of pretraining objectives While models such as ViLT, Oscar, and VinVL are trained on datasets that are comparable in size and quality to those of LXMERT and UNITER, they still perform substantially worse on the task. One explanation could be that contrary to the other models, UNITER and LXMERT both have multimodal pretraining objectives in addition to image-text matching: Visual question answering for LXMERT and word-region alignment for UNITER. 9 This could help the models to establish finer multimodal dependencies. Indeed, ViLT and VinVL show better results than UNITER and LXMERT in the cropped task (indicating that their object/predicate recognition performance even surpasses that of the other models), but worse results in the full task. Our hypothesis is that the pretraining objectives of UNITER and LXMERT enable them to learn more fine-grained multimodal dependencies than ViLT and VinVL, even though their performance on the cropped task is worse. 10 This gap in performance should not only be due to the training data associated with the additional pretraining objectives, as VinVL also uses data from Visual Question Answering task, but without training on the objective.

The impact of the multimodal pretraining objectives of UNITER and LXMERT can be an additional explanation for the drop in performance of CTRL_UNITER and CTRL_LXMERT, which were only trained using image-text matching as a multimodal pretraining objective. The gap in performance between those controlled models and the original models indicate that using more precise multimodal pretraining objectives and better annotated datasets can greatly improve the learning of multimodal dependencies.

The lack of suitable multimodal pretraining objectives could also offer an explanation for the poor performance of CLIP in our task. 9 ViLT also uses a word-patch alignment objective similar to word-region alignment, but the patches are not based on regions detected by an object detector and therefore the loss can not leverage any semantic labels for the patches during training, making this multimodal objective probably less useful.

10 Most directly comparable are probably the cases of ViLT and UNITER, which are both trained on the same datasets, but with different pretraining objectives.

The role of image encoders The authors of ViLT and VinVL motivate their work by suggesting that improved image features are mandatory for improved multimodal reasoning of V&L transformers. Here, we observe that these improved features only translate to better results in the cropped task (where ViLT and VinVL perform best). We speculate that the improved image encoders allow for a better understanding of visual entities, but not necessarily of the dependencies between them. In order to obtain more conclusive interpretations regarding the role of image features, we require more targeted experiments which control for other confounding factors present here (such as different pretraining objectives).

The role of model architecture In addition to the lack of suitable pretraining objectives, the worse performance of CLIP compared to the other models could also be due to the fact that it does not support any kind of inter-modal fusion of features within the model (image and text are processed in separate submodules that do not allow for intermodal interaction). This shortcoming of CLIP is also discussed in [START_REF] Kim | ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision[END_REF], where the authors find representations from CLIP to be not useful for the more advanced multimodal reasoning task NLVR2 [START_REF] Suhr | A Corpus for Reasoning about Natural Language Grounded in Photographs[END_REF].

However, there seems to be no major effect of architecture with respect to multimodal fusion in the case of single and dual stream transformers: LXMERT and UNITER have comparable performances, even though one is dual-stream transformer and the other a single-stream transformer.

Performance for Nouns vs. Predicates

Here, we compare performance for pairs in which the sentences differ with respect to the noun, to sentences with a different predicate. Detailed results for all models are reported in Appendix A.4.3. Overall patterns show a slightly better performance for cases in which the noun was switched, especially in the cropped task. This is in line with findings that V&L models are better at grounding nouns than verbs (Hendricks and Nematzadeh, 2021).

Analysis of individual nouns and predicates

For a given concept (noun or predicate), we consider all pairs that contain this concept in at least one of the two sentences, i.e. cases in which a model's understanding of a concept is instrumental for making the correct decision.

Figure 3 shows the per-concept accuracies of the best performing model, LXMERT. Appendix A.4.4 shows the per-concept accuracies for all models in their original implementations.

Girl

Predicates

Figure 3: Per-concept accuracies for LXMERT. We display nouns and predicates for which we have at least 10 evaluation triplets. Standard deviation calculated using bootstrapping (100 re-samples).

We observe large variation in accuracy scores of predicates, and less variation for nouns. We could not find any simple reasons that explain the predicates' variability. For example, verbs can have good or bad performances (e.g., "running" vs "talking"), and the same can be said for predicates that are composed of both verb and noun (e.g., "holding a bottle" vs. "wearing a helmet"). That said, factors that may influence model performance on specific nouns or predicates are further discussed in Section 5.4.

Additionally, we observe that for some concepts, the models perform better if the concept is the target, and for others, performance is better if it is the distractor. This is, e.g., the case for the pair "sing" vs "stand", where the models consistently perform better if "sing" is the target predicate. Appendix A.4.5 shows the accuracy for each (target, distractor) concept tuple.

Confounding Factors

Here we discuss possible factors influencing the models' performances.

Object salience In most of the images in our evaluation set, the target and distractor persons in the image are not of equal size, and not equally salient (sometimes one is more in the foreground than the other). We explore whether there is an effect on the models' decisions by correlating the models' predictions with target and distractor bounding box size and location.

More specifically, we measure the difference in similarity for target and distractor sentence s(I 1 , S 1 ) -s(I 1 , S 2 ) and correlate it with the difference in bounding box size of the target and distractor object. Further, we also correlate it with the difference of distances from the center of the image.

For LXMERT, we find no significant correlation (Bounding box size: Pearson r = -0.03, p = 0.16, bounding box distance from center: Pearson r = 0.03, p = 0.14). Correlation scores for other models can be found in Appendix A.4.6. While there are statistically significant correlations for some models, these are small and of varying direction. The largest correlations are found for CLIP (Bounding box size: Pearson r = 0.14, p < 0.01, bounding box distance from center: Pearson r = -0.24, p < 0.01), indicating that the performance of CLIP could be affected, to some extent, by object salience.

Concept recognizability

We also correlate the models' similarity judgments differences to differences in concept recognizability, which we operationalize by taking the object or attribute confidence score for a given concept in an image from a Faster R-CNN [START_REF] Shaoqing Ren | Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[END_REF] trained on Visu-alGenome. 11 For most models, we find a small positive correlation (see Appendix A.4.6), indicating that the models' similarity judgments are affected by the varying degree to which the concepts are recognizable in the image.

Linguistic biases Another aspect, already mentioned earlier, is that models' performance could be affected by linguistic biases in the training data, such as the frequency and co-occurrence of words and phrases.

To explore this possible effect, we correlate the difference in similarity for target and distractor sentence with the difference of target and distractor sentence perplexity. We calculate the perplexity for each sentence using a single-modality BERT model (bert-base-uncased), that was fine-tuned for 3 epochs on the textual data of Conceptual Captions.

For LXMERT, we find no significant correlation (Pearson r = -0.01, p = 0.48). For the other models, we find very small positive correlations (see Appendix A.4.6). We conclude that the models do not rely only on shallow heuristics of the training data in the textual modality.

Conclusion

This work examines whether state-of-the-art V&L models learn multimodal syntactic dependencies, by focusing on a case study on simple predicatenoun dependencies. Our controlled experiments and analyses on a range of recent models reveal that their capability track such dependencies is variable, with some models (e.g., LXMERT and UNITER) show performance above chance level and others (e.g., CLIP) performing even below chance.

In contrast to the recent trend in the field focused on increasing pretraining data and using simple general-purpose pretraining objectives [START_REF] Brown | Language Models are Few-Shot Learners[END_REF][START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], here we observe that best performance is achieved, rather, with highquality pretraining data, and more fine-grained pretraining objectives.

More specifically, our results suggest that multimodal pretraining objectives have a major impact on the model's learning of grounded predicatenoun dependencies. Models that include more targeted objectives such as visual question answering and word region alignment in addition to the general image-text matching objective show better performance. In addition, having highly descriptive pretraining datasets seems to help with learning fine-grained multimodal dependencies. In comparison, models trained on larger, web-scraped datasets do not perform well.

In the future, the proposed highly-controlled evaluation protocol can be used to conduct more targeted studies regarding the role of model architecture, pretraining objectives, as well as training data quality and quantity in order to build V&L models that are better at learning grounded predicate-noun dependencies, and possibly also other, move advanced multimodal reasoning tasks.

Limitations

While our analyses revealed patterns that seems to explain observed variability in the models' performance, the role of some architectural choices such as image encoding techniques remains ambiguous. A better understanding of all factors influencing the learning of grounded predicate-noun dependencies could be achieved by training sets of models on comparable conditions and by varying only one factor at a time (as done for example in [START_REF] Bugliarello | Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs[END_REF], regarding the role of model architecture).

The range of concepts evaluated is rather small and therefore not representative for the understanding of grounded predicate-noun dependencies in general. More targeted data collection will be necessary in order to obtain more large-scale evaluation datasets. Additionally, our zero-shot evaluation paradigm introduces a possible mismatch between training and evaluation: Models are trained using pairs of images and descriptions where the descriptions often describe all salient parts of the image, whereas in our evaluation set the descriptions focus on only one aspect/person in the image.

In the cropped condition, the images are not are not representative of the typical photographic framing of image-text corpora, which could deteriorate our results. That said, random cropping is a frequent data augmentation technique in computer vision research, where it has been successfully applied to improve generalization performance [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF].

Further, some scenes, actions and cultures are disproportionally represented in our evaluation dataset. As proposed in [START_REF] Liu | Visually Grounded Reasoning across Languages and Cultures[END_REF], it is important to pursue further work on more diverse datasets.

The proposed evaluation set relies on subjective annotations of perceived gender. Attempting to classify gender based on physical appearance is an illposed problem (e.g., due to limitations of object detectors, biases of the human annotators). Additionally, the annotations only consider binary gender classes (woman/man, girl/boy). Algorithms that perform such classifications are neither ideal nor desirable, as they perpetuate harmful stereotypes and exclude non-binary gender identities (e.g., [START_REF] Dev | Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies[END_REF][START_REF] Hamidi | Gender Recognition or Gender Reductionism? The Social Implications of Embedded Gender Recognition Systems[END_REF][START_REF] Su | Language (Technology) is Power: A Critical Survey of "Bias" in NLP[END_REF][START_REF] Bender | On the Dangers of Stochastic Parrots: Can Language Models Be Too Big[END_REF]. We explored whether it would be possible to use other classes, but we did not find many examples that would allow for an evaluation of sensitivity to predicate-noun dependencies in a controlled fashion. As our image selection is very constrained (we require a visual distractor, and a counter-example image with reverse properties), we found only sufficient examples for the categories used in the paper. We initially started a bottom-up data exploration in which we considered all labels present in the Open Images dataset, but found only very few examples for a few other categories (generally less than 5 examples after manual filtering). This might be due to the focus of the annotations in Open Images, future work could explore the use of other datasets that are focused on other types of annotations, the main challenge being the requirement for sufficiently large datasets in order to find matching examples and counter-examples. Future efforts should be dedicated to creating datasets that aim at more inclusive annotations.

We acknowledge the severity of these issues, and emphasize that our work does not promote applications of gender classification in downstream tasks, but only uses it as a basis for analysis of existing models.

A Appendix

A.1 Image Pre-Filtering

We consider labels that occur at least 100 times in the dataset.

As some labels are similar and sometimes used interchangeably by the annotators, we create groups of synonyms for some labels and treat labels within a group as identical in the following. The groups of synonyms can be found in Table 3.

Further, we verify that the bounding boxes of the target and distractor objects are big enough (at least 20% width and 20% height of the image) and that the bounding box sizes of target and distractor objects don't differ by more than a factor of 2.

Finally, we ensure that there is at least one counter-example for each triplet before starting the manual image selection phase.

"Table ", "Desk", "Coffee table" "Mug", "Coffee cup" "Glasses", "Sunglasses", "Goggles" "Sun hat", "Fedora", "Cowboy hat", "Sombrero" "Bicycle helmet", "Football helmet" "High heels", "Sandal", "Boot" "Racket", "Tennis racket", "Table tennis racket" "Crown", "Tiara" "Handbag", "Briefcase" "Cart", "Golf cart" "Tree", "Palm tree" "Football", "Volleyball (Ball)", "Rugby ball", "Cricket ball", Tennis ball"

A.2 Dataset statistics

Figure 4 shows the number of triplets for each noun and predicate. For a given noun or predicate, we count all pairs that contain this concept in at least one of the two sentences, i.e. cases in which correct understanding of a concept is useful for making the correct decisions.

A.3 Details on V&L Models

A.3.1 Pretraining Datasets

Table 4 details the multimodal datasets used for V&L models. Dataset sizes as reported in the corresponding papers. Note that these sizes are also affected by the fact that some models leverage validation sets for pretraining, while others constrain the data to the training sets. Also, different approaches for dataset overlap detection have been applied. The pretraining data size for CLIP is reportedly 400M image-text pairs.

A.3.2 Number of parameters

Table 5 compares the number of trainable parameters for each model that was tested in their original implementations.

A.4 Additional Analyses

A.4.1 Results for CLIP with varying pretraining data [START_REF] Sharma | Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning[END_REF], COCO [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF], SBU captions [START_REF] Ordonez | Im2Text: Describing Images Using 1 Million Captioned Photographs[END_REF], VG (Krishna et al., 2017, Visusal Genome;), F30K (Flickr 30K;[START_REF] Young | From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions[END_REF], OI (Open Images), and QA (including VQA2.0 [START_REF] Goyal | Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering[END_REF], GQA [START_REF] Hudson | GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering[END_REF], and VG-QA [START_REF] Zhu | Visual7W: Grounded Question Answering in Images[END_REF]).

  Target sentence: A girl is sitting.Distractor Sentence: A man is sitting.Target sentence: A man is sitting.Distractor Sentence: A girl is sitting.

Figure 2 :

 2 Figure 2: Counter-balanced evaluation: Each triplet has a corresponding counter-example, where target and distractor sentence are flipped.

Figure 4 :

 4 Figure 4: Number of triplets per concept. Note that the x-axis is on logarithmic scale.

Table 1 :

 1 Accuracy of models trained in original conditions when provided the full images and when only exposed to the target object in the cropped task.

	Model	Full Cropped
	LXMERT 0.57	0.69
	UNITER 0.54	0.64
	ViLBERT 0.28	0.66
	ViLT	0.40	0.75
	Oscar	0.32	0.67
	VinVL	0.30	0.76
	CLIP	0.20	0.59
	Chance	0.25	0.25

Table 2 .

 2

		Accuracy
	Model	Full Cropped
	CTRL_UNITER	0.24	0.63
	CTRL_LXMERT	0.20	0.56
	CTRL_ViLBERT	0.27	0.66
	CTRL_VL-BERT	0.24	0.66
	CTRL_VisualBERT 0.20	0.64
	Chance	0.25	0.25

Table 2 :

 2 Accuracy of models trained in controlled conditions when provided the full images and when only exposed to the target object in the cropped task.

Table 3 :

 3 Groups of label synonyms. Each line corresponds to one group.

  Table 6 presents the accuracy scores of multiple publicly available checkpoints for CLIP trained on different training data.

										Total size
	Model	CC COCO SBU VG QA F30K OI # images # image-text pairs
	LXMERT		✓		✓	✓			0.18M	9.18M
	UNITER	✓	✓	✓	✓				4.16M	9.59M
	ViLBERT ✓							3.10M	3.10M
	ViLT	✓	✓	✓	✓				4.05M	9.85M
	Oscar	✓	✓	✓		✓	✓		4.10M	6.50M
	VinVL	✓	✓	✓		✓	✓	✓	5.65M	8.85M

Table 4 :

 4 Pretraining datasets of the tested models: CC (Conceptual Captions;

For the model CLIP, we feed the image and both sentences at the same time, and obtain a similarity score for both sentences, where s(I1, S1) = 1 -s(I1, S2).

 4 The model succeeds if the similarity scores fall into one of four possible configurations: s(I1, S1) > s(I1, S2) ∧ s(I2, S1) > s(I2, S2); s(I1, S1) < s(I1, S2) ∧ s(I2, S1) < s(I2, S2); s(I1, S1) > s(I1, S2) ∧ s(I2, S1) < s(I2, S2); s(I1, S1) < s(I1, S2) ∧ s(I2, S1) > s(I2, S2

).5 If the bounding boxes of the target and visual distractor object overlap to a high degree, the cropped image might still contain (parts of) the distractor object.

The original pretraining datasets for UNITER and LXMERT are also larger in terms of the number of imagetext pairs. However, LXMERT is actually trained on much fewer unique images than in the controlled conditions (180K vs. 3.1M). Therefore, we assume that the sheer size is not the driving factor of performance.

If there are multiple objects/attributes with the corresponding label in the image, we take the maximum confidence.

Acknowledgements

We thank the anonymous reviewers for their useful comments and feedback.

Research supported by grants ANR-16-CONV-0002 (ILCB), ANR-11-LABX-0036 (BLRI), ANR-21-CE28-0005-01 (MACOMIC), AMX-19-IET-009 (Archimedes Institute) and the Excellence Initiative of Aix-Marseille University (A*MIDEX). This work was performed using HPC resources from GENCI-IDRIS (Grant 2021-[101693]).

Model

A.4.2 Controlling for linguistic robustness

As the sentences used in our evaluation dataset are built from a template, they do not vary in syntax.

We verify that results obtained do not depend on the exact template chosen.

We vary the original templates by using the definite article ("the") at the beginning of sentences, and using verbs in simple present instead of present progressive tense (e.g., "the woman sits." or "the man holds a camera.").

Results with these alternative sentences are show in Table 7. We find that overall result patterns are highly similar to those with the original sentences in Table 1

A.4.3 Switching noun vs. switching predicate

Table 8 presents the accuracy for cases in which target and distractor sentence differ with respect to the predicate, or noun. We report scores for all models in their original implementations.

A.4.4 Analysis of individual nouns and predicates for all models

Figure 5 shows the accuracies for split up for the different predicates and nouns for all models. For more details, refer to Section 5.3.

A.4.5 Accuracies for (target, distractor) tuples

Figure 6 shows the accuracy for target-distractor tuples for all models.

A.4.6 Confounding Factors

In Table 9 we show the correlation scores for several confounding factors as described in Section 5.4. 0.12 (p=0.00) -0.17 (p=0.00) 0.06 (p=0.00) 0.15 (p=0.00) VinVL 0.12 (p=0.00) -0.12 (p=0.00) 0.05 (p=0.01) 0.04 (p=0.05) CLIP 0.14 (p=0.00) -0.24 (p=0.00) 0.08 (p=0.00) 0.17 (p=0.00) : Accuracy for (target, distractor) tuples for all models. We display tuples for which we have at least 10 evaluation triplets. Note that chance performance is at 0.5, because we report per-triplet (and not per-pair) accuracy.