Mustafa Shukor

Corentin Dancette

Matthieu Cord

eP-ALM: Efficient Perceptual Augmentation of Language Models

Large Language Models (LLMs) have so far impressed the world, with unprecedented capabilities that emerge in models at large scales. On the vision side, transformer models (i.e., ViT) are following the same trend, achieving the best performance on challenging benchmarks. With the abundance of such unimodal models, a natural question arises; do we need also to follow this trend to tackle multimodal tasks? In this work, we propose to rather direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. In particular, they still train a large number of parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP) trained on huge image-text datasets, and add significant inference overhead. In addition, most of these approaches have focused on Zero-Shot and In Context Learning, with little to no effort on direct finetuning. We investigate the minimal computational effort needed to adapt unimodal models for multimodal tasks and propose a new challenging setup, alongside different approaches, that efficiently adapts unimodal pretrained models. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning across Image, Video, and Audio modalities, following the proposed setup.

Introduction

Going large scale has led to outstanding performances that consistently improve across tasks, modalities, and domains on current benchmarks. Most of the progress so far has been in the vision and language domains. For Computer Vision, the ViT family [21] starts from the tiny model with 5M parameters to the enormous ViT-e [START_REF] Chen | Pali: A jointlyscaled multilingual language-image model[END_REF] with 4B pa-Question: "are either of the players wearing orange socks ?"

"no"

Audio Encoder (AST)

Video Encoder (TimeSformer)

Language Decoder (OPT)

Image Encoder (ViT)

Figure 1. Illustration of eP-ALM to adapt unimodal models for multimodal tasks. The Language Model (Decoder) is augmented with perceptual context to steer its text generation. To condition the decoder on a given modality, the [CLS] tokens are extracted from several layers of a modality-specific encoder and then linearly projected before concatenation at different levels of the language decoder. Only unimodal models are used, and all pretrained modules are kept frozen.

rameters and the largest ViT-22B with 22B parameters [START_REF] Dehghani | Scaling vision transformers to 22 billion parameters[END_REF]. More captivating, are the scales of Large Language Models (LLMs), such as the BLOOM [76] and OPT [START_REF] Zhang | Opt: Open pre-trained transformer language models[END_REF] families, ranging from hundreds of millions of parameters to 175B, in addition to other models that go beyond 100B [START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF][START_REF] Smith | Using deepspeed and megatron to train megatronturing nlg 530b, a large-scale generative language model[END_REF] up to 1T parameters [27]. These huge scales come with a need for very large pretraining datasets and long training times. The current prevalent paradigm to solve multimodal tasks, in particular, Vision-Language tasks is to leverage pretrained models, and then further train end-to-end [START_REF] Chen | Pali: A jointlyscaled multilingual language-image model[END_REF]56,79,[START_REF] Singh | Flava: A foundational language and vision alignment model[END_REF][START_REF] Wang | Image as a foreign language: Beit pretraining for all vision and visionlanguage tasks[END_REF] on large image-text datasets. However, the training cost is huge and unaffordable for a large portion of the community, as these approaches still train all model parameters, even after initialization, on a huge amount of data.

With the abundance of unimodal models, a natural question arises; Do we need also to follow this trend to tackle multimodal tasks? or rather direct effort to efficient adaptations of existing models?

Drawing inspiration from the recent work in Augmented Language Models (ALMs) [START_REF] Mialon | Augmented language models: a survey[END_REF], in this paper, we advocate for adapting pretrained LMs to solve multimodal tasks. Specifically, by augmenting LMs with perceptual encoders.

Several approaches have deviated from the end-to-endtraining paradigm by freezing some pretrained modules and training only the adaptation parameters, such as, additional cross-attention [3], vision encoder [START_REF] Tsimpoukelli | Multimodal few-shot learning with frozen language models[END_REF] and Adapters [23].

Even though these approaches have taken a big step towards more parameter-efficient models, there are still many costly components that hinder their adoption by the large community, such as the training and inference memory and time cost.

In this work we argue that current approaches are far from optimal and it is possible to find more efficient approaches, in terms of the number of trainable parameters, training data, and compute, to adapt pretrained unimodal models for multimodal tasks. A better alignment of visual and language representations might help to devise extremely efficient adaptation approaches.

To investigate this hypothesis, we go a step further to efficiently leverage LLMs, and propose [START_REF] Agrawal | Rethinking evaluation practices in visual question answering: A case study on out-of-distribution generalization[END_REF] a new technique to adapt unimodal models by freezing more than 99 % (up to 99.94%) of their parameters, alongside (2) a minimal and challenging setup to adapt pretrained unimodal models for Image/Video/Audio-Language tasks (e.g., VQA [START_REF] Goyal | Making the v in vqa matter: Elevating the role of image understanding in visual question answering[END_REF]97], Image and Audio Captioning [START_REF] Chen | Microsoft coco captions: Data collection and evaluation server[END_REF]48]). In this setup, we favor unimodal-only models, avoiding multimodal pretraining or massively trained multimodal encoders, and considering the typical LLMs architecture as the backbone. All that while freezing as much as possible of model parameters. The approach is illustrated in Fig. 1.

Specifically, we adopt the publicly released OPT model [START_REF] Zhang | Opt: Open pre-trained transformer language models[END_REF] and unimodal encoders (e.g., ViT,TimeSformer [6], AST [START_REF] Gong | AST: Audio Spectrogram Transformer[END_REF]), which are kept frozen. We finetune directly the adaptation parameters on publicly available benchmarks of downstream tasks such as for VQA, GQA, Image Captioning, Video QA, Video Captioning, and Audio Captioning.

Based on this setup we investigate different design choices and propose very efficient approaches backed by the following interesting findings:

• Training a single linear layer directly on downstream multimodal datasets, and following the same setup, outperforms other work on Image/Video/Audio-Language tasks. With a few additional trainable parameters and a single learned prepended token, we can significantly improve the performance, while respecting a budget of 1% of trainable parameters, and keeping almost the same inference cost.

• Our approach enjoys better generalization (OOD, Zero-Shot) and is data-efficient (training on 1% of the data achieves 80% of performances) with better few-shot results than other approaches.

• While reaching good performance with small to midscale language models (i.e, 350M-2.7B) the improvement still increases by jointly scaling both vision and language models. When scaling both models, we can still outperform other approaches with only 0.06% of trainable parameters.

• Existing approaches do not behave well on the proposed challenging setup, without large multi-modal pretraining.

Related Work

Vision-Language Models (VLMs). Previously, visionlanguage tasks have been solved with models heavily customized for the particular task at hand [START_REF] Cadene | Murel: Multimodal relational reasoning for visual question answering[END_REF]26,41,[START_REF] Jiang | In defense of grid features for visual question answering[END_REF]47]. The success in Self Supervised Learning [9, [START_REF] Grill | Bootstrap your own latent-a new approach to self-supervised learning[END_REF]36,[START_REF] Tian | What makes for good views for contrastive learning?[END_REF]96] and the importance of good initialization have pushed researchers to transfer these ideas to VLMs and started Vision-Language Pretraining (VLP) on large scale video-text [START_REF] Fu | Violet: End-to-end video-language transformers with masked visual-token modeling[END_REF][START_REF] Li | Lavender: Unifying video-language understanding as masked language modeling[END_REF][START_REF] Wang | Omnivl: One foundation model for image-language and video-language tasks[END_REF], image-text datasets in general domains [START_REF] Chen | Uniter: Universal image-text representation learning[END_REF][START_REF] Kim | Vilt: Visionand-language transformer without convolution or region supervision[END_REF][START_REF] Li | Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation[END_REF]56,[START_REF] Lu | Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF]79], as well as specific domains, such as Cooking [START_REF] Shukor | Structured vision-language pretraining for computational cooking[END_REF], Medical Images [71] and Event Extraction [START_REF] Li | Clip-event: Connecting text and images with event structures[END_REF]. VLP is a step to move away from the burden of customization by having one pretrained model, exploited for several downstream tasks. Recently, we have witnessed impressive work that go a step further towards more unification, by unifying the model, the training objective, and input-output format [START_REF] Chen | Pali: A jointlyscaled multilingual language-image model[END_REF][START_REF] Lu | Unified-io: A unified model for vision, language, and multi-modal tasks[END_REF][START_REF] Wang | Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework[END_REF][START_REF] Wang | Image as a foreign language: Beit pretraining for all vision and visionlanguage tasks[END_REF]. All these models train most of the model parameters, even after initialization, which becomes more and more costly with the current trend in scaling data, model size, and compute [START_REF] Chen | Pali: A jointlyscaled multilingual language-image model[END_REF][START_REF] Yu | Coca: Contrastive captioners are image-text foundation models[END_REF]. Another approach for VLM is to exploit existing pretrained models by keeping them frozen and training only the adaptation parameters [3,23,[START_REF] Liang | Modular and parameter-efficient multimodal fusion with prompting[END_REF]. This work advocates for the latter favoring training efficiency in terms of memory and time.

Adapting Language Models. Large Language Models (LLMs) [START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF]38,76,[START_REF] Smith | Using deepspeed and megatron to train megatronturing nlg 530b, a large-scale generative language model[END_REF][START_REF] Zhang | Opt: Open pre-trained transformer language models[END_REF] have impressed the world in this last few years, showing unprecedented performance on a myriad of NLP tasks. Scaling LLMs to hundreds of billions of parameters has been motivated by the capabilities that surprisingly emerge [START_REF] Wei | Emergent abilities of large language models[END_REF] at this scale and lead to sudden jumps of relevant metrics on hard downstream tasks [START_REF] Hendrycks | Measuring massive multitask language understanding[END_REF][START_REF] Patel | Mapping language models to grounded conceptual spaces[END_REF][START_REF] Srivastava | Beyond the imitation game: Quantifying and extrapolating the capabilities of language models[END_REF]. This generalization ability pushed researchers to start adapting these models for other modalities [3,[START_REF] Tsimpoukelli | Multimodal few-shot learning with frozen language models[END_REF], tasks [35,88,[START_REF] Yang | An empirical study of gpt-3 for few-shot knowledge-based vqa[END_REF][START_REF] Zeng | Socratic models: Composing zero-shot multimodal reasoning with language[END_REF] and domains [START_REF] Singh | Progprompt: Generating situated robot task plans using large language models[END_REF]. Currently, most of the focus is concentrated on exploiting LLMs for visionlanguage tasks, such as Flamingo [3] which trains 10B parameters to adapt a frozen 70B parameter language model, and other successful efficient techniques that are based on vision-conditioned prompt tuning (Frozen [START_REF] Tsimpoukelli | Multimodal few-shot learning with frozen language models[END_REF], Prompt-Fuse [START_REF] Liang | Modular and parameter-efficient multimodal fusion with prompting[END_REF], LiMBeR [START_REF] Merullo | Linearly mapping from image to text space[END_REF]) and adapters (MAGMA [23]). This work has demonstrated good performance, showing that it is possible to devise very efficient approaches to adapt existing language models [38,[START_REF] Wang | GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model[END_REF]. On the video side, little work has been proposed, mostly based on Adapters [78,[START_REF] Yang | Zero-shot video question answering via frozen bidirectional language models[END_REF]. The closest to our approach is PromptFuse [START_REF] Liang | Modular and parameter-efficient multimodal fusion with prompting[END_REF] which finetunes directly for VQA, however, they use encoder-decoder language models and train a soft prompt that is prepended to the input.

Efficient Learning. Parameter-Efficient learning is an interesting line of research that consists of adapting pretrained models using very few trainable parameters. Prompt Tuning [54] is one such approach that appends a few learnable tokens, or Soft Prompts to contextualize the input and steer the output of the frozen model toward the desired task. Other approaches use Adapters [START_REF] Bapna | Simple, scalable adaptation for neural machine translation[END_REF][START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF], which are trainable MLP, consisting of 2 linear projection layers with activation in between and inserted inside the model to adapt the self-attention and feedforward layers. Many other approaches have been proposed in the context of NLP such as LoRa [START_REF] Edward | LoRA: Low-rank adaptation of large language models[END_REF], Bitfit [START_REF] Ben Zaken | Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models[END_REF], Hyperformer [START_REF] Karimi Mahabadi | Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks[END_REF], Compacters [START_REF] Karimi Mahabadi | Compacter: Efficient low-rank hypercomplex adapter layers[END_REF] and (IA) 3 [62]. These approaches have been successfully adapted to other modalities such as image [13, 43], image-text [START_REF] Sung | Lst: Ladder side-tuning for parameter and memory efficient transfer learning[END_REF][START_REF] Sung | Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks[END_REF][START_REF] Zhou | Conditional prompt learning for vision-language models[END_REF], with very little work on video [START_REF] Pan | St-adapter: Parameter-efficient image-to-video transfer learning for action recognition[END_REF] and Audio [START_REF] Kim | Integrated parameter-efficient tuning for general-purpose audio models[END_REF].

Another line of research is Data-Efficient techniques, where the objective is to attain similar performance by significantly reducing the training datasets. Recently, some efforts have been proposed for vision [89], language [START_REF] Du | Glam: Efficient scaling of language models with mixture-of-experts[END_REF] and visionlanguage [START_REF] Chen | Visualgpt: Data-efficient adaptation of pretrained language models for image captioning[END_REF][START_REF] Cheng | Vindlu: A recipe for effective video-and-language pretraining[END_REF]79], which mostly focus on designing better training objectives [79]. However, little work has been done to investigate the connection between parameter efficiency and data efficiency, which is considered in this work.

Framework

To solve multimodal tasks, we propose to augment pretrained LLMs with perception through unimodal perceptual encoders (Fig. 1). We detail our approach in the following.

eP-ALM

We augment a pretrained LM with perception through several modality-specific encoders. The encoders interact with LM through linearly projected, modality-specific [CLS] tokens. To ease the adaptation, we leverage some parameterefficient techniques, such as Prompt Tuning. In this section, we detail the design principles of our approach, which is illustrated in Fig. 2.

Language Model (LM) We adopt OPT models [START_REF] Zhang | Opt: Open pre-trained transformer language models[END_REF], which are autoregressive language decoders consisting of Self-Attention and Feed Forward layers. They are trained with next token prediction objective on 180B tokens mostly in English and gathered from different datasets [START_REF] Baumgartner | The pushshift reddit dataset[END_REF][START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF]. The [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. For Video, we use TimeSformer [6] that consists of a ViT-like model augmented with temporal attention and pretrained on kinetics [START_REF] Carreira | A short note about kinetics-600[END_REF]. For Audio, we adopt AST [START_REF] Gong | AST: Audio Spectrogram Transformer[END_REF], a vanilla adaptation of ViT to digest spectrograms, that is pretrained on AudioSet [START_REF] Jort | Audio set: An ontology and humanlabeled dataset for audio events[END_REF]. Even though we consider only these 3 encoders, the extension of the approach to other types of encoders and modalities is straightforward.

Perceptual Prompt Injection. LMs are usually controlled via different textual prompts, such as questions and instructions. Here, the LM is controlled by both the text and the perceptual encoders. Specifically, the projected perceptual tokens are prepended to the textual tokens. Naively using all visual tokens, adds significant computation costs during training and inference, due to the quadratic complexity of attention layers with the number of tokens. This becomes more apparent with LLMs. To mitigate this, we consider only the [CLS] token of the perceptual encoders and prepend it to the text tokens. This increases the total number of tokens by 1 which maintains almost the same inference speed.

Connecting Models with Cross-Modal Hierarchical Linear layers. When freezing the perceptual encoders and language models, the minimal number of trainable parameters are those that amount to connecting these two models while adjusting the embedding dimensions in case of a mismatch. Therefore, we base our approach on this constraint and train only one linear projection layer (single connection, Fig. 2) to connect both models. To exploit the hierarchical representation encoded in pretrained models, instead of taking only the [CLS] token of the last output layer, we take the [CLS] tokens from several layers of the perceptual model, and we inject these tokens into several layers of the LM (shared connection). The tokens coming from early layers are injected earlier and are then replaced by those coming from deeper layers. We favor only the deeper layers (e.g., the last 6 layers of the ViT-B/16, and the last 12 layers of OPT-350M) where the representations are more abstract and less modality-specific. Moreover, using the same linear projection at different representation levels might not help to capture the particularity of such a hierarchy, to this end, we also experiment with different linear layers for each level (multiple connections).

Multimodal Adaptation with Parameter-Efficient Techniques. We explore several parameter-efficient techniques to ease the adaptation to multimodal tasks. The main technique we use is Prompt Tuning [54]: it consists of prepending trainable tokens or Soft Prompts to the textual tokens input of the LM. This gives useful context to steer the model output. Contrary to hard prompts that are manually engineered, this provides a more flexible and easier approach for task-dependant contextualization. For the sake of efficiency, we prepend only 10 learnable tokens. We also experiment Adapters [START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF] as detailed later. The approach can be formalized as follows (better read with Fig. 2):

[CLS] i = C(E i (X)), i = N E /2, ..., N E , t j = LM j ([CLS] i , p j-1 , t j-1 ), j = N L /2, ..., N L , (1)
where [CLS] i is the perceptual token of the input X extracted from the layer i of the perceptual encoder (E i ) with N E layers.

[CLS] i is projected using the linear connection C and prepended, alongside the Soft Prompt p to the embeddings of the textual tokens t j-1 coming from previous layer in the LM (LM j-1 ). This operation is repeated each 2 layers in the LM (with N L layers).

Efficiency-driven Training Framework Setup

Current approaches still rely on many costly components that hinder their adoption by the large community. Specifically; they (1) still train a lot of parameters (e.g. vision encoders [START_REF] Tsimpoukelli | Multimodal few-shot learning with frozen language models[END_REF] and adapters [23] with ∼325M params/5.11%), (2) still maintain the multimodal pretraining with image-text pair datasets on top of the unimodal pretraining [23, 69, 90],

(3) leverage multimodal encoders such as CLIP, pretrained on 400M image-text pairs [23, 69], (4) add significant computation overhead during inference, due to the long visual prompt, especially when evaluating with In Context Learning (ICL), that becomes common with LLMs [23,[START_REF] Merullo | Linearly mapping from image to text space[END_REF]. In this work, we propose a new setup to adapt unimodal models for multimodal downstream tasks. The setup is more challenging and is motivated by the quest for the least effort needed to exploit pretrained models. The setup is the following:

• Training only adaptation parameters (e.g., Soft Prompt, linear connection), while keeping as much as possible of pretrained parameters frozen (parameter efficient).

• Avoiding multimodal pretraining and finetuning directly on downstream multimodal datasets (data/compute efficient).

• Using only pretrained unimodal models, and avoid using multimodal encoders pretrained on huge datasets (data efficient).

• Keeping fast inference (e.g., 1 additional token), by avoiding long prompts, and using additional heavy modules (compute efficient).

• Using decoder-only language models (e.g., OPT), the current architecture adopted by LLMs (due to its pretraining efficiency and open-ended generation capacity).

Specifically, we train only the linear connection and the soft prompt directly on the downstream multimodal tasks. This amounts to less than 1% of trainable parameters that we can push further to 0.06% with big models.

The Pretrain Zero-shot Setup. The focus of this work is direct finetuning on target datasets. However, the proposed mechanism (Sec.3.1) can be adapted straightforwardly to the pretrain-zeroshot setup. In the appendix, we show that eP-ALM outperforms previous work and it is competitive with recent SoTA following the zero-shot evaluation.

Experiments

Implementation details. We use OPT-2.7B in our main model, eP-ALM, and we experiment in Section 4.2 with OPT models of various sizes. We extract the [CLS] tokens of the last 6 layers of perceptual encoders and prepend them, after a linear projection, to the text tokens of the last 12 layers of the OPT. Note that we replace the previous [CLS] with the new one to keep the same number of tokens.

For VQA and VideoQA, we cast the problem as openended generation and compute the accuracy after a strict comparison between the output text (without truncation) and the ground truth one. Note that this setting is more challenging compared to classification-based VQA and not in favor of our approach as the model might generate semantically correct answers but using different words. We use a special token ('</a>') to separate the question from the answer. For captioning, we report the widely adopted CIDEr and BLUE@4 scores. We finetune with the classical crossentropy loss used to train the original OPT for VQA and Captioning tasks. We use the AdamW optimizer with a learning rate (lr) of 1e-5 warmed up to 2e-5 then decreased to 1e-6 using a cosine scheduler. We train for 8 epochs with a batch size of 64 (128 for GQA) and an image resolution of 224. Training our approach with OPT-2.7B for VQA v2 can be done on a single V100 GPU 32GB for few hours. More details are given in the appendix. We find the method sensitive to the text decoding approach (Tab. 8). Following other work, we use greedy decoding with beam search for the main results (Sec. 4.1), and multinomial/random sampling for the ablation study (Sec. 4.2).

eP-ALM Variants. Our main model, eP-ALM (illustrated in Figure 2), has multiple linear connections; specific learned linear layers for each [CLS] token injected in the model. In addition to Prompt Tuning. We also test variants of this model: eP-ALM ada (eP-ALM with Adapters instead of Soft Prompts), eP-ALM lin (trains a shared linear connection with all [CLS] tokens, and no prompt tuning) and eP-ALM pt (lin + Soft Prompt). For Adapters, we follow other work [23] and add sequentially one adapter module after self-attention and feedforward layers in all the blocks of OPT. While this might give better results, it adds a significant number of trainable parameters.

Main Results

In this section, we present the main comparison with other approaches. We present the results for the image modality in Section 4.1.1, the video modality in Section 4.1.2, and the audio modality in Section 4.1.3.

Image-Text Results

We use a frozen ViT-B/16 pretrained on ImageNet1K as the image encoder. We consider the following image-text benchmarks; VQA v2 [START_REF] Goyal | Making the v in vqa matter: Elevating the role of image understanding in visual question answering[END_REF], GQA [42] and COCO Caption [START_REF] Chen | Microsoft coco captions: Data collection and evaluation server[END_REF]. We use Karpathy splits for VQA v2 and COCO, unless specified otherwise.

Baselines. As we are the first to propose this setup, to have a fair comparison, we reimplemented some of the existing approaches and use the same vision (ViT-ImageNet) and language (OPT) models for all: 1) B P romptF use ; which is equivalent to PromptFuse [START_REF] Liang | Modular and parameter-efficient multimodal fusion with prompting[END_REF] and uses Prompt Tuning (N=10). We add a linear projection for the last [CLS] token. The [CLS] token is prepended to the input of the LM. Note that we could not avoid adding a trained linear projection as there is a mismatch between the dimensions of the vision and language model. 2) B M AGM A ; which is equivalent to MAGMA [23] and uses Adapters. We prepend the [CLS] token to the input of LM after linear projection. Note that, we consider only the [CLS] token as we find it better than prepending all image tokens (eP-ALM * M AGM A ). We also find that training the ViT degrades the performance, thus we keep it frozen in favor of their approach. 3) B LimBEr ; which is equivalent to LimBEr [START_REF] Merullo | Linearly mapping from image to text space[END_REF] and only trains the linear projection to project visual tokens and prepend them to the input text. Similarly, we only consider the [CLS] token as it gives better accuracy.

Comparison to Other Work. Based on our study (Sec. 4.2), we use ViT-B/16 and OPT-2.7B in our main model and in our replication of other approaches. In Table 1 we compare with other work on VQA v2, GQA, and COCO Caption. We significantly outperform other approaches with at least +10 points on VQA v2, +9 points on GQA and we double the scores on COCO Caption. eP-ALM pt -L with OPT-6.7B and ViT-L gives the best scores while training only 0.06% of model parameters.

Note that for COCO Caption, other works give very low scores (thus we did not report them). Out of Distribution (OOD) Generalization: Do Parameter-Efficient Models Generalize Better? Here we investigate whether our parameter-efficient approach can perform well in OOD scenarios. To this end, we follow other approaches [START_REF] Agrawal | Rethinking evaluation practices in visual question answering: A case study on out-of-distribution generalization[END_REF] and train our model on the training set of a given benchmark, and evaluate it on the validation set of another benchmark, without multimodal pretraining. We measure the performance gap, i.e. the accuracy difference between a model trained on a different benchmark and the same model trained on the target benchmark. Tab.3 shows that eP-ALM, that trains 0.06% of total parameters, is very competitive in terms of OOD accuracy with other baselines, that train all model parameters and pretrain on large amount of data. Specifically, we outperform VILBERT on VQAv2 by more than 2 points. Interestingly, the OOD-IID gap for eP-ALM, is at least 2 times lower compared to ALBEF [56] and VilBERT [START_REF] Lu | Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF]. This reveals that our parameter-efficient approach generalizes relatively well in OOD scenarios.

Method

Video-Text Results

We investigate how much our approach generalizes to other modalities. To this end, we evaluate eP-ALM for Video QA on MSRVTT-QA [97] and MSVD-QA [97] and for Video Captioning on MSR-VTT [START_REF] Xu | Msr-vtt: A large video description dataset for bridging video and language[END_REF]. For the video encoding, we use the TimeSformer-base [6] model pretrained on Kinetics-600 [11]. We use 8 and 16 224x224 frames for VQA and captioning respectively.

Comparison to other work to the best of our knowledge, FrozenBiLM [START_REF] Yang | Zero-shot video question answering via frozen bidirectional language models[END_REF] is the only parameter-efficient work proposing to adapt LMs for video-language tasks. It uses Adapters to adapt the frozen CLIP-ViT and Bidirectional LM for Video QA. We compare our approach to our reimplementation of this baseline; where we train only the Adapters and the linear projection layer to project the last [CLS] token and prepend it to the input text ones. The results in Tab. 4 show that eP-ALM outperforms this baseline by a significant margin. The reason why the latter does not give good results might be due to prepending the visual tokens to the input of OPT. We can reduce the number of parameters and slightly degrade the performance by using a shared linear connection (eP-ALM vs eP-ALM pt ).

Zero-Shot Results

To explore the generalization of our approach, we evaluate on Zero-Shot for VideoQA, where the model is trained on a dataset different from the target one. Table 5 shows a comparison with other approaches. eP-ALM, trained on VQA v2 (standard split), outperforms other approaches trained on significantly more data. Specifically, eP-ALM outperforms Flamingo-3B [3] on MSRVTT-QA by more than 2 points, and attains double the scores of FrozenBiLM [START_REF] Yang | Zero-shot video question answering via frozen bidirectional language models[END_REF]. Contrary to some of other approaches that cast the task as classification (similarity-based) [START_REF] Yang | Just ask: Learning to answer questions from millions of narrated videos[END_REF] or constrained generation through masking, considering only a subset of answers (1k or 2k) [START_REF] Li | Lavender: Unifying video-language understanding as masked language modeling[END_REF][START_REF] Yang | Zero-shot video question answering via frozen bidirectional language models[END_REF][START_REF] Zellers | Merlot reserve: Neural script knowledge through vision and language and sound[END_REF], our approach is evaluated (with a character-wise comparison with the groundtruth) with unconstrained Open-ended Generation (OE Gen) and can generate answers with arbitrary lengths. This is more challenging and not in favor of our approach.

Audio-Text Results

We investigate the generalization of our approach to the audio domain. The encoder is AST-base model [START_REF] Gong | AST: Audio Spectrogram Transformer[END_REF] pretrained for classification on AudioSet [START_REF] Jort | Audio set: An ontology and humanlabeled dataset for audio events[END_REF]. We evaluate on AudioCaps dataset [48], the largest benchmark for Audio Captioning. We train with mel spectrograms of 128 bins and frequency and time masking with a batch size of 8.

To the best of our knowledge, no prior work has been proposed to efficiently adapt LM for audio-text tasks, thus we compare with other end-to-end trained SoTA that takes only the audio signal as input. Tab. 6 shows that our approach is very competitive with previous work, showing the potential of efficient adaptation of LM for the audio modality.

Comparison with SoTA

To contextualize the work, we compare eP-ALM to other SoTA that trains large number of parameters and most often with large-scale pretraining. Tab. 7 shows a comparison with both zero-shot (ZS) and Finetuning (FT) setups. The performance of eP-ALM is generally higher than ZS scores and still below FT ones. However, the performance gap with FT models, is smaller with the audio and video modalities.

Ablation Study

In this section, we ablate different component of our work.

Comparison between different text generation methods.

We find the approach sensitive to the text decoding strategy. In Tab. In the following, we run some ablations for Image-Text tasks, mostly on VQA v2.

Comparison with different variants and baselines. We start by comparing the different variants to other work in Fig. 3. All models use OPT-350M and ViT-B/16. Other approaches lag significantly behind our model. B M AGM A gives the best results (23.3% acc.) among them, followed by B P romptF use (18.82% acc.) and finally B LimBEr (10.75 % acc.). We also compare with another MAGMA baseline (B M AGM A * ) that prepends all visual tokens to the input, and we find a significant degradation compared to passing only the [CLS] token. This reveals that prepending all visual tokens directly to the input hinders the adaptation.

We can notice a consistent improvement of eP-ALM when adding more trainable parameters. The most parameterefficient model is eP-ALM lin which has 30.72%, while the best has 34.34% (with Adapters eP-ALM ada ). Interestingly, eP-ALM lin with only one linear layer succeeds to get good performance on this challenging setup, revealing that the language and visual representation spaces are not very far. Other parameter-efficient techniques such as Prompt Tuning can help to get additional points (30.72 with eP-ALM lin vs 31.27 with eP-ALM pt ). Moreover, using different layers for each injected [CLS] token seems to give significant improvement (31.27 with eP-ALM pt vs 33.08 with eP-ALM).

Finally, we show that eP-ALM surpasses the "full finetuning" baseline (grayed line) that finetune all parameters by 1.27 points (31.79 vs 33.08). This reveals that training all weights of pretrained models on small datasets can reduce their generalization capability and degrades performance. [CLS] tokens VQA v2

From ViT layers To OPT layers Test Acc. Injecting the [CLS] tokens lately (layers 12-24) in the OPT is better than injecting them in all layers or only in the input.

As a trade-off between performance and efficiency, we favor eP-ALM which we carry on for the following study.

Extraction and Injection Level of [CLS] Tokens.

Here we investigate which [CLS] tokens to extract from the ViT and where is the best position to inject them inside the OPT model. Table 9 shows that extracting the last [CLS] tokens (from the last 6 layers) is better than using only the last one, as done in other approaches (Acc 30.53 vs 33.08). In addition, using all [CLS] tokens seems to degrade the performance. Moreover, prepending [CLS] tokens to all OPT layers degrades slightly (33.08 vs 32.15), and prepending to the input of OPT gives the worst results. This might indicate that it is easier to merge visual and textual tokens deeper in the model, where the representations are more abstract, compared to the first layers where we have more modalityspecific features and higher representation mismatch. Scaling LM. An interesting question that we investigate is the impact of scaling the language model's parameters on our approach. Ideally, we would like to have an approach that efficiently exploits LLMs for other tasks and modalities, without having access to enormous computational resources. In Table 4, we show that the scores increase with the model size with the biggest jump being between OPT-350M (33.08 vs 37.29) and OPT-1.3B (∼ ×4 the model size). The consistent improvement with scale shows the effectiveness of the approach when considering very big models.

As a trade-off between performance and model size, we favor OPT-2.7B and use it for all other experiments. Scaling Visual Model. We also study how the model behaves when scaling the visual encoder. In Figure 5, we can notice that the scores increase with the size of the ViT. The best is ViT-L/16 (41.36) and the worst is ViT-S/16 (Acc 38.73). However, the ViT resolution or the number of image patches/tokens does not seem to have a significant effect on the final performance after a resolution of 16.

Scaling Compute. Table 10 shows that our approach scales with compute, as training for more epochs leads to 4 points gain in VQA accuracy. Interestingly with OPT-6.7B and ViT-L (eP-ALM pt -L), we achieve a score of 43.6 by training only 0.06% of model parameters (∼4M params). Qualitative Results. We show some qualitative results of our eP-ALM model with OPT-2.7B in Fig. 6. For VQA, we can notice that our model is able to correctly answer the questions. Moreover, some of the answers are richer and more accurate than the manually labeled ground truth in the dataset. This also reveals that the exact matching evaluation protocol is not in favor of the open-ended generation produced by our model. Interestingly, it seems that the model learned the answering style in the training set (i.e, short and concise answers). For Captioning, the model can generate coherent sentences describing the image globally. However, it still misses some details in the image.

Conclusion

In this work, we propose a new challenging setup to efficiently adapt unimodal models for multimodal tasks, which is centered around augmenting existing LMs with perception. Without multimodal pretraining, and with almost 4M trainable parameters consisting of a linear connection and a Soft Prompt, we can adapt a frozen 7B model and reach an accuracy of 54.5% on VQA v2, with unconstrained openended generation. We validate the effectiveness of the approach with Images, Video, and Audio modalities. This direct finetuning setup has several advantages; (a) training data/compute efficiency, (b) attains generally higher performance than pretrain-zeroshot setup, (c) easy to adapt to new tasks, modalities or other LLMs, where no costly pretraining is needed. However, the mechanism proposed in eP-ALM can be adapted in a straightforward manner to this setup.

Even though the results are still far from the state-of-theart approaches that train most of the model parameters on much more data, the extremely small percentage of trainable parameters (0.06%) and the increasing scores with model size and compute make the work promising towards finding an intermediate point, between extremely efficient and extremely inefficient approaches, which is hopefully closer to the former.

The method has some limitations, which we illustrate in the appendix. In general, the model struggles to capture fine-grained details in the images, favors coherent generation over factual one, might hallucinate some objects not present in the image, and lacks common sense reasoning. Our approach inherits most of the limitations and biases of pretrained models, especially the LM, and training only a few adaptation parameters does not seem to avoid the transfer of these biases. Finally, the model is trained with next token prediction and is able to produce coherent text, however, it is still not clear how this paradigm can lead to real reasoning capabilities. Video Captioning: we sample randomly 16 frames of resolution 224x224 for each video and train for 25 epochs with a batch size of 64.

Audio Captioning we train for 30 epochs with frequency and time masking of 24 and 96 respectively. The mel bins is 128 and the audio length is 1024. Batch size 32. For Deep Prompt, we inject new soft prompts in all 32 blocks of OPT (each with length 10).

B. eP-ALM Variants

We detail the different variants proposed in this paper (here we consider ViT-B/16 and OPT-350M for simplicity). These variants are illustrated in Fig. 7: eP-ALM lin : we extract the [CLS] tokens from the last 6 layers of the frozen ViT and inject them in the last 12 layers of the frozen OPT. To reduce inference cost, for each couple of layers (here 2), we replace the previous [CLS] with the new one (thus only increasing the number of tokens by 1 the whole process). All visual [CLS] tokens are projected by one trainable linear projection layer (shared) to fit their dimension to that of the OPT.

eP-ALM pt : we augment eP-ALM lin with Prompt Tuning, which consists of prepending trainable tokens (i.e, soft prompt) to the input of the LM. This might help the model to adapt well to the new task by providing context to the text input. For the sake of efficiency, we prepend only 10 learnable tokens. eP-ALM: while one linear projection is appealing, it might not be able to capture all the particularity of different [CLS] tokens. To overcome this, we use different projections for each [CLS], while keeping the soft prompt. eP-ALM ada : another alternative to Prompt Tuning are Adapters. We follow other approaches [23] and add sequentially one adapter module (downsample, activation then upsample) after self-attention and feedforward layers in all the blocks of OPT. While this might give better results, it adds a significant number of trainable parameters.

C. Pretrain-Zeroshot Setup

The focus of this work is on direct finetuning, where we propose an efficient cross-modal interaction mechanism with low data regime. However, the proposed mechanism can be adapted in a straightforward manner to the pretrain-zeroshot evalution setup. In this section, we show the effecitveness of eP-ALM with zero-shot evaluation after pretraining on CC3M. Specifically, we pretrain eP-ALM pt -L on CC3M for 4 epochs (which takes 35hours on 2 gpus V10032GB), and evalute on COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] and NoCaps (all) [2] datasets. We experiment with ViT-L, initialized from ImageNet and CLIP.

Tab. 11 show a comparison with other approaches. Without using CLIP encoder, eP-ALM significantly outperforms other work on both datasets. Using CLIP (which is trained to produce visual features aligned to text) reduces the improvement gap, where eP-ALM still outperforms all baselines on B@4 and METEOR metrics. This validates that in case of unaligned visual encoder (e.g., pretrained on ImageNet) our cross-modal interaction mechanism is efficient to align both modalities.

Note that, eP-ALM is significantly more efficient than LimBEr and MAGMA, as they train more parameters for very long time (∼> 670 GPUhs). As MAGMA is trained on a lot more data, we compare with MAGMA trained on CC3M obtained from [START_REF] Merullo | Linearly mapping from image to text space[END_REF].

D. Ablation Study

Here we present an additional ablation study.

D.1. Image-Text

Training All Parameters Here we investigate how much gain we can obtain by unfreezing the pretrained models. We experiment on VQA v2 with eP-ALM. Table 12 shows that finetuning the pretrained models in our eP-ALM gives slight improvement, despite a large number of trainable parameters. Note that, we find that using a very small learning rate (lr=1e-7) is the only option (while keeping an lr of 1e-5 for the connectors) to unfreeze these models without significant degradation.

D.2. Video-Text

Video Encoder: here we compare different encoders to process the videos. We compare the TimeSformer [6] that has both spatial and temporal attention and trained for video classification with a simple baseline, ViT trained on Ima-geNet, that ignores the temporal dynamics. For ViT, we take the average of [CLS] tokens of the processed frames while for TimeSformer we consider the single [CLS] token. Table 13 shows that using video-specific encoders gives significantly better results for video captioning. In addition, we find that using 16 frames instead of 8 gives slight improvement.

Injection and Extraction level of [CLS] tokens: here we show the importance of leveraging the hierarchical representation in both the video encoder and language model. Table 14 shows the results on MSVD-QA. We show that keeping the interaction between cross-modal tokens to the last layers

D.3. Audio-Text

Comparison with different variants. Here we compare different variants of our approach to different baselines for audio captioning. We evaluate on AudioCaps dataset [48], the largest benchmark for Audio Captioning. We train with mel spectrograms of 128 bins and frequency and time masking with a batch size of 8.

To the best of our knowledge, no prior work has been proposed to adapt LM for audio-text tasks. However, there is some recent work adapting audio models using parameterefficient techniques, such as Deep Prompts and Adapters [START_REF] Kim | Integrated parameter-efficient tuning for general-purpose audio models[END_REF]. Tab. 15 shows a comparison with different approaches. We find that prepending the audio tokens to the input of OPT does not give reasonable performance. To investigate this more, we train another baseline where the audio tokens are concatenated in the last 12 layers of OPT (eP-ALM l19-31+Adapter and eP-ALM l19-31+DeepP T ). This leads to significant improvement. Time and Frequency Masking: following other approaches [START_REF] Gong | AST: Audio Spectrogram Transformer[END_REF]73] we train eP-ALM with time and frequency masking on AudioCaps. Table 16 shows that masking significantly helps, however, using too much masking hurt the performance.

Masking Window AudioCaps

E. Limitations

Even though we show appealing results for very efficient training, the method has several limitations, which we illustrate in Fig. 8. For VQA, we can notice that the model is unable to capture fine-grained details in the images (e.g., number of colors and the zebra in the first 2 examples), which might be due to constraining the interaction with the vision model through the [CLS] tokens, that generally capture global information about the image. In the case of hard questions, the model favors a coherent generation of a relevant question followed by its correct answer, instead of answering the main question ("A: what color is the phone?? black" in example 3).

For Captioning, the model seems to favor outputting a coherent sentence, even though it is not entirely correct ("many" cows in a "crowded" city). Secondly, the model might hallucinate some objects that do not appear in the image ("apples" in example 2). Finally, the model lacks common sense reasoning, making him unable to understand that elephants are not small, and being far from the camera does not change this fact (example 3).

Our approach inherits most of the limitations and biases of pretrained models, especially the LM, and training only a few adaptation parameters does not seem to avoid the transfer of these biases. Finally, the model is trained with next token prediction and is able to produce coherent text, however, it is still not clear how this paradigm can lead to real reasoning capabilities.

Figure 3 .

 3 Figure 3. Comparison with other baselines on VQA v2. eP-ALM uses ViT-B/16 and OPT-350M. Our approach significantly outperforms other approaches. eP-ALM already surpasses the dashed "upper" baseline that trains all model parameters. Allocating more adaptation parameters help to increase the scores. Marker size indicates model size.

Figure 4 .

 4 Figure 4. Scaling LM; we scale the OPT from 125M to 6.7B parameter. eP-ALM becomes more effective with scale. The biggest performance jump is when scaling beyond 1B parameters (1.3B).

Figure 5 .

 5 Figure 5. Scaling Vision Model; the score increases with the size of the ViT. Increasing the patch resolution beyond 16 does not help.

Caption: a veryFigure 6 .

 6 Figure 6. Qualitative results of eP-ALM: the model is able to generate accurate answers and coherent descriptions of the image.

Figure 7 .

 7 Figure 7. Illustration of the different variants of eP-ALM; eP-ALM lin is the most efficient variant that only trains the linear projection layer, eP-ALMpt adds trainable Soft Prompts (i.e Prompt Tuning), and eP-ALM ada replaces the Soft Prompt in eP-ALM (last figure) with trainable Adapters. All models extract the [CLS] tokens from the last layers of ViT and prepend/replace them in the last layers of OPT.

Perceptual Encoder (E) OPT Image or Audio or Video Text Transformer Block Transformer Block Transformer Block Language Model (LM) Linear Connection (C): Single, Multiple, Shared

  Illustration of the adaptation mechanism in eP-ALM. The perceptual input (image/video/audio) is fed to the perceptual encoder E (e.g., ViT) and the corresponding text to the LM (e.g., OPT), which then generates a text conditioned on the perceptual input. The multimodal interaction is done via the [CLS] tokens acting as Perceptual Prompt, and are extracted from the last layers of the encoder, then injected in the last layers of LM, after passing by the Linear Connection C. The previous [CLS] token is replaced by the new one coming from a deeper layer, keeping the number of tokens fixed. The first layers (grayed) of each model are kept intact without any modality interaction. We ease the adaptation with a Soft Prompt that is prepended to the input of LM.

	Transformer Block
	Transformer Block
		[CLS]	Soft
	Transformer Block	Token	Prompt
		Perceptual	Perceptual
		Token	Prompt
			Textual
			Token
		Trained	Frozen
			Text
	Figure 2. authors released a family of models with different scales,
	starting from 125M up to 175B model size. Besides being
	open source and trained on English data, the different model
	sizes allow us to readily investigate the effect of scale, and
	help to devise new approaches with affordable model sizes.
	Perceptual Encoders We favor only unimodal models.
	For images, we use the vanilla ViT model [21] which consists
	of Self Attention and FeedForward layers and is pretrained
	for image classification on ImageNet

Table 1 .

 1 Comparison with other work after direct finetuning on VQA v2, GQA, and COCO Caption. eP-ALM significantly outperforms other approaches. eP-ALM uses ViT-B/16 and OPT-2.7B. eP-ALM-L uses OPT-6.7B and ViT-L/16. †: use standard split. on the validation set. Table2, shows the superiority of our approach over other baselines. Interestingly, we can achieve 80% (41.9 vs 52.77) of the performance when training on 1% of the data. This validates the approach on low resources scenarios and shows that, in addition to being parameter-efficient, our model is also data-efficient.

		VQA v2			GQA		COCO
		Val	Test	Val	Test	B@4 CIDEr
	PromptFuse † [60]	34.1 †	-	-	-	-	-
	BLimBEr	34.1	33.5	30.81 29.4	-	-
	BP romptF use	40.4	39.5	33.74 31.51	15.05 48.26
	BMAGMA	32.2	31.8	30.98 28.93	-	-
	eP-ALMpt	48.8	47.8	43.8 40.3	27.52 91.92
	eP-ALM	50.7/53.3 †	50.2	45.0 40.4	29.47 97.22
	eP-ALMpt-L *	54.58/54.47 † 54.47	46.86 42.7	31.24 107.0

* : trained more than 8 epochs.

Few-shot Results: Are Parameter-Efficient Models also Data-Efficient? In this section, we investigate how dataefficient our model can be. To this end, we train on a very small portion (randomly sampled) from the VQA training set and evaluate

Table 2 .

 2 Few-shot Results on VQA v2 validation set (standard split).

* : longer training.

Table 3 .

 3 Out

	Method	Multimodal Trained	Train data	Test data	Gap
		PT data param. (%)		VQA v2 GQA
	ALBEF [1]	14M	100%	VQA v2 GQA	-50.3	50.1 -	-21.8 -14.1
	VILBERT [1]	3M	100%	VQA v2 GQA	-41.8	42.6 -	-20.4 -22.7
	eP-ALMpt-L	0	0.06%	VQA v2 GQA	-45.19	41.39 -9.59 --5.8

-Of-Distribution Generalization on GQA and VQA v2 (standard split). The Gap shows the performance degradation when the model is trained on a different dataset.

Table 4 .

 4 Comparison

	Method	Trained	MSVD-QA MSRVTT-QA	MSRVTT
		param (%)	Test	Test	CIDEr B@4
	BF rozenBiLM (from [101])	3.72 %	14.58	6.33	-	-
	eP-ALM	0.89 %	38.64	36.16	47.31 38.51
	eP-ALMpt	0.54 %	38.79	35.62	45.30 39.34

with different approaches after direct finetuning on MSVD-QA, MSRVTT-QA, and MSRVTT Caption.

Table 5 .

 5 Zero-Shot results on Video QA. OE Gen: unconstrained Open-Ended Generation. † evaluated on questions with top 1k answers.

	Method	Training data	Train. Param. (%) OE Gen MSRVTT-QA MSVD-QA
	JustAsk [100]	ActivityNet-QA	89.6%	✗	2.7	-
	JustAsk [100]	HowToVQA69M	89.6%	✗	2.9	7.5
	LAVENDER [57]	WebVid2.5M+CC3M	100%	✗	4.5	11.6
	MERLOT Reserve [105]	YT-Temporal-1B	100%	✗	5.8	-
	FrozenBiLM † [101]	400M-CLIP + VQA v2	2.9%	✗	6.9	12.6
	Flamingo 3B [3]	M3W+ALIGN+VTP	40%	✓	11.0	27.5
	eP-ALM	VQA v2	0.9%	✓	13.17	24.82
	eP-ALM †	VQA v2	0.9%	✓	14.54	27.09

Table 6 .

 6 Comparison with other work for Audio Captioning on AudioCaps

	Method	Trained		AudioCaps
		param (%)	B@1 B@2 METEOR CIDEr SPICE SPIDEr
	Kim et al. [49]	100%	0.614 0.446	0.203	0.593	0.144	0.369
	Koizumi et al. [53]	100%	0.638 0.458	0.199	0.603	0.139	0.371
	Eren et al. [24]	100%	0.710 0.490	0.290	0.750	-	-
	Xu et al. [99]	100%	0.655 0.476	0.229	0.660	0.168	0.414
	Mei et al. [68]	100%	0.647 0.488	0.222	0.679	0.160	0.420
	Gontier et al. [32]	100%	0.699 0.523	0.241	0.753	0.176	0.465
	Liu et al. [63]	100%	0.671 0.498	0.232	0.667	0.172	0.420
	eP-ALM	0.90 %	66.08 47.57	22.69	63.61	16.29	-
	Test set.

  8, we compare with different text decoding methods; multinomial/random sampling [25] and greedy decoding

	Dataset (Metric)	SoTA (ZS)	eP-ALM (FT)	SoTA (FT)
	AudioCaps (CIDEr)	-	63.6	66.7 (Liu et al. [63])
	MSRVTT-QA (Acc)	17.4 (Flamingo80B [3])	36.7	44.1 (OmniVL [92])
	MSR-VTT (CIDEr)	-	50.7	60 (MV-GPT [77])
	COCO (CIDEr)	84.3 (Flamingo80B [3])	107.0	145.3 (OFA [93])
	VQAv2 (Acc)	56.3 (Flamingo80B [3])	53.3	84.3 (PaLI [15])
	GQA (Acc)	29.3 (FewVLM [45])	42.7	60.8 (VL-T5 [18])

Table 7 .

 7 Comparison of eP-ALM with text generation-based SoTA that train significant number of parameters, including methods with large-scale pretraining. Best and next best scores are bolded and underlined respectively. FT: Finetuning. ZS: Zero-shot.

	with beam-search (1 to 5 beams). Greedy decoding signifi-
	cantly outperform multinomial sampling, and increasing the
	number of beams leads to additional improvements, to the
	detriment of increasing inference cost.
	Decoding Method # of beams	VQA v2 GQA Test Test B@4 CIDEr COCO	MSVD-QA MSRVTT-QA MSRVTT Test Test B@4 CIDEr
	Multinomial	1	43.86 37.02 13.37 58.60	31.18	26.96	20.77 14.13
	Greedy	1	54.47 42.70 31.24 107.0	38.64	36.16	47.31 38.51
	Greedy	3	54.90 42.99 33.35 113.0	38.93	36.66	50.66 39.02
	Greedy	5	54.92 42.95 33.86 115.54	-	-	-	-

Table 8 .

 8 Comparison with different text generation mechanisms using eP-ALM-L.

Table 9 .

 9 Ablation

	12	12 to 23	30.53
	6 to 12	12 to 23	33.08
	1 to 12	12 to 23	31.17
	6 to 12	1 to 23	32.15
	12	1	18.82

study. Extracting the [CLS] tokens from the last layers of ViT (layers 6 to 12) is better than taking only the last token (layer 12).

Table 10 .

 10 Scaling Compute. Evaluation on VQA v2 standard split.

	Method	number of VQA v2 epochs Val
	eP-ALM	8	38.9
	eP-ALM	32	42.9
	eP-ALMpt-L	8	42.5
	eP-ALMpt-L	32	43.6

Table 11 .

 11 Zero-shot comparison on COCO and NoCaps, after pretraining on CC3M (2.84M examples).

	Method			Trainable params. CLIP Enc.	COCO B@4 CIDEr METEOR B@4 CIDEr METEOR NoCaps
	MAGMA (NFRN) [23]		243M	✗	8.2	22.4	-	4.5	20.9	-
	LimBEr (NFRN) [69]		8.4M	✗	-	36.2	-	-	28.5	-
	eP-ALMpt-L			4.2M	✗	11.50 42.47	15.44	12.53 36.79	15.50
	MAGMA (on CC3M from [69])	243M	✓	9.7	47.5	14.6	-	38.7	-
	LimBEr [69]			12.5M	✓	12.6	54.9	16.1	-	43.9	-
	FROMAGe [52]			4.2M	✓	9.65	-	11.53	-	-	-
	eP-ALMpt-L			4.2M	✓	12.97 51.29	16.23	13.30 39.5	15.55
	Trainable Models LM VQA v2
	VM	LM	size test Acc.
	✗	✗	350M	33.08
	✗	✓	350M	35.44
	✓	✓	350M	35.47

Table 12 .

 12 Ablation study: we study how much gain we can obtain by also training the pretrained vision and language models. We see slight improvement by training the pretrained models.

Table 13 .

 13 Ablation (Caption) MSRVTT Caption.(layers 19 to 31) of the OPT leads to significantly better results. Extracting several tokens from different tokens of the TimeSformer gives slight improvement. However, using hierarchical video transformers [59, 64] might lead to better results. We noticed also that Adapters generally give better results than Prompt Tuning, this might be because when training on videos we sample randomly some frames, which prevents the model to overfit in the case of small datasets.

	Method	MSRVTT
			CIDEr B@4
	ViT-B Avg.	17.96 12.77
	ViT-B Avg. (16 f)	17.82 12.85
	TimeSformer	20.11 13.53
	TimeSformer (16 f)	20.58 14.12
	Adaptation	[CLS] tokens	MSVD-QA
	approach	from encoder layers to OPT layers	test Acc.
		12	1	13.49
	Soft Prompt	12 12	1 to 31 19 to 31	27.16 30.86
		6 to 12	19 to 31	31.18
		12	1	12.40
	Adapters	12	1 to 31	34.86
		12	19 to 31	35.94

Table 14 .

 14 Ablation study: we investigate the extraction and injection position of [CLS] tokens for Video QA.

Table 15 .

 15 Comparison with other work for Audio Captioning on Audio-Caps Test set.

	Method	Trained	AudioCaps
		param (%)	CIDEr B@4
	BAdapter	3.76 %	2.96	-
	eP-ALMl19-31+Adapter	3.76 %	31.17	8.09
	eP-ALMl19-31+DeepP T	0.93 %	32.57 10.66
	eP-ALMpt	0.55 %	35.17 10.73
	eP-ALM	0.90 %	37.14 11.37

Table 16 .

 16 Ablation Study: time and frequency masking help for Audio Captioning.

		Frequency	CIDEr B@4
	256	64	33.94 10.21
	192	48	35.67 10.40
	96	24	37.14 11.37
	0	0	36.01 10.23

Acknowledgments

This work was partly supported by ANR grant VISA DEEP (ANR-20-CHIA-0022), and HPC resources of IDRIS under the allocation 2022-[AD011013415] and 2023-[AD011013415R1] made by GENCI. The authors would like to thank Theophane Vallaeys for fruitful discussion.

Appendix

The appendix is organized as follows; in Sec. A, we give more implementation details about the experiments that we conduct. We illustrate and explain the different variants of eP-ALM in Sec. B. We compare eP-ALM to other approaches following the pretrain-zeroshot setup (Sec.C). We then present more ablation studies on image-text and video-text tasks in Sec. D. Finally in Sec. E we show some qualitative results and discuss the limitation of the proposed approach.

A. Implementation Details

We use OPT-2.7B in our final model. We extract the [CLS] tokens of the last 6 layers of perceptual encoders and prepend them, after linear projection, to the text tokens of the last 12 layers of the OPT. Note that we replace the previous [CLS] with the new one to keep the same number of tokens. We finetune with the classical cross-entropy loss used to train the original OPT for VQA and Captioning. We use the AdamW optimizer with a learning rate of 1e-5 warmed up to 2e-5 then decreased to 1e-6 using a cosine scheduler. For Adapters, we use sequential Adapters after self-attentions and feedforward layers with a downsampling factor of 8 and ReLU activation. For Soft Prompt, we implement it as a linear embedding layer that takes numbers from 0 to the length of the prompt (here 10). We experiment also with adding an MLP after the prompts as done with other approaches [START_REF] Sung | Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks[END_REF]. We use the prompt with MLP for most of the experiments as we find that it gives slightly better results. The soft prompt and adapters are trained with a fixed lr of 1e-5. eP-ALM pt -L is trained with a light-weight prompt (only trainable tokens without MLP), starting learning rate of 2e-4 and a fixed learning rate of 1e-3 for the prompt with a total batch size of 16.

VQA/GQA: we use a special token for VQA ( ′ < /a > ′ ) to separate the question from the answer. We train for 8 epochs with a batch size of 64 (128 for GQA) and an image resolution of 224. Training our approach with OPT-2.7B for VQA v2 can be done on a single V100 GPU 32GB for 1.8 days (as the perceptual encoder is frozen, saving its output tokens can save a lot of training time). For Few-shot experiments, we train longer (for 64 epochs) with a higher starting learning rate (1e-4 warmed up to 2e-4 and decreased to 1e-5). Those marked by a * are trained for 100 epochs as in PromptFuse [START_REF] Liang | Modular and parameter-efficient multimodal fusion with prompting[END_REF].

Image Captioning we train for 8 epochs with a batch size of 64 and an image resolution of 224.

Video QA: we sample randomly 8 frames of resolution 224x224 for each video and train for 25 epochs with a batch size of 32. For Zero-Shot experiments, we train only for 4 epochs with starting learning rate of 1e-4. We use only the spatial self-attention of TimeSformer to train on VQA v2.