Canhuang Luo

Wei Chen

Rufin Vanrullen

Ye Zhang

Carl Michael Gaspar
email: carl.gaspar@zu.ac.ae

Carl Michael Gaspar

Nudging the N170 forward with prior stimulation -Bridging the gap between N170 and recognition potential

Keywords: object recognition, visual masking, Chinese characters, ERP, N170, recognition potential

HAL is

Introduction

Event related potential (ERP) components are important for inferring the time course of perceptual and cognitive processing [START_REF] Carreiras | The what, when, where, and how of visual word recognition[END_REF][START_REF] Dien | The neurocognitive basis of reading single words as seen through early latency ERPs: A model of converging pathways[END_REF][START_REF] Grainger | Watching the Word Go by: On the Time-course of Component Processes in Visual Word Recognition[END_REF]. The most typical type of component, the kind studied in this paper, is a peak in the evoked response, which can be either a positive or negative deflection. Inference about processing can be relatively straightforward when components are classified almost exclusively by the stimulus conditions leading up to that component. In this case, delays and distortions in the component are interpreted as delays and distortions in processing. This is certainly the case with components like the readiness potential [START_REF] Libet | Time of Conscious Intention to Act in Relation to Onset of Cerebral Activity (Readiness-Potential)[END_REF] and the mismatch negativity (MMN) [START_REF] Näätänen | Early selective-attention effect on evoked potential reinterpreted[END_REF].

The mismatch negativity (MMN) is a response evoked by an unexpected stimulus in a sequence of regularly occurring stimuli. Importantly, an early MMN evoked by auditory stimuli is just as much an MMN as a late MMN evoked by visual stimuli [START_REF] Winkler | Preattentive binding of auditory and visual stimulus features[END_REF] However, confusion can arise when components are also classified by their morphology -their latency and shape. The importance of latency in defining some components is often reflected in component nomenclature like N170 [START_REF] Bentin | Electrophysiological studies of face perception in humans[END_REF], which refers to a negative deflection occurring around 170 ms after an object from a familiar category appears. In some cases, peaks that are offset in time (i.e. with longer latencies in most cases) and slightly different in shape are thought to reflect different processes.

While that interpretation is consistent with a cascade across processing levels, an equally valid interpretation of peak offset is a delay in processing. How can we decide between these two interpretations? This paper reports unique support for the delay interpretation in a comparison between 2 nominally different ERP components: the N170 and the recognition potential, or RP [START_REF] Rudell | The Recognition Potential Contrasted with the P300[END_REF][START_REF] Martin-Loeches | An early electrophysiological sign of semantic processing in basal extrastriate areas[END_REF]. The approach we take arises from the possibility that the presence of forward-masking during typical recordings of RP can lead to processing delay relative to the N170, which rarely involves forward masks. Masks are typically random patterns and so their mere presence is unlikely to involve a dramatic switch in processing. In order to test our delay-hypothesis we simply nudge the N170 into an evoked response that is morphologically equivalent to an RP, by using forward masks with progressively smaller ISI.

Importantly, the experimental conditions at the extreme ends of our parametric manipulation match the conventional experimental designs of N170 and RP studies. The remainder of the Introduction describes N170 and RP in greater detail and motivates their comparison despite being studied in different literatures with very little cross talk. Experimental design is elaborated upon and more precise predictions are made in the context of the delay hypothesis.

The N170 is the first negative deflection in response to visually observed objects from a familiar category, like faces. It is described as being a critical temporal marker for the extraction of information from visually observed familiar objects [START_REF] Martín-Loeches | The gate for reading: Reflections on the recognition potential[END_REF][START_REF] Maurer | Left-lateralized N170 Effects of Visual Expertise in Reading: Evidence from Japanese Syllabic and Logographic Scripts[END_REF] (which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. [START_REF] Maurer | Left-lateralized N170 Effects of Visual Expertise in Reading: Evidence from Japanese Syllabic and Logographic Scripts[END_REF][START_REF] Tanaka | A Neural Basis for Expert Object Recognition[END_REF]. However, the latency of N170 can be influenced by various factors. For example, up-side down faces evoke an N170 that is delayed by about 10 ms, compared to that evoked by upright faces [START_REF] Rossion | The N170 occipito-temporal component is delayed to inverted faces but not to inverted objects: Electrophysiological evidence of face-specific processes in the human brain[END_REF]. Another possible factor, which to our best knowlege has not been directly investigated, is inter-stimulus intervals (ISI). In a survey we made (Figure 1) , one can see a monotonic relationship between N170 latency and ISI [START_REF] Cao | Rapid adaptation effect of N170 for printed words[END_REF][START_REF] Dimigen | Trans-saccadic parafoveal preview benefits in fluent reading: A study with fixation-related brain potentials[END_REF][START_REF] Fu | Neural adaptation provides evidence for categorical differences in processing of faces and Chinese characters: An ERP study of the N170[END_REF][START_REF] Harris | Rapid face-selective adaptation of an early extrastriate component in MEG[END_REF][START_REF] Kuehl | Exploring the time course of N170 repetition suppression: A preliminary study[END_REF][START_REF] Lin | Left-lateralized N170 response to unpronounceable pseudo but not false Chinese characters-the key role of orthography[END_REF][START_REF] Martens | Masked and unmasked electrophysiological repetition effects of famous faces[END_REF][START_REF] Shen | Face identity is encoded in the duration of N170 adaptation[END_REF][START_REF] Su | Taking a Radical Position: Evidence for Position-Specific Radical Representations in Chinese Character Recognition Using Masked Priming ERP[END_REF]. In fact, a very short ISI can delay the N170 up to ~80 ms later than the typical latency, which is an enormous effect compared to the face inversion effect. Moreover, if a study uses very short ISI between forward masks and targets (for example, [START_REF] Su | Taking a Radical Position: Evidence for Position-Specific Radical Representations in Chinese Character Recognition Using Masked Priming ERP[END_REF]; studies marked with orange color in Figure 1), we can see that not only is the latency delayed, but also the waveform is wider and deformed enough to make peak isolation challenging. Taken together, the results drawn from these separate studies suggest that the N170 can be nudged forward and widened by the recency of prior stimulation. The most extreme delay should occur with zero ISI (e.g., [START_REF] Su | Taking a Radical Position: Evidence for Position-Specific Radical Representations in Chinese Character Recognition Using Masked Priming ERP[END_REF], at which point the N170's morphology might be indistinguishable with another ERP component: the recognition potential (RP).

RP is usually studied in a totally separate body of literature focusing on the time course of word recognition [START_REF] Dien | The neurocognitive basis of reading single words as seen through early latency ERPs: A model of converging pathways[END_REF][START_REF] Martín-Loeches | The gate for reading: Reflections on the recognition potential[END_REF]. Unlike the N170, the RP is usually measured with a paradigm called rapid stream stimulation (RSS) which presents multiple masks before a target in succession and multiple masks after. However, RP is similar to N170 in many other aspects. Primarily, both the RP and N170 are responses to the visual presentation of familiar objects. And there are also a number of morphological similarities between these two ERPs. Both of them are almost always N1 components (for an non-N1 RP, see [START_REF] Rudell | The Recognition Potential Contrasted with the P300[END_REF]2010). Both ERPs are observed at similar channels and have similar topographies (S. [START_REF] Bentin | ERP Manifestations of Processing Printed Words at Different Psycholinguistic Levels: Time Course and Scalp Distribution[END_REF][START_REF] Hinojosa | Common basal extrastriate areas for the semantic processing of words and pictures[END_REF][START_REF] Martín-Loeches | Functional differences in the semantic processing of concrete and abstract words[END_REF][START_REF] Martin-Loeches | An early electrophysiological sign of semantic processing in basal extrastriate areas[END_REF][START_REF] Maurer | Category specificity in early perception: face and word N170 responses differ in both lateralization and habituation properties[END_REF][START_REF] Maurer | Fast, visual specialization for reading in English revealed by the topography of the N170 ERP response[END_REF][START_REF] Maurer | Left-lateralized N170 Effects of Visual Expertise in Reading: Evidence from Japanese Syllabic and Logographic Scripts[END_REF]. The one major morphological difference between N170 and RP is delay: the RP typically occurs around 250 ms post-stimulus, which is longer than the typical latency of N170. And because of the delay, RP is often linked to higher level processing stages than the processing stage revealed by N170 [START_REF] Dien | The neurocognitive basis of reading single words as seen through early latency ERPs: A model of converging pathways[END_REF]. However, this widely-accepted interpretation of RP does not consider the presence of forward-masking during typical recordings of RP. If the forward mask merely delays the brain signal in response to the target, could the RP just be a delayed brain response such that its mask-free latency would actually be ~170 ms? To put it simply, is RP actually an N170 delayed by forward masks?

Taken together, our study aims to answer two questions: 1) Is the N1 systematically delayed and warped by ISI? 2) Is RP a delayed version of N170? Answers to these questions can only be obtained by establishing the relationship between ISI and N170. In this study, we measure electrophysiological responses evoked by rapidly displayed Chinese characters, and we focus mostly on characteristics of the N1 and vertex positive potential (VPP). VPP is measured at a vertex channel such as Cz and is considered to reflect the same process the N170 does [START_REF] Joyce | The face-sensitive N170 and VPP components manifest the same brain processes: The effect of reference electrode site[END_REF]. Our subjects also performed a 1-back task on Chinese characters, and we measured their reaction times. In these respects, our study is like many others in both the N170 and RP literatures. However, our study is unique in three ways: First, across 14 conditions, we parametrically manipulate the temporal proximity between the target and a forward-mask image, from 0-gap (quasi-RP condition; gap denotes ISI) to infinity-gap (or nomask, the typical N170 condition). Second, we focus on the specification of waveformmorphology in order to determine whether the N1 component obtained in N170 and gap conditions is the same. Finally, we obtain lots of data for each subject (150 trials for each of the 14 conditions). Altogether, we are able to obtain unique "gap-metric" functions, the ERPcomponent equivalent of a psychometric function for 3 major aspects of N1 morphology: N1 latency, N1 single-trial jitter, and topographical similarity to the N170. In addition, reaction times measured across gap conditions also allows us to assess the functional consequences of gap-related waveform changes. Our experimental design allows us to isolate the effect of a single masking stimulus on variability in N1 morphology. All these allow us to answer the first question and part of the second question. Considering that applying only one forward mask per trial is not typical in the majority of RP studies (but see: Dien, Brian, Molfese & Gold, 2013), to completely answer the second question, typical RP was also measured with the RSS paradigm as a control condition and compared with all the other 14 conditions. These studies provide more limited results than the adaptation studies shown above the x axis, because: they either have only a single ISI, they do not report N170 latencies, or those latencies are difficult to estimate from the figures because the grand-average waveforms are complex. The study marked in green is an eye-tracking & EEG study, the average interval between reading fixations during a combined EEG and eye-tracking study on reading can be consider like an "ISI".

Materials and methods

Participants

Nine healthy adult participants (all right-handed, age ranging from 24 to 27, 4 males) with normal or corrected-to-normal vision took part in the study and were paid for their participation.

Data from two participants were rejected due to too many artifact-contaminated trials (e.g., eye blinks, head movements and α wave). Valid data from seven participants were reported here.

All participants were native Chinese who had more than 10 years of education in speaking and reading Chinese characters. Written informed consent was obtained from each participant prior to the experiment.

Stimuli

Procedure

After electrode-cap placement, participants were seated in a dim light and soundattenuated shielded room, at a viewing distance of 60 cm from the computer monitor. Stimuli were presented with MATLAB (MathWorks, Natick, Mass) using the Psychophysics Toolbox (Version 3.0.12; http://psychtoolbox.org) [START_REF] Brainard | The Psychophysics Toolbox[END_REF][START_REF] Kleiner | What's new in psychtoolbox-3[END_REF] on a light gray background.

The sequence of events for each trial, summarised in Figure 2, is as follows: In each trial, a "+" fixation was presented at first in the center of the screen for 500 ms, which was followed by a blank for 800 ms. Then, a forward mask, a character, and a backward mask were presented orderly, each for 250 ms. Each trial was ended by a blank of 800 ms. The masks in each trial were randomly chosen from all the masks. Participants were instructed to make a response by pressing the 'm' key as quickly and accurately as possible when a repeated character was detected in successive trials. The ISI (i.e., temporal gap) between the preceding mask and the character varied from 0 to 600 ms with a step of 50 ms. So totally, there were 13 gap conditions.

Additionally, there were two more conditions, one was without the preceding mask in order to elicit the classic N170 (no-mask condition), in which the blank interval between the fixation and the character was 1050 ms ; for the other one the RSS paradigm was used, in which 3~5 masks were presented prior to the targets and 4~6 masks were presented after to elicit typical different conditions are described in Figure 2. All 15 conditions were separated by blocks; each block had only one condition. There were 58 trials in each block where 8 characters were randomly chosen to be repeated 'targets' (same character as shown on previous trial). Each condition was tested with three blocks, totally 174 trials for one condition. The experiment started with a practice block of 58 trials (these stimuli were not used in the main test). The order of blocks and trials in each block were randomised, and the RSS condition (control) was conducted after the subjects completed all the other conditions.

EEG recording and preprocessing

Brain electrical activity was recorded from 32 scalp sites using tin electrodes mounted in an elastic cap (BrainAmp, Brain Product). Vertical and horizontal eye movements were monitored using two electrodes placed on the outer canthus of right eye and in the inferior areas of the left orbit. The ground electrode was placed along the midline, ahead of Fz, and impedances were systematically kept below 7 kΩ. Signals were digitised at a sampling rate of 500 Hz and band-pass filtered at 0.016~70 Hz. Potentials were referenced on-line to the FCz electrode and averaged-referenced off-line. Participants were asked to minimise eye movements, head movement, and swallowing during the recording.

EEG data was low-pass filtered at 30 Hz and high-pass filtered at 0.1 Hz. Data was epoched from 1000 ms before the onset of stimulus to 1000 ms after the onset of stimulus. Only non-repeated trials were extracted. Artefact rejections were applied over the epoched data.

Trials with α wave and abnormal trend and blink during the presentation of stimuli were rejected by visual inspection. After trial rejection, the remaining numbers of trials did not vary much across conditions (see refer to the first to third quartiles; i.e., Q2 is the median.

Analysis

N1 latencies (mean ERP)

The peak latencies and amplitudes of N1 were automatically extracted at the maximum (negative) amplitude value between 130 and 330 ms at P7 electrode (left occipio-temporal area), and the peak latencies and amplitudes of VPP were automatically extracted at the maximum (positive) amplitude at Fz electrode (fronto-central area) within the same time window.

Automatic extraction was performed using ERPLAB (Lopez-Calderon & Luck, 2014) and EEGLAB functions [START_REF] Delorme | EEGLAB: An open source toolbox for analysis of singletrial EEG dynamics including independent component analysis[END_REF].

N1 widths (single-trial jitter)

N1 component width, measured directly on the mean ERP waveform, is difficult to define.

Some subjects lack distinctive peaks around the N1 (P1 and P2) to serve as delimiters. Also, subjects vary in the shape of their N1 component; some having more kurtosis than others, and others being more asymmetrical. Therefore, we measure single-trial N1 components and take the standard deviation of their latencies across trials -their temporal jitter -to be a proxy measure of component width. Several studies have used this method to establish reliable differences in N1 latency variability across experimental conditions (e.g., [START_REF] Navajas | Uncovering the mechanisms of conscious face perception: A single-trial study of the N170 responses[END_REF][START_REF] Zhang | Single-trial ERP evidence for the three-stage scheme of facial expression processing[END_REF], and across subject populations [START_REF] Milne | Increased Intra-Participant Variability in Children with Autistic Spectrum Disorders: Evidence from Single-Trial Analysis of Evoked EEG[END_REF]

Decay function relating N1 latency and jitter, to gap

We used the following exponential function to relate both N1 latency, and N1 jitter, to gap, separately for each subject:

y = m * 𝑒 !" #
, where y can be either N1 latency (ms), or N1 jitter (standard deviation, ms), g is gap (ms), m specifies the starting point (y at gap 0), and k is the inverse of the decay rate (small k means fast decay). Best fitting parameters were obtained by minimising the sum-of-squares error using Matlab's fminsearch.m (Matlab R2014a).

Statistical analyses of correlations

For statistical analyses of correlations between N1 latency and RT, and between N1 latency and VPP latency, we used the lmer program of the lme4 package [START_REF] Bates | Fitting Linear Mixed-Effects Models using lme4[END_REF] for estimating fixed and random coefficients (subjects). Our focus is on the coefficient between N1 latency and RT, or between N1 and VPP latencies, modelled as fixed effects in separate models. Variation in intercept across subjects is modelled as a random effect; hence the use of a linear mixed model. Our main results concern the coefficient associated with our fixed effects, which is evaluated in two ways. First, we examine an anovabased model comparison between a subjects-only model and the subjects-plus-coefficient model (the full model). Second, and more importantly, we use a parametric bootstrap of the full model to generate a confidence interval for the coefficient. This package is supplied in the R system for statistical computing (The R Development Core Team, 2011). For a comparison with more traditional methods, we also include the results for repeated-measures multiple regression analysis (rmMRA). Performing an rmMRA is simple: Multiple linear regressions are performed separately for each subject, then a t-test can be performed to determine if the mean of a coefficient value across subjects, is significantly different from zero.

Results

Gap-metric functions

Our main data are mean ERP for each condition and each subject. A nice visual summary for each subject can be constructed by what we call a 'stack plot': ERP stacked vertically across conditions (in order of gap), and colour-coded for amplitude. The stack plot for subjects 1 is shown in Figure 3 (code for producing this stack plot is avalaible on Github:

https://github.com/SourCherries/firecracker. Stack plots for the remainder of subjects can be found in supplementary Figure 1). A large time-course, from -800 to 600 ms is shown, so that one can see responses to both masking and target stimuli. The main observation is a systematic change in shape and overall timing from the N170 condition to the widest gap, and then to smaller and smaller gaps: as a mask is introduced and its temporal proximity to the target increases, the neural response to the target appears to delay and widen proportionally. We can quantify these relations more precisely by examining the relation between gap (ms) and N1 peak latency (ms), N1 single-trial jitter (SD ms), and reaction time (ms). Each gap function (N1 latency, jitter and RT), measured separately for each subject, is normalized by its respective value for the N170 condition. We do this because the N170 does not have a 'gap', since there is no mask in that condition (arguably, 'gap' is effectively infinity).

Normalization is also a way to easily check if values are different from their natural baseline (N170): latencies should be mostly above 0, as with jitter values and RT. For N1 latency and RT, N170 latencies are subtracted. And for jitter, which are in units of standard deviation (ms), we take the log of jitter divided by N170-jitter. Gap-metric functions for N1 latency, jitter, and reaction time, are shown in Figure 4. For each subject, N1 latencies and jitters are a clear exponential function of gap. There are also clear individual differences in decay rate, so we order gap functions for latency and jitter from subjects with the smallest to the highest decay rates (from high k to small k), which are related To further confirm that the jitters are the main factor that widen the target N1s, we removed the variance of the single trial latencies by aligning the single-trial peaks then averaged the trials that belong to each condition. All the average N1s appeared to have similar width after removing the single trial jitters (Supplement Figure 2).

In agreement with the studies in our survey (Figure 1), our results (Figure 3 and4)

show that the forward-mask changes the timing of the N1 waveform, depending on how close it is to the target image: the smaller the gap, the greater the change in N1 (both delay and widening). If that is true, then one would also expect that the speed of a subject's own neural response to the mask -as an intermediate event between mask and target -would have a similar effect: if their mask-N1 is very slow (long latency) then their 'effective gap' is much smaller than the 'nominal gap' value; but if their mask-N1 is very rapid (short latency) then their 'effective gap' is closer to the 'nominal gap' between mask-and target-image onsets.

Therefore, individual differences in mask-N1 latency should accurately predict individual differences in target-N1 decay rates. As expected, individual differences in decay rate, for both N1 latency and jitter, are excellently predicted by the latency of a subject's N1 response to the masking stimuli (Correlation between Mask N1 latency and parameter k of target N1 latency: r = -0.8583, p = 0.0134; Correlation between Mask N1 latency and parameter k of target N1 jitter: r = -0.9118, p = 0.0042) (Figure 6).

N1 latency and RT

We employ both a linear mixed model (LMM) and repeated-measures multipleregression analysis (rmMRA). For LMM, we perform 2 model comparisons. First, we compare subject-intercept only, versus subject-intercept plus N1-latency, with reaction time (RT) as the dependent variable. These models are not nested; they have different fixed-effect structures.

Therefore, we estimate these models using maximum likelihood estimation rather than restricted maximum likelihood [START_REF] Knoblauch | Modeling psychophysical data in R[END_REF]. A LMM without N1-latency fits significantly worse than the complete model, with a delta-Chi-square (1 d.f.) = 20.55, p = 5.798e-06, for the decrease in log likelihood. Similarly, rmMRA results (summarised in Table 2, along with result for the full LMM model), demonstrate a significant N1-latency and RT

correlation. An additional nested, model comparison, that examined the effect of including random slopes was insignificant, with a delta-Chi-square (1 d.f.) = 0.96, p = 0.33. Evaluation of the N1-latency and RT correlation can be made on the significance of the LMM model comparison, and the rmMRA. However, p-values obtained from model comparisons of LMM [START_REF] Knoblauch | Modeling psychophysical data in R[END_REF], and from rmMRA [START_REF] Baayen | Mixed-effects modeling with crossed random effects for subjects and items[END_REF], tend to be liberal, so we focus mainly on the confidence intervals of effects, which are based on a parametric bootstrap of 1000 samples from our model based on N1-latency and random subject intercepts (using restricted maximum likelihood). The 95-percent confidence interval for the coefficient for N1-latency is [0.344, 0.845]. The point estimate (shown in Table 2) is 0.59, so

N1 and VPP latencies

In the N170 literature, the VPP component is sometimes measured alongside the N170 component. [START_REF] Joyce | The face-sensitive N170 and VPP components manifest the same brain processes: The effect of reference electrode site[END_REF] find that the amplitudes of the N170 and VPP components vary in a precisely inverse manner and the peaks of the N170 and VPP are temporally coincident. If the N170 latencies were delayed by the forward mask, VPP latencies should also be delayed along with the N170 latencies; as with the timing of every pair of N1 and VPP in each gap condition. In other words, the N170 latencies should correlate with the VPP latencies, as shown by the LMM below.

For LMM, first we compare subject-intercept only, versus subject-intercept plus VPPlatency. We estimate these models using maximum likelihood estimation. A LMM without VPP-latency fits significantly worse than the complete model, with a delta-Chi-square (1 d.f.)

= 173.7, p = 2.2e-16, for the decrease in log likelihood. Similarly, rmMRA results (summarised in Table 3, along with result for the full LMM model), demonstrate a significant VPP-latency and N1-latency correlation. An additional nested, model comparison, that examined the effect of including random slopes was insignificant, with a delta-Chi-square (1 d.f.) = 2.06, p = 0.15.

Evaluation of the VPP-and N1-latency correlation can be made on the significance of the LMM

Topographies

Analysis described here measures matches in topographical distributions between time frames, and between conditions. Inner-products were used instead of (normalised) correlations because measuring correlation would require centring and normalising each topography on a frame-by-frame basis. However, such normalisation, we argue, would throw away information about changes in polarity across time, as well as overall voltage energy.

Assuming the N1 topography from the N170 condition is a unique marker associated with processing at N1 [START_REF] Maurer | Fast, visual specialization for reading in English revealed by the topography of the N170 ERP response[END_REF], the strength of its inner-product with topographies from other gap conditions should roughly predict the timing of the N1 in those conditions. How well are N1 latencies in each gap condition predicted by their topographical match with the N170 condition at N1?

In each gap condition, we measured the time course of inner-products between the N1 topography from the N170 condition to all other topographies in that condition. If the N1 peak is similar across gap conditions, in the sense that it reflects the same process that underlies the N1 in the N170 condition, then latency of the maximum inner-product along the time-course for a given gap condition should roughly match the timing of the N1 in that condition. In addition, this logic also applies to the RSS condition. To be clear about this analysis, we include markers in Figure 7 that depict both the timing of N1 latency (black triangles), and the timing of the max inner-product with N170 topography (red, upside down triangles). One can clearly see that maximum latencies follow N1 latencies across gap condition. To precisely quantify this correlation for every subject, we include a scatter plot between maximum-inner-product and N1 latencies in Figure 8, using different coloured markers for each of the 7 subjects. This plot is convincing : across gap and subject, topographical distributions across time are most similar to the N170 topography at the latency of the N1 (r = 0.9424, p = 8.7283e-51). A same analysis was done to match the RP (RSS condition) topography at RP latency with every frame of every condition. The question is similar: are topographies in each gap condition most similar to the RP topography at their respective N1 latencies? The answer to the question is yes. The results using RP topography are basically identical with the results using N170 topography; the RP version of figure 7 and 8 can be found in the suplementary material (Supplementary Figure 4 and5)

Figure 8. The latency of maximum inner-product with N170 topography (at N1) for each gap condition, predicts the N1 latency for that condition, for each subject. For clarity, note that, the x-axis here corresponds to the black triangles depicted in Figure 7 (for subjects S1); while the y-axis corresponds to the red triangles in that Figure.

Discussion

The current study tested the hypothesis that the N170 can be nudged forward by forwardmasking into a component that is morphologically equivalent to an RP. All aspects of waveform morphology revealed a full picture of how ISIs and forward masks influence the target N1 and therefore provided compelling evidence that: (1) The delayed N1 can be recognized as N170; and (2) RP is a delayed version of N170. For each subject, N1 latency fell from its highest level in the 0-gap (quasi-RP) condition, toward the asymptotic level given by the latency in the nomask (N170) condition. Individual differences in the decay parameter were well predicted by how quickly their mask-evoked N1 occurred: Faster N1 responses to the mask resulted in slower decay rates. Single-trial N1 jitter, our approximation of N1 width, behaved in the same manner as N1 latency: for each subject, an exponential decay with gap increase; and decay The delayed N170 should be categorized as N170

The finely-spaced gap conditions spanning 0~600 ms allowed us to uncover a continuum of changes in N1 latency and peak width. In agreement with the results of the studies using the short ISIs sumarised in our introduction, our results showed that target N1 was delayed when a short ISI was used. Moreover, the width of target N1 became wider and noisier when the gap is short, in line with [START_REF] Su | Taking a Radical Position: Evidence for Position-Specific Radical Representations in Chinese Character Recognition Using Masked Priming ERP[END_REF]. Both latency delay and the wider width can be explained by short ISIs and forward masks using an exponential decay function, and the decay rate can be predicted by the mask N1 latency. When we removed the jitter and aligned the single-trial peak, the N1 widths of different conditions were actually similar (Supplementary Figure 2), indicating that the single-trial jitter was the main cause of the widened width of N1.

In addition to the studies summarized in Figure 1, of which 4 sudies utilized adaptation [START_REF] Cao | Rapid adaptation effect of N170 for printed words[END_REF][START_REF] Fu | Neural adaptation provides evidence for categorical differences in processing of faces and Chinese characters: An ERP study of the N170[END_REF][START_REF] Kuehl | Exploring the time course of N170 repetition suppression: A preliminary study[END_REF][START_REF] Shen | Face identity is encoded in the duration of N170 adaptation[END_REF], many other N170 studies have adopted short ISI, such as studies using priming [START_REF] Campanella | Right N170 modulation in a face discrimination task: An account for categorical perception of familiar faces[END_REF][START_REF] Jemel | Is the N170 for faces cognitively penetrable? Evidence from repetition priming of Mooney faces of familiar and unfamiliar persons[END_REF] and rapid serial visual presentation (RSVP) [START_REF] Gao | Attentional biases among body-dissatisfied young women: An ERP study with rapid serial visual presentation[END_REF][START_REF] Luo | Three stages of facial expression processing: ERP study with rapid serial visual presentation[END_REF]W. Luo et al., 2013;[START_REF] Robinson | Word and face processing engage overlapping distributed networks: Evidence from RSVP and EEG investigations[END_REF]. If the N1 latency was out of the typical N170 time window (as almost all these RSVP studies have shown), these studies would have faced one problem, that is: when did the measured effect actually happen? At ~170ms where the typical N170 occurs? Or at the actual latency the effect was measured? This question is difficult to answer based only on the results of these prior studies. Even if the topography of the N1 seemed similar with the topography of a typical N170 [START_REF] Gao | Attentional biases among body-dissatisfied young women: An ERP study with rapid serial visual presentation[END_REF][START_REF] Luo | Three stages of facial expression processing: ERP study with rapid serial visual presentation[END_REF]W. Luo et al., , 2013)), there was no evidence supporting the idea that the N1 was merely pushed forward by the previous stimulus, and that the topography was not by chance akin to N170. Now our current study has provided this missing evidence.

Given that both the signal of the whole brain and the behavioral responses were delayed by the forward masks, we can conclude that the whole information process was delayed. One plausible expalanation for the delay could be that it was the result of the refractory period of the visual system [START_REF] Berry | Refractoriness and neural precision[END_REF][START_REF] Teich | Refractoriness in the maintained discharge of the cat's retinal ganglion cell[END_REF][START_REF] Teich | Relative refractoriness in visual information processing[END_REF]. Before processing the next stimulus, neurons require time to recover from processing the previous, which is compatible with the inverse correlation between mask N1 latency and the parameter k (1/decay rate) (Figure 6), suggesting that the slower the brain responds to the forward mask, the more the target N1 latency is affected. This explanation based on the refractory period could hold for not only the paradigms using forward masks, but also the paradigms presenting any kinds of visual stimuli preceding the target with short or 0 ISI.

Another implication of a delay in the entire whole-brain signal could be that, not only N170, but also other visual evoked potentials, such as P1, N2 and N400 etc., could also be delayed, as long as there is a forward stimulus. Thus, greater caution should be taken when pinpointing a cognitive process to a specific time window when short ISI is used, especially N400 studies (e.g. [START_REF] Kutas | Reading senseless sentences: Brain potentials reflect semantic incongruity[END_REF], in which normaly the stimuli of sentences are presented wordby-word in sucession with very short or zero ISI.

RP is a delayed version of N170

In order to determine if RP is a delayed version of the N170, we measured RP in both the 0-gap condition (with one forward mask, analogous to Dien et al., 2013) and the RSS condition.

The former allowed us to determine if RP was really a delayed response, for the 0-gap condition was the end of the continumn of the gap conditions. The latter condition, employing RSS, confirmed that the delayed N1 was same as the typical RP, as the topographical similarity result showed (Figure 7, 8 and Supplementary Figure 3,4,5). Therefore, the results from both conditions have provide the proper evidence for determining whether RP is a delayed version of the N170.

There are a number of implications for an ambiguity between N170 and RP components.

Differently labelled components with different latencies are often assumed to reflect different levels of processing. This can be seen in one review that has included RP and N170: Due to latency differences, [START_REF] Dien | The neurocognitive basis of reading single words as seen through early latency ERPs: A model of converging pathways[END_REF] relate RP and N170 to different stages in word processing.

However, our results now provide us with greater reason to question interpretations like this:

Perhaps the same stage of processing was reflected by both components, only with a change in latency. Another implication of RP-N170 ambiguity is that the semantic-processing often associated with RP may prove to be more important to the N170 than is realised in that literature.

Most N170 studies focus on the unique visual attributes of a stimulus-category that are often associated with experts. Nonetheless, name-object associations have been found to significantly modulate N170 amplitudes [START_REF] Heisz | Semantic Learning Modifies Perceptual Face Processing[END_REF]; consistent with the idea that the boundary between RP and N170 processing is quite vague. Nonetheless, none of this would be surprising on a physiological level, since we already know that single-cell responses in human IT, believed to underlie much of N170, are sensitive to both visual and semantic attributes [START_REF] Quiroga | Invariant visual representation by single neurons in the human brain[END_REF]. Finally, RP is often described as being more selective for differences in object category, especially familiarity [START_REF] Martín-Loeches | The gate for reading: Reflections on the recognition potential[END_REF]. While this was mostly conjecture in the past, we have recently shown that, in some conditions, the use of a forward mask can amplify N1-amplitude differences between stimulus categories [START_REF] Luo | Rapid stream stimulation can enhance the stimulus selectivity of early evoked responses to written characters but not faces[END_REF]. If N170 and RP are not as different as previously thought, then the advantage of using forward-masking as a tool to enhance differential responses should be explored in a wider range studies, especially those in the area of expert object recognition.

This ERP/N170 paper is unusual in that, instead of focusing on stimulus selectivity, we focus on waveform morphology: component latency, width, and topographical distribution. Our parametric manipulation of mask-to-target SOA (gap) allowed us to precisely measure a gradual, continuous change in these morphological features between the opposite ends of what are considered to be 'N170' and 'RP' conditions. And while the systematic transition between 'N170' and 'RP' is consistent with the idea that the RP is a delayed version of the N170, form does not necessarily equal function. Nonetheless, we remain optimistic, based on two results.

First, our correlation between N1 latency and reaction time suggests that delays in N1 are delays in information processing. Second, we show in another study that the RP and N170 share the same stimulus selectivity in 3 different types of stimulus-category contrasts [START_REF] Luo | Rapid stream stimulation can enhance the stimulus selectivity of early evoked responses to written characters but not faces[END_REF].

This is an important first step. Ideally, the time-course (and channel distribution) of information processing should be inferred from the data rather than simply assumed to occur at the latencies of pre-defined components [START_REF] Rousselet | Healthy aging delays scalp EEG sensitivity to noise in a face discrimination task[END_REF][START_REF] Rousselet | Quantifying the Time Course of Visual Object Processing Using ERPs: It's Time to Up the Game[END_REF];. By taking such an approach, one can determine if ERP sensitivity to stimulus information truly occurs where we expect it to: near the N1 or RP component, depending on the presence of a forward mask and its temporal proximity to the target stimulus. However, that would require a different, more ambitious experimental design, requiring many more trials, and using meaningful stimulus variations that we expect the visual brain to be sensitive to.

One can argue that neither the RSS paradigm nor forward-masking are essential for obtaining RP, they are just tools to amplify RP more effectively (Martín-Loeches, 2007). Let's take a closer look into the RP studies that didn't use forward masks. First, in the first RP study by [START_REF] Rudell | The Recognition Potential Contrasted with the P300[END_REF], the RP with 250 ms latency was a difference wave derived from the subtraction between the target condition presenting one target image on each trial and the control condition displaying only a meaningless image on each trial. This genuine 250 ms RP was, obviously, different from the RP delayed by the forward mask(s). Second, [START_REF] Marí-Beffa | ERP analyses of task effects on semantic processing from words[END_REF] didn't use forward masks and yet claimed they were measuring RP. However, the latency of their RP, about 200 ms, was in the range of N170 latency. Last but not least, the RP observed by [START_REF] Proverbio | RP and N400 ERP components reflect semantic violations in visual processing of human actions[END_REF], who didn't use forward masks as well, was in fact the second negative defection, unlike the other RP that was the first.

In short, the RPs that were measured without forward masks were either indistinguishable from N170 or distinguishable from the RP measured with forward masks. Since the RP reported by Rudell (see [START_REF] Rudell | The Recognition Potential Contrasted with the P300[END_REF][START_REF] Rudell | Effects of long-time reading experience on reaction time and the recognition potential[END_REF] was actually a positive wave instead of N1, was that RP also a delayed brain signal? The answer is yes, as long as forward masks were used. Besides, the reason why that RP was a positive wave might be due to the bipolar recording Rudell used.

Conclusion

In summary, systematic properties of our unique 'gap-metric' functions and our timecourses of topographical similarity, all suggest that shorter ISI leads to longer N170 latency, in other words, the delayed N1 should still be recognized as N170. The plausible interpretation of the delay could be the refractory period of the visual system. The implication of the relationship between ISI and the N170 latency also applies for other visual ERPs, that is if a short ISI is used, the component delay should be taken into account when interpreting the time course of the measured effect. Our study also suggests an effective tool to estimate the N170 latency when the peak is hard to identify (i.e., the topographical similarity measurement with dotproduct)

Taken together with the results of the gap conditions, the RSS condition further confirmed that the RP waveform is a delayed version of the N170. This implies potential benefits from greater cross-talk between RP and N170 literatures, most especially in how these literatures emphasise different types of information-processing, and in the potential use of forward masks as a tool for enhancing stimulus selectivity of the N170 [START_REF] Luo | Rapid stream stimulation can enhance the stimulus selectivity of early evoked responses to written characters but not faces[END_REF]. In general, our findings suggest that greater caution should be taken when classifying ERP components based on morphology, and that the effect of specific experimental designs (e.g., ISI, forward masks) on the morphology of these components should be considered when comparing components.

But given the flexible nature of component morphology we demonstrate in this study, perhaps morphology-based classification should be replaced or at least validated by data-driven approachs to ERP analysis (e.g., [START_REF] Rousselet | Healthy aging delays scalp EEG sensitivity to noise in a face discrimination task[END_REF][START_REF] Rousselet | Quantifying the Time Course of Visual Object Processing Using ERPs: It's Time to Up the Game[END_REF].

Figure 1 ,

 1 Figure 1, Summary of studies that used short ISI. The studies marked in yellow are forward-masking studies.

Figure 2 .

 2 Figure 2. Trial sequence for each condition. The sequence shown at the top is for the RSS (typical RP) condition, the second sequence is for the 0-gap (quasi-RP) condition. The sequence shown at the bottom is for the no-mask (N170) condition. And the sequence in the middle shows how intermediate gap conditions are generated by varying the duration of the central event (a blank screen, outlined with a dashed line in this figure). Note how the overall timing is equated between the 0-gap and no-mask conditions.

Figure 3 .

 3 Figure 3. ERP stack plot for subject 1 (top). Mean ERPs are stacked vertically across conditions and voltage is colour-coded, with a corresponding colour-bar to the right.

Figure 4 .

 4 Figure 4. Gap functions for N1 latency, N1 jitter and reaction time, from top to bottom. The upper dash lines represent RP (RSS condition), and the bottom dash lines represent N170 (no forward mask condition).

Figure 5 .

 5 Figure 5. Decay rates of N1 latency and jitter are related by a power function. The fitted line is k_n1_jitter = m*k_n1_latency^gamma, where best fit parameters are m=0.0884 and k = 1.6315.

  function almost perfectly between latency and jitter (Figure5). Therefore, N1 jitter and latency decay 'together'.

Figure 4

 4 Figure 4 also shows a clear trend for an inverse relation between RT and gap (lines are fit here instead of exponential functions, and subjects are ordered by intercept). RT is more closely examined in the next section.

Figure 6 .

 6 Figure 6. Latency of N1 response to masking stimulus predicts k-parameter (1/decay-rate) of target-N1, for both N1 latency and N1 jitter.

  ms delay in N1, we expect a delay in reaction time of about 6 ms.

Figure 7 .

 7 Figure 7. Time-course of inner product between N170 topographical distribution (at N1 latency), and topographies at every other time-frame, and condition. To improve visualisation, each inner-product time-course is normalised by its maximum inner-product. In each condition, a black triangle (pointing up) shows the latency of the N1 for the mean ERP in that condition; and a red triangle (pointing down) shows the latency of the topography in that condition which has the highest inner-product with the N170 topography (at N1), within 0 to 400 ms. According to our hypothesis, black and red triangles should roughly match up.

  by latency of mask-evoked N1. Target N1 of all the conditions had similar width after removing single-trial N1 jitter. Importantly, reaction times were also delayed by gap reductions, and were significantly correlated with N1 latency, consistent with the idea that a delayed N1 signals a delay in information processing. Finally, we demonstrate that the specific topographical distribution of voltage at N1, measured in the N170 condition, is both unique to the N1 time-point in that condition, and best matched to a small temporal window around the N1 time-point, for every gap condition and RSS condition (typical RP): 1) topographical similarity to the N170 (at N1) follows the delay in N1 latency as the gap between mask and target narrows. 2) topographical similarity to the N170 in the RSS condition appears at same time as the latency of the typical RP.

Table 1 ). RSS 0 50 100 150 200 250 300 350 400 450 500 550 600 N170

 1

Q1 126 125 124 128 128 125 133 121 119 122 118 107 124 126 114 Q2 138 136 136 136 134 136 138 134 138 139 134 142 140 140 133 Q3 141 139 142 145 138 142 145 143 143 144 137 144 145 143 143 Table 1. Number of trials per condition, summarised by quartiles across subjects. Q1 to Q3

  ). Single-trial N1 peaks were estimated from 150 ms windows centred on the latency of the N1 in the mean ERP of the appropriate gap condition and subject. Within each window of each trial, we used custom Matlab scripts and publicly available Matlab code to estimate the latency of the most negative local peak (findpeaks.m, authored by Professor Tom O'Haver at University

	of	Maryland,	Version	6.0,	Last	revised	March,	2016;

http://terpconnect.umd.edu/~toh/spectrum/findpeaks.m).

Table 2

 2

. Summary of fixed effects for both LMM (using restricted maximum likelihood) and rmMRA models of RT, predicted by N1-latency. Additionally, for the LMM, we have a random effect (not shown in this table): the standard deviation associated with subject (intercept) is 61.44, which ends up being approximately the same as the SD of intercepts measured with rmMRA. t>2.4 is significant at 5% level (two-tailed).

Table 3 .

 3 Summary of fixed effects for both LMM (using restricted maximum likelihood) and

			LMM			rmMRA
		Coefficient	SE	t	Coefficient	SE	t
	Mean RT, ms	524.84	35.32	14.86	503.06	23.65	21.27
	N1 latency,	0.59	0.12	4.81	0.71	0.14	4.95
	ms

rmMRA models of N1-latency, predicted by VPP-latency. Additionally for the LMM, we have a random effect (not shown in this table): the standard deviation associated with subject (intercept) is 9.2, which ends up being slightly higher than the SD of intercepts measured with rmMRA (8.8). t>2.4 is significant at 5% level (two-tailed).

Acknowledgments

National Natural Science Foundation of China (NSFC) to Carl M. Gaspar (grant No.: 31371132). This study was funded by the National Natural Science Foundation of China (NSFC) (grant No.: 31400974), Zhejiang Provincial Natural Science Foundation of China (ZJNSF) (grant No.: LY18C090010), and Zhejiang Provincial Dominant and Characteristic Disciplines Cultivation project of College of Education of Hangzhou Normal University (grant No.:9JYXK023) to Ye Zhang.


Rufin Vanrullen
email: rufin.vanrullen@cnrs.fr

Cognitive Science Deep Learning in Alternate Reality

Similarities and differences between deep learning models and primate vision have been the focus of recent research. Audition is, comparatively, less studied. A new report describes the emergence of human-like auditory perception in a deep neural network, and suggests a promising way to relate perceptual behavior to specific aspects of the environment.

. A new article by Francl and McDermott [5] fills this gap, by revealing how properties of human audition can emerge in deep networks, and how such properties are causally dependent on features of the natural environment.

Francl and McDermott implemented a family of deep convolutional neural networks receiving realistic auditory inputs: simulated natural sounds rendered in a virtual acoustic environment including simulated rooms with background noises, a simulated head and torso, simulated outer ears (pinnae) and a simulated cochlea. The resulting inputs were similar to the range of natural audio signals that enter the human auditory system, except for one crucial difference: they were entirely under the control of the experimenters, who could manipulate the type and 3D position of audio sources, in addition to various properties of the environment (background noises, room structure and reverberation properties, etc.). Because of this, the authors could build a large-scale dataset of labelled examples for training their networks; the training objective was to classify the exact position (azimuth, elevation) of target sources. Through alternating steps of model selection (identifying promising network hyper-parameters such as the number of layers, channels, or the kernel size of the convolutions) and model training with backpropagation, they converged on a set of deep neural networks that could localize sounds, on a separate set of test stimuli, with comparable accuracy to human subjects. Using a virtual world not only allowed the authors to construct the dataset required for supervised training, but it also gave them fine-grained control over every feature of the environment. This way, they could recreate various experimental conditions (e.g., simulating multiple sources with time lags, a room with no echo, or a modified ear shape). They tested their trained model on a battery of simulated psychophysical tests normally applied to human participants. These tests included: sensitivity to inter-aural time and level differences; azimuthal localization of broadband sounds, and its bandwidth dependence; dependence of elevation judgments on ear-specific cues, spectral smoothing or spectral frequency; localization of multiple simultaneous sources; precedence effect (i.e., a bias towards the location of the first of two near-simultaneous targets). Systematically, the model behaved like human listeners, even though it had never been trained on-or even experienced-these experimental conditions before. This agreement between human and model behavior is already remarkable; but the truly groundbreaking part comes next. The authors again took advantage of their virtual environment, this time not to recreate existing stimuli or experimental conditions, but rather to explore new, physically implausible environments-in other words, alternate realities. In one such reality (the "anechoic" world), they cancelled all sound reverberations. In another (the "noiseless" world), they turned off all background noises. In yet another environment, they replaced all natural sounds with narrowband noise bursts (the "unnatural sounds" world). The deep neural network was retrained from scratch to localize sounds in each of these alternate worlds. Subsequently, they applied the same battery of psychophysical tests to each version of the model. Compared to the model trained under naturalistic conditions (and to human participants), the one trained in a noiseless world localized sounds with abnormal bandwidth dependence. Training with unnatural sounds altered elevation sensitivity, particularly its reliance on high-frequency cues and on the specific ear shape. Finally, training in the anechoic world resulted in impaired sensitivity to interaural time differences, and an absence of the "precedence effect". The latter had long been hypothesized to arise in response to environmental reverberations, but this had been difficult to confirm. Testing a system trained in an alternate anechoic reality elegantly and unambiguously establishes this causal relation: without reverberation, there is no precedence effect. Of course, extending this compelling conclusion from a deep network to the human brain, ultimately, requires a leap of faith in the model's completeness.

Fifty years ago, scientists used to rear animals in altered environments to understand the relation between ecological sensory stimulations, behavior and brain functional networks, and tease apart innate from environmental factors. For instance, kittens reared in a world made only of horizontal and vertical stripes lacked sensitivity to oblique orientations [START_REF] Hirsch | Visual experience modifies distribution of horizontally and vertically oriented receptive fields in cats[END_REF], while those reared in stroboscopic illumination could not develop proper motion-selective responses [START_REF] Cynader | Cats reared in stroboscopic illumination: effects on receptive fields in visual cortex[END_REF]. Now, similar causal links between environmental features and behavioral properties can easily be established in deep neural networks trained in virtual environments. Thus, AI training in an alternate reality can complement animal research, or serve as a pilot exploration prior to launching costly animal studies.

Deep learning has emerged in the last decade as an essential technique for neuroscience and behavioral research [START_REF] Richards | A deep learning framework for neuroscience[END_REF]. One can implement theoretical principles in deep neural networks, and verify that they produce the expected behavior. One can lesion the networks or ablate some of their components to assess which aspect of the theory is crucial. But importantly, as highlighted here, the combination of deep learning with simulations of virtual worlds presents an additional opportunity-to selectively alter any property of the environment. Training deep networks in such alternate realities thus offers an innovative, invaluable tool to establish true causal links between features of the natural environment and behavior. Beyond the new auditory study by Francl and McDermott, a few sparse examples of this "alternate reality" approach already exist in vision. For instance, visual networks trained in increasingly noisy environments tend to increasingly rely on feedback processes [START_REF] Mittal | Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules[END_REF]. Recurrent networks that can replicate human behavior in illusory contour perception lose this ability when they are pretrained in an alternate world that includes only geometric shapes, without natural scene statistics [START_REF] Pang | Predictive coding feedback results in perceived illusory contours in a recurrent neural network[END_REF]. In the near future, it is both likely and desirable that this approach should be extended to numerous other behavioral functions and sensory modalities, with as many handcrafted alternate realities as necessary. The limit is our imagination.

Competing interests: The author declares no competing interests


Benjamin Devillers

Bhavin Choksi

Romain Bielawski

Rufin Vanrullen

Does language help generalization in vision models?

Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets. A recent model (CLIP) was found to generalize well in zeroshot and transfer learning settings. This could imply that linguistic or "semantic grounding" confers additional generalization abilities to the visual feature space. Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering, few-shot learning, transfer learning and adversarial robustness. In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training. We conclude that work is still required for semantic grounding to help improve vision models.

Introduction

Learning vision models using language supervision has gained popularity [START_REF] Quattoni | Learning visual representations using images with captions[END_REF][START_REF] Srivastava | Multimodal learning with deep boltzmann machines[END_REF][START_REF] Frome | Devise: A deep visualsemantic embedding model[END_REF][START_REF] Joulin | Learning visual features from large weakly supervised data[END_REF][START_REF] Pham | Found in translation: Learning robust joint representations by cyclic translations between modalities[END_REF][START_REF] Desai | VirTex: Learning Visual Representations from Textual Annotations[END_REF][START_REF] Hu | Transformer is all you need: Multimodal multitask learning with a unified transformer[END_REF][START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF][START_REF] Bulent Sariyildiz | Learning visual representations with caption annotations[END_REF] for two main reasons: firstly, vision-language training allows to build massive training datasets from readily available online data, without manual annotation; secondly, language provides additional semantic information that cannot be inferred from visiononly datasets, and this could help with semantic grounding of visual features.

Recently [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] introduced CLIP, a language and vision model that shows outstanding zero-shot learning capabilities on many tasks, and compelling transfer-learning abilities. A recent report [START_REF] Goh | Multimodal neurons in artificial neural networks[END_REF] showed that CLIP produces neural selectivity patterns comparable to "multimodal" concept cells observed in the human brain [START_REF] R Quian Quiroga | Invariant visual representation by single neurons in the human brain[END_REF][START_REF] Reddy | Concept cells through associative learning of high-level representations[END_REF]. From these results, it is tempting to assume Figure 1: Size of the training dataset used by the models (number of images, in log scale). ICMLM and VirTex are trained on CoCo, TSM on HowTo100M, CLIP on a (not publicly available) scrape of the internet, RN50 is trained on ImageNet-1k, the AR models and SIN models are trained on augmented versions of ImageNet-1k and BiT-M is trained on ImageNet-21k. that CLIP's generalization properties stem from semantic grounding provided by the joint visionlanguage training.

Here, we show that CLIP and other visionlanguage models do not perform better than visiononly, fully supervised models on a number of generalization settings and datasets. Representation similarity [START_REF] Kriegeskorte | Representational similarity analysis -connecting the branches of systems neuroscience[END_REF] analysis reveals that the multimodal representations that emerge through vision-language training are different from both linguistic and visual representations-and thus possibly unsuitable for transfer-learning to new visual tasks. In conclusion, additional work on linguistic grounding is still needed, if it is to improve generalization capabilities of vision models.

We provide our code for reproducibility1 .

Models

We use a number of publicly available vision, text or multimodal pretrained models, and compare their representations and generalization abilities.

To facilitate interpretation and comparisons between the models, Figure 1 reports the training dataset size for each of the visual models (including the vision-language models). They are all based on the same backbone (a ResNet50 architecture).

In CLIP, the authors train the joint embedding space of a visual network (hereafter called simply CLIP) and a language network (hereafter called CLIP-T) using contrastive learning on 400M imagecaption pairs. Note that in the present paper, the visual backbone of CLIP is a ResNet50, even though the visual-transformer-based CLIP model could reach higher performance; this choice allows for a fair comparison with the other visual models that are all based on the ResNet50 architecture. In addition, we also consider TSM [START_REF] Alayrac | Self-supervised multimodal versatile networks[END_REF], another multimodal network trained with a contrastive loss on video, audio and text inputs from the HowTo100M dataset [START_REF] Miech | Howto100m: Learning a text-video embedding by watching hundred million narrated video clips[END_REF] (containing more than 136M video clips with captions.

For training, the authors effectively used 120M video clips of 3.2s sampled at 10 fps). The effects of CLIP's and TSM's contrastive training paradigm can be compared with VirTex and ICMLM-two other recent multimodal networks. In VirTex, the visual feature representations are optimized for an image captioning task [START_REF] Desai | VirTex: Learning Visual Representations from Textual Annotations[END_REF], and for a text-unmasking task in ICMLM [START_REF] Bulent Sariyildiz | Learning visual representations with caption annotations[END_REF]. Such text-based objectives aim to provide a form of linguistic grounding using significantly fewer images than CLIP (VirTex and ICMLM models are trained on the COCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] with approximately 120K captioned images).

To understand the potential effects of linguistic training, we compare the multimodal networks to vision-only networks. We include a baseline architecture (ResNet50) trained on ImageNet-1K [START_REF] He | Deep residual learning for image recognition[END_REF] (1.3M labelled images). Second, we consider a similar architecture (ResNet50 backbone) called BiT-M [START_REF] Kolesnikov | Big transfer (bit): General visual representation learning[END_REF], trained on ImageNet-21K, a much larger dataset (14M labelled images).

While generalization and robustness properties can often be derived from access to large labelled image datasets (as in BiT-M), obtaining such labels is costly. An alternative is to train models with additional datapoints based on assumptions about the real-life data distribution-as done, e.g., with adversarial training. In this study, we use the Adversarially Robust (AR) ResNet50 models pro-vided by [START_REF] Engstrom | Adversarial robustness as a prior for learned representations[END_REF], trained on the 1.3M ImageNet training set plus 110 adversarial attacks of each image (i.e. more than 140M images overall). The different model variants (AR-L2, AR-LI4, AR-LI8) correspond to distinct adversarial attacks (refer to [START_REF] Engstrom | Adversarial robustness as a prior for learned representations[END_REF] for more details). This adversarial training was found to produce more perceptually aligned features and to improve generalization (e.g. transfer learning) in some settings [START_REF] Hadi Salman | Do adversarially robust imagenet models transfer better?[END_REF]. Another such technique was used for StylizedImageNet (SIN) models [START_REF] Geirhos | Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness[END_REF], where a variant of the ImageNet dataset (1.3M images) was designed via style-transfer to specifically reduce the network's reliance on texture information. The authors provide weights for models that are (i) only pretrained for SIN images (SIN), (ii) trained on SIN and Ima-geNet (SIN+IN) combined, or where (iii) a SIN+IN model is finetuned on ImageNet (SIN+IN-FIN).

For the vanilla ResNet50, SIN, AR and BiT-M models, we use activations after the final average pooling operation as feature representations. Although all these models share a ResNet50 backbone, there are minor differences in their implementations. We assume that such small architectural differences would not dramatically affect the feature spaces learned by these models.

Finally, we also use two text-only language models, GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] and BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], in our feature-space comparisons. As these models are not designed to process visual inputs, they cannot be tested on visual generalization; but we can use their representations of class labels (or sentence captions) as a basis for comparison with visual or multimodal network representations. In a similar way, the language stream of the CLIP model (CLIP-T) can be treated as a third language model for our comparisons.

Generalization tasks

In [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF], CLIP was systematically tested in a zero-shot setting. However, this requires a language stream to describe the different possible targets, which is not available in standard vision models. To compare the generalization capabilities of multimodal and vision-only models, we thus focus on few-shot, transfer and unsupervised learning. In each case, we evaluate performance on MNIST (LeCun et al., 1998), CIFAR10, CIFAR100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], Fashion-MNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF], CUB-200-2011 (CUB) (Wah et al., Figure 2: 1-shot, 5-shot and 10-shot accuracy over our evaluation datasets. Multimodal networks (ICMLM, VirTex, CLIP, TSM, in blue) have typically worse performance than the other models for all datasets.

Figure 3: Unsupervised clustering accuracy over our evaluation datasets. Clustering is obtained using Scikit-learn Spectral Clustering algorithm. Multimodal networks (ICMLM, VirTex, CLIP, TSM, in blue) are worse than visiononly models (in various colors) on average. 2011) and SVHN [START_REF] Netzer | Reading digits in natural images with unsupervised feature learning[END_REF] 2 . These datasets test generalization capabilities for natural images of various classes.

Few-shot learning

As a first generalization experiment, we compare few-shot learning accuracy. For this experiment, we directly pass N randomly selected samples for each class (N -shot learning) through the pretrained models to obtain a feature representation for each sample. Then, we define a class prototype by averaging the feature representations of all the samples in each class. We measure the performance of vision-only and text-vision models for N = 1, 5 and 10. Each time, the reported performance is averaged over 10 trials with different class prototypes (i.e., different random selection of samples). Figure 2 shows the performance of each model on each dataset. For CIFAR10, CIFAR100 and CUB (all the natural images datasets), BiT-M has the best accuracy. On the other hand, ICMLM, Vir-Tex, CLIP and TSM do not perform better than the vision-only models. 2 For more details on these datasets, see appendix A.

Figure 5 shows the average performance of each model across datasets, in the leftmost 3 panels.

Unsupervised clustering

Our second generalization test is an unsupervised clustering task over the same datasets. For this, we apply an out-of-the-box spectral clustering algorithm [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] using the cosine of two feature vectors as a metric. We provide the number of required clusters (number of classes) to the clustering algorithm: this ensures that all classes are represented by a cluster. The clusters are computed only on the test-sets.

To compute the accuracy on the prediction, we need to assign labels to each cluster. To do so, we use a greedy algorithm: we first choose the cluster containing the most elements in common with a given class and assign it the corresponding label. We then continue with the second cluster that has the most elements in common with another class, and so on until all clusters have been labelled.

Figure 3 shows the unsupervised clustering performance on individual datasets. It shows a similar ranking to the few-shot learning task where BiT has the best performance overall and the visio- linguistic models lag behind the vision-only models. Figure 5 panel 4 (from left) shows the performance of the unsupervised clustering algorithm averaged over all datasets.

Transfer learning

To further evaluate the models' generalization properties, we use a transfer learning setting as described in [START_REF] Hadi Salman | Do adversarially robust imagenet models transfer better?[END_REF]. We use the same datasets as in the other tasks, each time training a linear probe using the Adam optimizer. We train each linear probe for 20 epochs with a learning rate of 1e-3 and a weight decay of 5e-4.

Robustness to adversarial attacks

Another important test for generalization is the robustness to input perturbations (a form of out-ofdistribution generalization). Here, we compare the adversarial robustness of different models against untargeted and targeted random projected gradient descent (RPGD) attacks [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF]. We use L 2 and L ∞ norms to distinguish any normspecific effects. Figure 6 shows the success rate of the 100-step RPGD attacks on 1000 images taken from the ImageNet validation set. We use the foolbox API [START_REF] Rauber | Foolbox: A python toolbox to benchmark the robustness of machine learning models[END_REF] to perform all the attacks with configurations provided by (Engstrom et al., 2019a).

Summary

Overall, models trained with multimodal information (CLIP, VirTex, ICMLM, TSM) do not achieve better performance than the visual-only ResNet-based models. This systematic observation across multiple image datasets and generalization tasks (including few-shot, transfer and unsupervised learning, as well as adversarial robustness) goes against the assumption that linguistic grounding should help generalization in vision models.

Among the multimodal networks, CLIP does indeed appear to be more generalization-efficient than VirTex, ICMLM and TSM. As mentioned in [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], directly predicting highly variable text captions (as done in VirTex or ICMLM) is a difficult task that does not scale well. CLIP (and TSM) avoid generating text, relying instead on a contrastive loss between visual and linguistic embeddings. However, even with the potential benefits provided by this contrastive loss, CLIP (and TSM) do not outperform the vision models.

Finally, BiT-M, a simple vision-only model trained on a very large labelled dataset, turns out to be the overall best performing model for fewshot learning, unsupervised clustering and transfer learning, and on par with the standard ResNet50 for adversarial robustness.

Although these results are fairly consistent across datasets, there are still some differences.

For the CUB dataset, BiT-M largely outperformed the other models. This result is to be expected as the bird species in CUB are also part of ImageNet-21K labels. Then, among visiolinguistic models, CLIP is the only one competitive with the remaining visual models on this dataset.

MNIST and SVHN require classification of digits. According to [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], CLIP should be able to generalize to this task, as its training set included numerous images with text and digits. Indeed we observe that CLIP can perform as well as some of the vision models for these datasets. However, SIN and AR models perform generally better than other models.

For datasets with more natural images (CIFAR, FashionMNIST, CUB), vision models are generally better than their visio-linguistic counterparts.

Model comparison

To better understand the similarities and differences between the feature spaces learned by the various models, we now compare them using Representational Similarity Analysis (RSA) [START_REF] Kriegeskorte | Representational similarity analysis -connecting the branches of systems neuroscience[END_REF].

Method RSA is a comparison method originally used to compare fMRI data. It allows us to compare different models (with different latent space dimensions, norms, ...) which share the same structure.

This works by comparing the models' Representational Dissimilarity Matrix (RDM). RDMs are obtained by computing the 2 by 2 distances for each class of the latent representations (see figure 7). More specifically, for each visual model, we define for each class the set F c containing the feature vectors of all the images of class c, its average fc and its standard deviation σ c . The RDM matrix is then defined as [RDM i,j ] where

RDM i,j = fi -fj σ 2 i |F i | + σ 2 j |F j | 2 (1)
for each pair of class (i, j).

We use the norm of the unequal variance t-test [START_REF] Bernard | The generalization ofstudent's' problem when several different population variances are involved[END_REF] as our distance metric between the latent representations, because it allows to normalize the distances between class centroids with respect to their variances. Indeed, each class is represented by a cluster of latent vectors of different sizes.

In the case of language models (all transformerbased), we use as latent representations, the encoding of the sentence "a photo of x." where we replace "x" by the corresponding label. We then use the contextualization of the label as the text feature vector. Compared to the vision models, there is only one representation per class (only one sentence per class) hence a lack of associated with the feature vector of each class. As a result, the distance used in the RDM matrix becomes an L 2 norm.

The RDM matrix obtained with this method contains the respective distances between pre-defined concepts (in our case the 1000 classes of ImageNet). RDMs can therefore be considered as a standardized representation of latent spaces. This means that we can compare our models' representations by computing the Pearson correlation between their respective RDMs. The corresponding comparison matrix, for all pairs of models, is illustrated in Fig 8 .   Results Figure 9 shows the results of a hierarchical clustering (a) or t-SNE (Van der Maaten and Hinton, 2008) embedding (b) of the RDMs using Pearson correlation as a distance. Looking at the dendrogram, all the vision-only models are very close to one another with a maximum distance <0.2. Then, multimodal models stand a bit further (CLIP, TSM, VirTex, ICMLM); and finally, CLIP-T and the language models (BERT, GPT2) are the furthest away. This indicates that the language supervision (contrastive embedding, text-generation or text-unmasking objectives) has changed the structure of the ResNet latent space for CLIP, TSM, VirTex and ICMLM models (respectively). Yet these multimodal models are not truly linguistic either, as they are very distant also from the standard language models. This conclusion is also supported by the t-SNE plot, showing a cluster of BiT-M, RN50 and SIN vision models, a second cluster with the AR models, and further along the same direction, the multimodal networks (CLIP, VirTex, ICMLM, TSM). Note that, although this arrangement might suggest that multimodal networks possess adversarial robustness properties in common with AR models, this suggestion was not supported by our tests using actual adversarial attacks (Fig 6). Finally, the language models (BERT, GPT2 and CLIP-T) are separated from the rest, along a distinct direction. Overall, the analysis suggests that multimodal representations are neither visual nor linguistic, but surprisingly, not really in-between either3 . This is surprising as we should expect that representations trained with access to both vision and language would derive information from both modalities, and consequently end up somewhere in-between purely visual and purely textual representations.

Performance on linguistic tasks

This suggestion might be further supported by evaluating the usefulness of the learned visual representations on linguistic tasks. According to the above findings, visual representations obtained via multimodal training may fare no better than visiononly representations. To test this, for each vision model, we collect the ImageNet features for each image class, and train a standard word embedding (Skip-Gram method) while constraining the class label words to these visual feature vectors. The resulting linguistic space will thus capture some of the structure of the vision model's latent space.

Method

Architecture We train Skip-Gram models (Mikolov et al., 2013a) on Wikipedia using the Gensim library [START_REF] Řehůřek | Software Framework for Topic Modelling with Large Corpora[END_REF]. Before training, some of the embedding vectors (corresponding to the ImageNet class labels) are set to the latent representations of a vision model, and frozen during training. This training procedure forces the word embedding space to adopt a similar structure to the vision model's latent space (at least for the frozen words, i.e. the class labels).

Visual words We denote 'visual word embeddings' (resp. visual words) as the word embeddings (equivalent to the visual feature vectors) obtained from the vision models (resp. the associated word token) on ImageNet classes. Some of the classes are composed of multiple words (e.g. "great white shark"). We leverage the WordNet [START_REF] Miller | WordNet: An electronic lexical database[END_REF] structure of ImageNet classes to only keep the hypernym of the class that contains only one word (e.g. "great white shark" becomes "shark"). 1.00 0.28 0.24 0.16 0.08 0.12 0.13 0.13 0.15 0.15 0.15 0.15 0.13 0.13 0.12 0.28 1.00 0.40 0.12 0.02 0.09 0.05 0.24 0.22 0.23 0.25 0.27 0.18 0.21 0.17 All of the ImageNet categories that have the same one-word hypernym are grouped together into one unique hyperclass. For instance, the "shark" hyperclass contains the classes "great white shark" and "tiger shark". Finally, to obtain the visual word embeddings, we average the visual representation of all the images of each hyperclass from the Im-ageNet validation set. This gives a total of 824 visual words.

Besides, we choose a vocabulary of 20,000 words (taken from the most frequent tokens in Wikipedia). Only 368 visual words are among the 20,000 most frequent words, so we extend our vo-cabulary to also contain the 456 other visual words, resulting in a total vocabulary of 20,456 words.

Embedding dimension Since the vision models do not all share the same feature dimensions, in order to compare all Skip-Gram models, we reduce the dimensionality of the feature spaces of all vision models to 300 dimensions using a PCA. The PCA is computed using the visual features of all images in the ImageNet validation set. Consequently, the Skip-Gram word embeddings are trained with 300 dimensions.

Training We train the Skip-Gram models for 5 epochs, using the standard negative sampling strategy. We use window sizes of 5 words and a learning rate of 1e-3. We use the "vectors_lockf" feature of the Gensim library to freeze certain word embeddings during training.

For the dataset, we use a recent dump of Wikipedia and we split it into two sets containing 80% and 20% of the articles for the training and validation sets.

Evaluation

We evaluate our Skip-Gram embeddings on two tasks: word analogies and word pair similarities.

Word Analogy This standard task (Mikolov et al., 2013b) for evaluating the quality of word embeddings consists of quadruplets {A, B, C, D} (e.g. "man", "king", "woman","queen") supporting the relation "A is to B as C is to D". The task consists in finding the 4th one given the first three, by solving the equation in the latent space: D = B -A + C. The more accurate the model, the better its representation. We evaluate the word embeddings on the full dataset provided by (Mikolov et al., 2013b) that we split in two different sets: morphology analogies (such as "write", "writes", "work", "works"), and semantic analogies (such as "son", "daughter", "boy", "girl"). If vision-language training helps "ground" the visually-derived word embeddings, we expect this grounding to be more helpful in the resolution of semantic, rather than morphology analogies.

Word Pair similarity Another task for evaluating the quality of word embeddings is to ask humans to rate the semantic similarity of pairs of words (e.g. on a scale of 0 to 10, how close is "queen" to "king"? How close is "queen" to "woman"? etc.) [START_REF] Finkelstein | Placing search in context: the concept revisited[END_REF] and then compute the same similarity evaluations in the latent spaces of the models. The higher the (Pearson) correlation between the pairwise similarities of a model and human pairwise similarity judgments, the better the representation of the model.

Results

The baseline Skip-Gram produces the best word embeddings overall (black bars in Fig 10). This is to be expected since the embeddings are learned freely, without any additional constraint during training. Interestingly, this baseline advantage is weakest in the case of the semantic analogy task (Fig 10,leftmost panel), where some of the vision and visio-linguistic models are on par with the baseline. This shows that the frozen vectors do not necessarily impede the performance when the analogies are defined semantically (and might thus be presumed to contain some visual component). However, even for these semantic analogies, vision or vision-language word embeddings never significantly surpassed the baseline performance.

In the word pair similarity task, networks show variable performance levels, but without a clear distinction between vision-only and vision-language models. Among the visio-linguistic networks, CLIP and TSM, which are trained contrastively on a large amount of data (see Figure 1) have embeddings that correlate well with human word similarity judgements. However, when compared with the vision-only models, we do not observe a clearcut performance improvement. Indeed, the best vision-only model (BiT-M) is on par with CLIP and TSM. Interestingly, by comparing the results from the Fig 10 rightmost panel to the data plotted in Fig 1 , we observe that among our twelve models, the top six for the word pair similarity task (TSM, CLIP, BiT, and the three AR models) correspond to those models that were trained on the largest datasets.

For the analogy tasks (semantic and morphology), there is no particular trend. However in both cases, the best performing model (excluding the baseline) is a visual one: SIN+IN in the semantic case, and AR-L2 in the morphology case.

In summary, we find that multimodal training of visual features does not improve their usefulness for language tasks either, and we suggest that the amount of training data may be a more important factor for generalization.

Legitimacy of the visual word embeddings

In the previous results, for training the visuallyguided word embedding models, we averaged the visual feature vectors over many examples for each class. This averaging can potentially alter the quality of the embeddings, e.g. by discarding important information about the feature distributions. Thus, we check the validity of these averaged feature vectors4 , by verifying that they remain useful in a vision context. We use these visual feature vectors

Figure 10: Semantic Word Analogy (such as "son", "daughter", "boy", "girl"), Morphology Word Analogy (such as "write", "writes", "work", "works") and word pair similarity results for the visually constrained Skip-Grams.

The Baseline is a vanilla Skip-Gram model (300 dimensions) where all 20,456 word embeddings are free to be learned.

as class prototypes and evaluate the corresponding nearest-neighbor classification accuracy on the ImageNet validation set 5 with a method similar to section 3.1. For all models considered, classification accuracy was well above chance (p<0.01): this means that the class-specific vectors indeed remain useful as visual representations of their category. Furthermore, we computed the correlation between this visual classification accuracy of the word embedding, and the corresponding word analogy or word-pair similarity accuracy for each model. The resulting Pearson correlation coefficient was r=-0.0821 with the semantic Word Analogy performance, r=0.301 with the morphology Word Analogy performance, and r=0.797 with the Word Pair Similarity.

The significant high correlation of visual classification with word-pair similarity performance might be caused by the visual component of the word similarity judgments performed by human subjects. Indeed, many "similar words" also entail similar visual features (tiger, jaguar, cat, feline), and so the word-pair similarity task may not be a pure language task.

Discussion and Conclusion

It is a highly appealing notion that semantic grounding could improve vision models, by introducing meaningful linguistic structure into their latent space, and thereby increasing their robustness and generalization properties. Unfortunately, our experiments reveal that current vision-language training methods do not achieve this objective: the resulting multimodal networks are not better than 5 With the images regrouped into our 824 classes. vision-only models, neither for few-shot learning, transfer learning or unsupervised clustering, nor for adversarial robustness. In addition, compared to vision-only models, the multimodal networks' visual representations do not appear to provide additional semantic information that could serve as a useful constraint for a word-embedding linguistic space.

The present inability of linguistic grounding methods to deliver their full promise does not imply that this cannot happen in the future. In fact, we believe that exploring the current models' performance and representations, as we do here, can help us understand their limitations and adjust our methods accordingly. Specifically, we found that multimodal representations are neither visual nor linguistic, but are not really in-between either (Fig 9). In CLIP and TSM, for instance, the contrastive learning objective encourages the visual and language streams to agree on a joint embedding of images and corresponding captions. However, such agreement, by itself, does not constrain either latent space to remain faithful to its initial domain. As a result, CLIP's (and TSM's) visual representations may discard information that could prove critical for transfer-learning to other visual tasks. If this is true, we predict that adding domain-specific terms to the multimodal loss function (e.g. selfsupervision) could be a way to improve visual generalization, while retaining the advantages of multimodal training-possibly including semantic grounding.

Figure 4 :

 4 Figure 4: Transfer learning accuracy over our evaluation datasets. For each dataset and model, we train a linear layer to classify the models' visual features. Multimodal networks (ICMLM, VirTex, CLIP, TSM, in blue) have worse performance accuracy than vision-only models (in various colors).

Figure 5 :

 5 Figure 5: Average performance of the models across datasets, with standard error of the mean, for the various generalization tasks (few-shot learning, unsupervised clustering, transfer learning). Multimodal networks (ICMLM, VirTex, CLIP, TSM in blue) have worse generalization accuracy across all tasks.

  Fig 4 shows the performance of the models on this task, separately for each dataset, and Fig 5 (rightmost panel) reports the average across datasets. Multimodal networks fail again to improve generalization.

Figure 6 :

 6 Figure 6: Robustness of some of the models to untargeted (top) and targeted (bottom) random projected gradient descent (RPGD) attacks for varying epsilons, with L 2 (left) or L ∞ norm (right). AR models are robust by design. Multimodal networks (CLIP, Vir-Tex) are less robust than vision-only models (RN50, SIN+IN, BiT-M).

Figure 7 :

 7 Figure 7: How to compute representational Dissimilarity Matrices (RDMs). RDMs are built from the model's embedding space. The RDMs can then be used for a Representational Similarity Analysis by comparing them using a Pearson Correlation.

Figure 8 :

 8 Figure 8: Correlations of the RDMs of our evaluation models. The RDMs are computed as explained in Fig 7 using the ImageNet dataset.

Figure 9 :

 9 Figure 9: (a) Dendrogram of a hierarchical clustering of the RDMs. (b) t-SNE of the RDMs.

https://github.com/bdvllrs/ generalization-vision

Of course, we describe multimodal networks as neither visual nor linguistic, but this is to be understood in relative terms-they are relatively far from both visual models and linguistic models. In absolute terms, there is always a reasonable amount of similarity between multimodal networks and certain visual or linguistic models.

We here test the 300d vectors after the PCA dimensionality reduction.

Acknowledgements

This research was supported by ANITI ANR grant ANR-19-PI3A-0004, AI-REPS ANR grant ANR-18-CE37-0007-01 and OSCI-DEEP ANR grant ANR-19-NEUC-0004.

A Datasets

We briefly describe all the datasets used in our experiments.

A.1 CIFAR10 and CIFAR 100

These datasets contain images of animals and objects comprising either 10 (CIFAR10) or 100 (CI-FAR100) categories. All the 60,000 images -50,000 train and 10,000 test-are of 32 × 32 resolution with RGB color channels.

A.2 CUB dataset

Caltech-UCSD Birds, or CUB, dataset consists of 6033 images of 200 species of birds. Apart from the species name, the dataset also provides bounding boxes, approximate bird segmentation and attribute labels for each image, allowing for a finer analysis at a feature level.

A.3 MNIST

Considered one of the simplest datasets, MNIST contains 28 × 28 black and white images of handwritten digits from 0 to 9. It comprises of 60,000 training images and 10,000 test images.

A.4 Fashion MNIST

Based on article images from Zalando, an ecommerce platform, Fashion MNIST contains 28 28 black and white images of 10 clothing categories. Designed with an aim to act as a "direct drop-in replacement for the original MNIST", it contains the same number of training and testing images as that of MNIST.

A.5 StreetView House Numbers

StreetView House Numbers, or SVHN dataset consists of images of digits from 0 to 9. Compared to MNIST, it is generally considered a more real-world dataset for optimizing neural networks since it contains images of digits in a more natural setting-600,000 colored images of digits provided by Google Street View images.