Itay Yaron

Nathan Faivre

Liad Mudrik

& Matan Mazor

Haim Levanon

Tel

Running head: INDIVIDUAL DIFFERENCES DO NOT MASK EFFECTS OF UCP 1 Individual differences do not mask effects of unconscious processing

Keywords: unconscious processing, individual differences, consciousness

A wave of criticisms and replication failures is currently challenging claims about the scope of unconscious perception and cognition. Such failures to find unconscious processing effects at the population level may reflect the absence of individual-level effects, or alternatively, the averaging out of individual-level effects with opposing signs. Importantly, only the first suggests that consciousness may be necessary for the tested process to take place. To arbitrate between these two possibilities, we tested previously collected data where unconscious processing effects were not found (26 effects from 470 participants), using four Bayesian and frequentist tests that are robust to individual differences in effect signs. By and large, we found no reliable evidence for unconscious effects being masked by individual differences. In contrast, when we examined 136 non-significant effects from other domains, a novel non-parametric sign consistency test did reveal effects that were hidden by opposing individual results, though as we show, some of them might be driven by design-related factors. Taken together, four analysis approaches provide strong evidence for the restricted nature of unconscious processing effects not only across participants, but also across different trials within individuals. We provide analysis code and best-practice recommendations for testing for non-directional effects.

Individual differences do not mask effects of unconscious processing

Introduction

Our brains simultaneously perform complex information processing functions and yet, at any given moment in time, only a small subset of these functions is accompanied by conscious experience. This raises the question: which brain functions depend on consciousness, and which functions can take place without it?

One approach to investigating the scope and limits of unconscious processing is to measure the effect of different stimulus features on behaviour, while making sure that the stimulus itself is not consciously perceived (for review, see [START_REF] Kouider | Levels of processing during non-conscious perception: A critical review of visual masking[END_REF][START_REF] Reingold | Using direct and indirect measures to study perception without awareness[END_REF]. If a stimulus feature affects behaviour even when the participant is not aware of the stimulus, being conscious of the stimulus cannot be necessary for processing that feature.

For example, Vorberg and colleagues (2003) studied the role of consciousness in motor preparation. In a series of experiments, they presented an arrow stimulus (henceforth, the prime) which was followed by an arrow-shaped metacontrast target stimulus, rendering it invisible [START_REF] Breitmeyer | Implications of sustained and transient channels for theories of visual pattern masking, saccadic suppression, and information processing[END_REF]; see Figure 1A). Unconscious motor preparation priming was demonstrated by showing that participants responded faster to the target stimulus when its direction was congruent with the direction of the prime. This suggests that the direction of the prime has been represented unconsciously, triggering a motor plan. In similar studies, participants were reported to unconsciously perform arithmetic operations [START_REF] Sklar | Reading and doing arithmetic nonconsciously[END_REF], extract and integrate word meanings [START_REF] Damian | Congruity effects evoked by subliminally presented primes: Automaticity rather than semantic processing[END_REF][START_REF] Gaal | Can the meaning of multiple words be integrated unconsciously[END_REF][START_REF] Sklar | Reading and doing arithmetic nonconsciously[END_REF], or scenes and objects [START_REF] Mudrik | Integration without awareness: Expanding the limits of unconscious processing[END_REF], detect errors [START_REF] Charles | Distinct brain mechanisms for conscious versus subliminal error detection[END_REF], and exert inhibition over responses [START_REF] Gaal | Frontal cortex mediates unconsciously triggered inhibitory control[END_REF] to stimuli that were masked from awareness. Findings of high-level processing in the absence of consciousness served to inform and reform theories of consciousness [START_REF] Dehaene | Towards a cognitive neuroscience of consciousness: Basic evidence and a workspace framework[END_REF][START_REF] Lamme | Visual functions generating conscious seeing[END_REF][START_REF] Lau | Empirical support for higher-order theories of conscious awareness[END_REF][START_REF] Oizumi | From the phenomenology to the mechanisms of consciousness: Integrated information theory 3.0[END_REF].

However, more recent work has called into question some of these previous findings and their interpretations. First, many of the original results do not replicate when tested in independent samples of participants (using direct replications, e.g., [START_REF] Biderman | Evidence for implicit-but not unconscious-processing of object-scene relations[END_REF][START_REF] Moors | Unconscious arithmetic: Assessing the robustness of the results reported by karpinski, briggs, and yale[END_REF][START_REF] Stein | Unconscious semantic priming from pictures under backward masking and continuous flash suppression[END_REF], or conceptual replications, e.g., Hesselmann et al., 2015, 2016;[START_REF] Rabagliati | The importance of awareness for understanding language[END_REF]. Second, some of these findings might be driven by residual consciousness in a subset of trials due to unreliable awareness measures [START_REF] Meyen | Advancing research on unconscious priming: When can scientists claim an indirect task advantage[END_REF]Moors & Hesselmann, 2018;[START_REF] Rothkirch | What we talk about when we talk about unconscious processing-a plea for best practices[END_REF][START_REF] Shanks | Regressive research: The pitfalls of post hoc data selection in the study of unconscious mental processes[END_REF]Zerweck et al., 2021). Indeed, when re-analyzed to properly control for this possibility, some of these effects disappear [START_REF] Meyen | Advancing research on unconscious priming: When can scientists claim an indirect task advantage[END_REF][START_REF] Shanks | Regressive research: The pitfalls of post hoc data selection in the study of unconscious mental processes[END_REF]. As a result, the scientific pendulum seems to be receding back to a narrower account of unconscious processing, consistent with a functional role of consciousness in most aspects of cognition [START_REF] Balota | Unconscious semantic processing: The pendulum keeps on swinging[END_REF][START_REF] Moors | Continuous flash Stimulus fractionation rather than integration[END_REF][START_REF] Peters | Does unconscious perception really exist? Continuing the ASSC20 debate[END_REF]).

Yet, the field is still far from reaching a consensus regarding the scope and limits of unconscious processing. Although progress has already been made in recent years toward improving methodology in unconscious processing studies, revealing the functional role of consciousness in cognition and perception remains difficult. Here we consider a largely neglected limitation of unconscious processing studies: by focusing on the average of signed (i.e., directional) single-participant summary statistics (for example, subtraction of reaction times between two conditions), previous investigations require not only that unconscious processing should leave a trace on behaviour, but also that this trace should be qualitatively similar across different participants (i.e., that the experimental manipulation would affect most participants in the same direction). We note that though this second requirement is intuitive, it is orthogonal with the theoretical question at stake; our main concern is whether a given stimulus feature can affect behaviour in the absence of consciousness, yet this does not necessarily imply that it affects all participants in the same way. This way, previous analyses of unconscious processing may have been too conservative, potentially missing effects that happen to vary between different participants.

On the face of it, pronounced individual differences in unconscious processing effects on cognition seem possible, even likely. Indeed, previous investigations revealed heterogeneity in susceptibility to the attentional blink [START_REF] Martens | Quick minds don't blink: Electrophysiological correlates of individual differences in attentional selection[END_REF], in the effects of stimulus onset asynchrony (SOA) on metacontrast masking [START_REF] Albrecht | Individual differences in metacontrast masking are enhanced by perceptual learning[END_REF], and in the effects of visual imagery on conscious perception in a binocular rivalry setting [START_REF] Dijkstra | Between-subject variability in the influence of mental imagery on conscious perception[END_REF]. Some qualitative differences have been linked to variability in processing speed [START_REF] Martens | Quick minds don't blink: Electrophysiological correlates of individual differences in attentional selection[END_REF], genetics [START_REF] Maksimov | Can common functional gene variants affect visual discrimination in metacontrast masking?[END_REF], and brain anatomy and physiology [START_REF] Boy | Individual differences in subconscious motor control predicted by GABA concentration in SMA[END_REF][START_REF] Gaal | Pre-SMA gray-matter density predicts individual differences in action selection in the face of conscious and unconscious response conflict[END_REF]. Critically, in other behavioural paradigms, unconscious stimuli had opposite effects on different participants. [START_REF] Bolger | Causal processes in psychology are heterogeneous[END_REF] showed that while most participants responded faster to upright faces in a breaking continuous flash suppression task (b-CFS; [START_REF] Jiang | Processing of invisible stimuli: Advantage of upright faces and recognizable words in overcoming interocular suppression[END_REF], some responded faster to upside-down faces. Other studies found that priming effects changed in magnitude and even flipped in sign as a function of SOA and visibility [START_REF] Boy | Visibility predicts priming within but not between people: A cautionary tale for studies of cognitive individual differences[END_REF][START_REF] Schlaghecken | Masked prime stimuli can bias "free" choices between response alternatives[END_REF]. On the other hand, in other studies individual differences in prime visibility were not correlated with the magnitude of priming effects [START_REF] Albrecht | Individual differences in metacontrast masking are enhanced by perceptual learning[END_REF][START_REF] Boy | Visibility predicts priming within but not between people: A cautionary tale for studies of cognitive individual differences[END_REF]; but see [START_REF] Eimer | Links between conscious awareness and response inhibition: Evidence from masked priming[END_REF]. Taken together, it is not clear if, and to what extent, unconscious effects are subject to meaningful individual variability. If they do, then some previously reported null results might actually be true effects, masked by such variability.

The paper proceeds as follows. We first simulate a setting where a strong effect of unconscious processing on behaviour is entirely missed in standard analysis, due to pronounced inter-individual differences. We then show that the same effect is revealed when using three tests that are robust to population variability: Bayesian prevalence analysis [START_REF] Ince | Bayesian inference of population prevalence[END_REF][START_REF] Ince | Within-participant statistics for cognitive science[END_REF], Bayesian hierarchical modelling [START_REF] Haaf | Some do and some don't? Accounting for variability of individual difference structures[END_REF], and a frequentist test based on analysis of variance (ANOVA; [START_REF] Miller | Implications of individual differences in on-average null effects[END_REF]. When applied to data gathered from eight unconscious processing studies (reporting 26 non-significant effects), the same three tests support the null hypothesis according to which the behaviour of individual participants is unaffected by unconscious perception. This strengthens claims for a true absence of an effect in these studies.

Finally, we propose a non-parametric alternative that provides improved sensitivity and specificity, avoiding potentially unjustified statistical assumptions regarding the data-generating process. Our test successfully reveals effects on multisensory integration, visual search, and confidence ratings that could not be detected using standard directional analysis. However, similar to the three other approaches, it reveals no effects when applied to the studies of unconscious processing examined here. We conclude that existing data are most consistent with the absence of influences of unconscious stimuli on cognition and perception, not only at the population, but also at the single-participant level.

Simulating non-directional unconscious effects

To demonstrate how true causal effects of unconscious processing can be masked by interindividual differences in effect signs, we simulated a typical experiment using a within-participants manipulation (Figure 1). Specifically, we generated trial-by-trial data from a standard unconscious priming experiment. For each simulated participant, we generated reaction time data from two conditions (corresponding to congruent and incongruent primes in unconscious processing studies). Individual-level effect sizes (in milliseconds) were sampled from a normal distribution centred at zero ( , where denotes the true effect size of the participant and the between-participant standard deviation. Then, the trial-by-trial reaction times (RTs) of each participant and condition were generated according to each participant's true effect score ( ), the relevant condition ( , where denotes the incongruent condition, and denotes the congruent condition), and the within-participant standard deviation ( ) ( ).

In two simulations, we manipulated two factors: the between-participant standard deviation (SD) over effect sizes ( ), and the within-participant SD over RTs within each condition ( ). This resulted in two distinct scenarios under this framework: (1) a qualitative or non-directional differences scenario, where all individuals show an effect, but individual-level effects largely vary in magnitude and sign ( =15, =30; Figure 1B)), and (2) a global null scenario [START_REF] Allefeld | Valid population inference for informationbased imaging: From the second-level t-test to prevalence inference[END_REF][START_REF] Nichols | Valid conjunction inference with the minimum statistic[END_REF], where no single participant is affected by the experimental manipulation ( =0, = 100; Figure 1C). We simulated =200 trials from =15 participants per scenario, noting that the general principle holds for other sample sizes and number of trials.

First, we analyzed this simulated data using a two-sided paired t-test on the differences in mean RTs between the two conditions. This is the standard protocol for testing if unconscious processing took place. In both simulations, we obtained a null result, revealing no evidence for a difference in RT between the congruent and incongruent conditions (non-directional differences:

, 95% CI , , ; global null: , 95% CI , , ). Importantly, in the non-directional differences simulation, all participants were affected by the experimental manipulation (that is, their true effect sizes were different from zero). Thus, this commonly used approach systematically misses true causal effects of the experimental manipulation whenever they are inconsistent between participants.

To reiterate, a standard t-test misses existing individual-level effects because, operating on individual-level summary statistics, it is oblivious to within-participant variability in the dependent variable. In recent years, researchers sought to address this limitation, advocating for the use of statistical methods that incorporate both within and between-participant variability.

Specifically, three approaches were proposed. First, the prevalence Bayesian test approach [START_REF] Ince | Bayesian inference of population prevalence[END_REF][START_REF] Ince | Within-participant statistics for cognitive science[END_REF]henceforth PBT) estimates the prevalence of individual-level effects in a given population (the proportion of individuals showing an effect). The prevalence approach relies on a two stages procedure in which effects are tested at the individual level using a standard hypothesis-testing approach, and the true proportion of significant effects is then estimated using a Bayesian parameter estimation procedure. Second, the qualitative individual differences approach [START_REF] Rouder | Beyond means: Are there stable qualitative individual differences in cognition[END_REF][START_REF] Rouder | Are there reliable qualitative individual difference in cognition[END_REF][START_REF] Haaf | Some do and some don't? Accounting for variability of individual difference structures[END_REF] henceforth QUID) quantifies the relative support for the presence of "qualitative differences" in effects, that is, inter-individual differences in effect signs, by performing a Bayesian model comparison over a family of hierarchical models with different constraints [START_REF] Haaf | Some do and some don't? Accounting for variability of individual difference structures[END_REF]. Third, [START_REF] Miller | Implications of individual differences in on-average null effects[END_REF] introduce a parametric and frequentist test, based on ANOVA. Specifically, their Omnibus ANOVA test (henceforth OANOVA) tests the joint null hypothesis that there are no systematic differences between experimental conditions across individuals, or within individuals and across trials. Together, this is equivalent to the global null scenario we presented above.

We applied the tests to our simulated data, using the default priors from the original publications [START_REF] Ince | Bayesian inference of population prevalence[END_REF][START_REF] Rouder | Are there reliable qualitative individual difference in cognition[END_REF], and an implementation of the ANOVA model underlying the OANOVA test [START_REF] Miller | Implications of individual differences in on-average null effects[END_REF]; see https://github.com/mufcItay/NDT for an R based implementation of the test). For PBT, we examined whether the lower bound of the prevalence exceeded zero. We considered as evidence for an effect, as evidence for no effect (global null), and values between these thresholds ( ) as inconclusive [START_REF] Jeffreys | The theory of probability[END_REF], and used an of 0.05 for the OANOVA test. Reassuringly, all tests were able to differentiate between the two simulated scenarios, providing very strong evidence for an effect in the non-directional differences scenario, but not in the global null one. Specifically, According to PBT, about half of the population was estimated to show an effect of congruency on RT in the non-directional differences simulation (using a two-sided t-test for the individual-level test; = [26, 74],

=51%; the 95% highest density interval and maximum a posteriori, respectively), but this proportion was not reliably different from zero in the global null simulation ( = [0, 24], = 2%). Using the QUID method, a random effects model with individual-level effects was overwhelmingly preferred in the non-directional differences simulation ( = 1.51e+33), but a null model was preferred in the global null simulation ( = 0.08). Similarly, the OANOVA test revealed significant results in the non-directional differences scenario (F(15, 2970) = 13.95, p < .001), and a nonsignificant effect in the global null simulation (F(15, 2970) = 1.33, p =.177).

The simulations above demonstrate that adopting a non-directional approach, that is, an approach that takes into account the potential for opposite true effect signs among different participants, has the potential to reveal individual-level effects that would otherwise be missed due to high between-participant variability. Equipped with these validated tools, in the next section we use the QUID, PBT, and OANOVA tests to ask whether null results in the field of unconscious processing are driven by such inter-individual variability, or alternatively, whether they reflect the true absence of a causal effect.

Reexamining unconscious effects

To examine whether inter-individual differences masked true unconscious priming effects in previously reported studies, we collected and tested data from eight studies that reported null results [START_REF] Benthien | Does location uncertainty modulate unconscious processing under continuous flash suppression?[END_REF][START_REF] Biderman | Evidence for implicit-but not unconscious-processing of object-scene relations[END_REF][START_REF] Faivre | Multisensory integration in complete unawareness: Evidence from audiovisual congruency priming[END_REF][START_REF] Hurme | Neuronavigated TMS of early visual cortex eliminates unconscious processing of chromatic stimuli[END_REF][START_REF] Skora | Evidence that instrumental conditioning requires conscious awareness in humans[END_REF][START_REF] Stein | Dissociating conscious and unconscious influences on visual detection effects[END_REF]Zerweck et al., 2021;[START_REF] Chien | The limits of unconscious semantic priming[END_REF]all datasets and analysis scripts are publicly available online: https://github.com/mufcItay/NDT).

We collected data associated with 26 null effects (see Supplementary Table 1 for details about all effects), 19 focusing on differences in RT and 7 on differences in signal detection sensitivity, d' [START_REF] Green | Signal detection theory and psychophysics[END_REF]. We used the criteria set by the original authors for demonstrating unawareness (e.g., using objective and/or subjective measures of awareness), and a two-sided non-parametric sign-flipping test for filtering out significant priming effects1 . Finally, we excluded participants with fewer than five trials per experimental condition and/or zero variance in the dependent variable (e.g., when accuracy was measured). Together, these data allowed us to reexamine null unconscious processing effects using a non-directional approach that takes into account the potential for differences in effect signs when testing for group-level effects. We accordingly asked whether true effects of unconscious processing were masked by population heterogeneity in effect signs.

To that end, the effects of interest were tested using PBT, QUID, and the OANOVA tests (see Supplementary Figure 1 for an analysis of the significant directional effects which were excluded). PBT was applied to all 26 effects. In contrast, QUID and OANOVA were used on subsets of 20 and 21 of these effects, respectively (omitting five effects of signal detection sensitivity, d', from both tests, and one additional RT interaction from the QUID analysis, as its current implementation only supports simple RT effects). All tests agreed on finding no reliable evidence for non-directional unconscious effects. According to PBT, the MAP prevalence statistic was zero in 76.92% of the effects (maximal = 11.36%; see Fig. 2A), and the 95% HDI included zero in all of them. Similarly, for both QUID and the OANOVA tests, no single or p-value revealed evidence for an effect (maximal = 0.80 and all p-values > 0.05; see Figure 2B,C). Notably, QUID obtained moderate evidence for the global null model in 70% of the cases (see Fig. 2B). The remaining effects were inconclusive. Hence, for the effects collected here, in the case of unconscious processing, the three tests revealed a highly similar pattern of results, consistent with a strong interpretation of previously reported null results as revealing the genuine absence of a causal effect of unconsciously perceived stimuli on behaviour. and Global Null (GN) simulations described above, presented as square-shaped markers). Effect labels 2 appear on the x-axis. Panel A: the estimated prevalence of an unconscious effect in each of the cases, using PBT [START_REF] Ince | Bayesian inference of population prevalence[END_REF]. Markers indicate the maximum a posteriori ( ) for each of the effects, and segments depict the 95% highest density interval ( ). Grey markers depict above-zero MAPs. Yet, the reviewed approaches also have some limitations that make it harder to draw firm conclusions based on their results. First, in contrast to frequentist tests within the Null Hypothesis Statistical Testing (NHST) tradition, PBT and QUID provide no control over long-term error rates (the probability of finding a false positive result or missing a true result over an infinite number of tests, with the former being more critical to our point here). Such error control promises a much-needed 'fool-proof' method to infer the existence of unconscious processing effects without making too many mistakes in the long run [START_REF] Lakens | Improving inferences about null effects with bayes factors and equivalence tests[END_REF]. To illustrate, while an alpha level of 0.05 guarantees that only one in 20 tests will generate a significant result when there is no true difference between the conditions, using a Bayes Factor cutoff of 3, or an HDI of 95%, provides no such guarantee.

Second, both the model comparison approach used in QUID and the OANOVA test necessarily assume a parametric model of the data, making specific assumptions of normality and equal within-individual variance. In simulations, we find that violations of this second assumption can have dramatic effects on the specificity and sensitivity of both tests (see Appendix A). This can be addressed by more complex models that are capable of handling different distribution families, but as model complexity grows, unwanted effects of assumption violations may become harder to spot and quantify. Hence, taking a non-parametric approach provides safer inferences when the form of the data-generating process is not fully known.

Lastly, PBT begins with testing the significance of effects at the single subject level, thereby dichotomizing a continuous test statistic into one bit of information: significant or not.

This dichotomization results in information loss and introduces an additional free parameterthe individual alpha level. This step is well justified when estimating population prevalence, but it is unnecessary for our purpose of detecting a non-directional effect at the population level. As we describe below, using a continuous participant-level statistic makes our test more sensitive (see Appendix B for a direct comparison between the two approaches).

In the next section, we introduce a novel non-directional test that takes into account population heterogeneity to infer group-level effects. The test is both frequentist and nonparametric, which addresses the above issues. Similarly to the OANOVA test it promises a control for long term error-rates, but unlike it, our test does not assume a parametric model of the data-generating process. Using a continuous within-participant summary statistic, it is also more statistically powerful than approaches that focus on a dichotomous notion of effect prevalence (see Appendix B).

Sign Consistency: a non-parametric test that is robust to qualitative differences

Our test assumes that a within-participant effect is convincing if it is consistently evident across different trials. To test this, we can split the trials of an individual into two random halves, and ask whether both halves show the same qualitative effect (e.g., the performance in the congruent condition is higher than in the incongruent condition; see Fig. 3A). By doing this many times, we can measure how often the two halves agree. Following this strategy, we estimate the consistency of effect signs within each individual by measuring the frequency of consistent results across splits. Then, we compare the group-mean consistency score against a null distribution: 10,000 samples of group-level consistency scores, obtained after randomly shuffling the experimental condition labels within participants (here, to speed up the computational process, for each participant, 25 permutations were created, from which we randomly sampled a single permutation in each null distribution sample; [START_REF] Stelzer | Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): Random permutations and cluster size control[END_REF]. Hence, our null distribution reflects the expected consistency of within-participant effects when there is no link between the experimental condition and the dependent variable. An easy-to-use implementation of the sign consistency test is available as part of the signcon R package (https://github.com/mufcItay/signcon). incongruent RT distributions, to test if the direction of the difference in the two halves is consistent or not. Then, to determine if the group shows evidence for non-directional effects, the averaged consistency score across participants (plotted in green), which is the proportion of consistent splits across all splits, is compared to the null distribution (right panel). In this hypothetical case, the group does not show an effect, as the average score is well within the null distribution. Panel B: the results of applying the sign consistency test to effects that produced null results in a non-parametric directional test (N = 26). Significant results, for which the estimated mean sign consistency score is greater than 95% of the null distribution, are marked in blue. As in Figure 2, the x-axis lists effect labels.

We quantified the average within-participant sign consistency of effects for which a directional test did not produce significant results (the same effects reported in Fig. 2). For each individual, sign consistency was defined as the percentage of consistent signs across 500 random splits. Effect scores were calculated using a predefined summary function (i.e., taking the average RT or calculating signal detection sensitivity, d', in each condition, depending on the effect of interest). The results revealed a similar picture to the one provided by the Bayesian approaches (see Figure 3B). The vast majority of cases did not show significant sign consistency, with two exceptions: first, an effect of an unconsciously presented cue on wagering decisions [START_REF] Skora | Evidence that instrumental conditioning requires conscious awareness in humans[END_REF]; M = 63%, p = .001), and second, a scene-object congruency effect [START_REF] Biderman | Evidence for implicit-but not unconscious-processing of object-scene relations[END_REF]; M = 57%, p = .027). Although the two effects were not detected by PBT, the prevalence MAP estimate was above zero for both (7.12% and 6.43%, respectively). Thus, despite some evidence for sign consistency, the overall picture remained the same, hinting at minimal qualitative inter-individual differences in unconscious processing.

Together, four different analysis methods support the conclusion that by and large, unconscious priming effects are not masked by individual differences. Yet one can still claim that these statistical tests are simply not sensitive enough to detect qualitatively variable, nondirectional effects, even when those exist. To test this claim, and demonstrate how these methods can be used to reveal such hidden effects in other fields, we collected additional, openly accessible, datasets from studies conducted in different fields of research within experimental psychology. We then used our non-parametric test on these datasets, demonstrating its potential benefit in determining whether a null result at the group level hides true, but variable, effects at the individual participant level.

Testing within-participant sign consistency across experimental psychology studies

We used the sign consistency test to expose hidden effects that were not revealed by standard directional tests in various fields of research. We collected data from different openaccess databases (the Confidence Database [START_REF] Rahnev | The confidence database[END_REF], the Reproducibility Project (Open Science Collaboration, 2015), and the Classic Visual Search Effects open dataset [START_REF] Adam | Classic visual search effects in an additional singleton task: An open dataset[END_REF]). We also used social media to ask for previously reported null effects. We had two inclusion criteria: first, to estimate subject-level sign consistency, the independent variable had to be manipulated within single participants. And second, we filtered out effects that were significant in a non-parametric, directional test. Overall, we collected data associated with 136 nonsignificant effects (121 from the Confidence Database, four from the Reproducibility Project, eight from the Classic Visual Search Effects open dataset and three from the social media query).

In all cases, participants were excluded for having fewer than five trials per experimental condition and/or zero variance in the dependent variable.

We grouped the different effects into three categories, according to research topics and the analysis we used to test them: first, we tested for effects of participants' responses in 2-alternative forced choice tasks on their confidence ratings in all datasets from the Confidence Database [START_REF] Rahnev | The confidence database[END_REF]; retrieved on 23/1/2023), by comparing the mean confidence ratings between two different responses. Second, we used the same Confidence Database datasets to test for condition where participants performed the same task individually, and vice versa for the other half. As a result, contrasting the two conditions within participants is identical to contrasting early and late trials. Thus, although the interaction between social condition and multisensory integration showed significant sign consistency (M = 62.34%, p < .001, M = 58.54%, p < .001, and M = 66.62%, p < .001, for the three effects that showed sign consistency, paralleled by null results according to directional analysis), we cannot unambiguously interpret these results as suggesting a causal, non-directional, effect of the social manipulation. This is because, under this design, the social setting condition and the order of experimental conditions are perfectly correlated within individual participants, rendering both potential drivers behind the observed effect.

Similarly, the great majority of experiments in the Confidence Database showed significant non-directional effects of response on confidence, such that individual participants were more confident in making one response or the other. Here, order effects are not a concern, as the two responses are expected to be equally distributed within a block. However, since stimulus-response mapping was not counterbalanced within participants, we are unable to tell whether these effects reflect individual differences in stimulus preferences (e.g., enhanced sensory encoding for right-tilted gratings) or in response priming (e.g., reports of high confidence are primed by reporting a decision with the right finger).

As a general principle, counterbalancing of confounding experimental variables can be done either between participants (for example, using a different response-mapping for odd and even participants) or within participants (for example, changing the response-mapping between experimental blocks for all participants). While both approaches are effective in protecting against confounding of the mean tendency of the dependent measures, only within-subject counterbalancing is effective when testing for non-directional effects. Accordingly, unless all confounding variables (e.g., condition order or response-mapping) are randomized within participants, the interpretation of non-directional effects cannot be uniquely linked to causal effects of the experimental manipulation.

Importantly, although we cannot conclusively attribute these non-directional effects to social setting versus condition order in the first example, or to response versus stimulus in the second, they both constitute examples of true effects that were masked by inter-individual differences. The absence of a directional effect in Battich et al is indicated by the fact that on average, participants showed similar levels of multisensory integration in the first and second parts of the experiment ( , 95% CI , ,

). In the case of confidence effects, response mapping was not counterbalanced across participants in many of the considered datasets. This way, the absence of a directional effect of response is also indicative of the absence of a directional effect of stimulus.

Together, these previously hidden non-directional findings make the absence of significant nondirectional effects in unconscious processing a more convincing indication of the true absence of such effects at the individual-participant level.

Discussion

What is the scope and depth of unconscious processing? Previous claims about high-level unconscious processing effects have recently been criticized for methodological reasons [START_REF] Meyen | Advancing research on unconscious priming: When can scientists claim an indirect task advantage[END_REF][START_REF] Rothkirch | What we talk about when we talk about unconscious processing-a plea for best practices[END_REF][START_REF] Shanks | Regressive research: The pitfalls of post hoc data selection in the study of unconscious mental processes[END_REF], and for lack of replicability [START_REF] Hesselmann | Exploring the boundary conditions of unconscious numerical priming effects with continuous flash suppression[END_REF][START_REF] Moors | Scene integration without awareness: No conclusive evidence for processing scene congruency during continuous flash suppression[END_REF]Moors & Hesselmann, 2018;[START_REF] Stein | Unconscious semantic priming from pictures under backward masking and continuous flash suppression[END_REF].

Here, we point out that testing for effects that are consistent across individuals may be overly conservative for the question at stake. Instead, we examined if these null results might still be underlied by an effect, yet a non-directional one. That is, we tested the hypothesis that individual differences in unconscious processing mask true unconscious effects in individual participants.

Adopting a non-directional approach that is robust to inter-individual differences in effects, we used two Bayesian tests [START_REF] Ince | Bayesian inference of population prevalence[END_REF][START_REF] Rouder | Are there reliable qualitative individual difference in cognition[END_REF], an ANOVA-based test [START_REF] Miller | Implications of individual differences in on-average null effects[END_REF], and a novel non-parametric frequentist test. We examined previously reported non-significant results (N = 26), and showed they cannot be explained by inter-individual differences in effects. All tests converged on a similar picture: besides two effects that were picked up by one of the four methods, unconscious processing effects were not masked by substantial inter-individual differences.

It is important to note that our claim here is not about the presence of individual differences in unconscious processing in general, but about the likelihood that such differences in effect signs may be responsible for null group-level findings. Indeed, previous studies revealed inter-individual differences in the magnitude, not the sign, of unconscious processing effects [START_REF] Boy | Individual differences in subconscious motor control predicted by GABA concentration in SMA[END_REF][START_REF] Cohen | Unconscious errors enhance prefrontal-occipital oscillatory synchrony[END_REF][START_REF] Gaal | Pre-SMA gray-matter density predicts individual differences in action selection in the face of conscious and unconscious response conflict[END_REF]. For example, Van Gaal et al. (2011) used fMRI and a meta-contrast masked arrows-priming task, to show that grey matter density is correlated with the size of unconscious motor priming effects. Yet importantly, in this experiment effects were defined according to the assumption that incongruent trials are performed slower than congruent trials (trials where primes and targets pointed to opposing and the same direction, respectively). Here, in contrast, we asked whether relaxing the assumption of effect sign uniformity could reveal unconscious effects that remain undetected using standard directional approaches.

Overall, in two out of 26 effects, a sign consistency test detected an effect that was missed by a standard, directional test. However, even these two effects should be examined cautiously.

The effect found for the third experiment in [START_REF] Biderman | Evidence for implicit-but not unconscious-processing of object-scene relations[END_REF] (M = 57%, p = .027) was not detected by the three other tests and did not survive a correction for false discovery rate among unconscious processing effects [START_REF] Benjamini | Controlling the false discovery rate: A practical and powerful approach to multiple testing[END_REF]. Hence, it is likely that this effect reflects a type-1 error. Similarly, while we found significant sign consistency (M = 63%, p = .001) for a d' effect in the first experiment of [START_REF] Skora | Evidence that instrumental conditioning requires conscious awareness in humans[END_REF], the authors expressed concerns regarding possible contamination of their measured effect by conscious processing due to regression to the mean [START_REF] Shanks | Regressive research: The pitfalls of post hoc data selection in the study of unconscious mental processes[END_REF]. Thus, we suggest that our findings should be interpreted as suggesting no masking of unconscious processing effects by population heterogeneity.

While our focus here was on unconscious processing, a non-directional analysis approach can be useful in many fields of investigation where individual differences are expected. A null finding in a standard t-test or an ANOVA may indicate the true absence of an effect or a lack of statistical power, but it may also be driven by qualitative heterogeneity in participant-level effect signs. In the field of neuroimaging, the adoption of information-based, non-directional approaches famously revealed such effects that were otherwise masked by heterogeneity in neural activation patterns and fine brain structure [START_REF] Gilron | What's in a pattern? Examining the type of signal multivariate analysis uncovers at the group level[END_REF][START_REF] Kriegeskorte | Representational geometry: Integrating cognition, computation, and the brain[END_REF][START_REF] Norman | Beyond mind-reading: Multivoxel pattern analysis of fMRI data[END_REF]. In the context of this investigation, we found considerable evidence for cases where inter-individual differences mask group-level effects. These cases carry theoretical significance both in uncovering previously missed effects, and in revealing aspects of human cognition that are subject to considerable population variability [START_REF] Bolger | Causal processes in psychology are heterogeneous[END_REF][START_REF] Rouder | Beyond means: Are there stable qualitative individual differences in cognition[END_REF][START_REF] Rahnev | Response bias reflects individual differences in sensory encoding[END_REF].

Previously, [START_REF] Rouder | Are there reliable qualitative individual difference in cognition[END_REF] suggested that such qualitative individual differences may be expected in preference or bias-based effects (e.g., [START_REF] Schnuerch | The truth revisited: Bayesian analysis of individual differences in the truth effect[END_REF][START_REF] Rouder | Are there reliable qualitative individual difference in cognition[END_REF], but not in effects that are driven by low-level perceptual and attentional processes.

Consistent with this proposal, the absence of substantial evidence for variability in effect signs in unconscious processing was paralleled with strong evidence for such qualitative inter-individual differences in subjective confidence ratings (e.g., some participants are more confident in classifying a grating as oriented to the right, while others show the opposite preference)3 .

However, robust participant-level effects were masked by qualitative individual differences in other domains too, not all of them relate to higher-level preferences or biases. For example, nondirectional effects of distractor presence were found in a series of visual search experiments [START_REF] Adam | Classic visual search effects in an additional singleton task: An open dataset[END_REF]; sign consistency > 59.34%, p .040, for four out of eight measured effects).

These findings echo the non-directional effects of distractor-target compatibility on action planning that were revealed using the OANOVA test [START_REF] Miller | Implications of individual differences in on-average null effects[END_REF], and were not detected when using standard directional analysis (in both [START_REF] Machado | Action planning in the presence of distracting stimuli: An investigation into the time course of distractor effects[END_REF][START_REF] Machado | Distractibility with advancing age and parkinson's disease[END_REF]. Thus, aside from shedding light on previous non-significant results, our preliminary findings inform previous claims regarding the plausibility of population heterogeneity in effect signs in perceptual and attentional effects in general, providing some indication that such effects may be more prevalent than previously assumed.

To facilitate the adoption of this non-directional approach in experimental psychology, we release with this paper an R package with a simple-to-use implementation of our error-controlled and non-parametric sign consistency test (https://github.com/mufcItay/signcon). We note that unlike directional tests, the validity of the sign consistency test (and more generally, nondirectional tests) depends on counterbalancing of confounding variables not only across participants, but also across trials within a single participant. We recommend using this test to complement standard, directional tests, especially in interpreting null findings at the group level (see Box A for a more detailed description of best-practice recommendations for non-directional testing). This seems especially important in the field of unconscious processing, where null results are becoming more prevalent, and carry theoretical significance as hinting at possible functional roles for conscious processing.

Conclusions

Experimental demonstrations of unconscious processing have been reported for nearly 150 years now (e.g., [START_REF] Peirce | On small differences in sensation[END_REF]), yet their reliability and robustness has repeatedly been put into question (e.g., [START_REF] Holender | Semantic activation without conscious identification in dichotic listening, parafoveal vision, and visual masking: A survey and appraisal[END_REF][START_REF] Shanks | Regressive research: The pitfalls of post hoc data selection in the study of unconscious mental processes[END_REF]. Here, we examined the possibility that some of the findings against such processing, reporting null results, might hide effects at the individual level, yet in opposing directions. We employed four non-directional tests to re-examine 26 null effects. Our findings suggest no role for individual differences in explaining non-significant effects at the group level. Furthermore, by expanding our exploration outside the domain of unconscious processing, we found compelling evidence for effects that were shadowed by individual differences in effect signs, nuancing views about the universality of cognitive and perceptual effects. We provide a user-friendly implementation of the non-directional sign consistency test, and recommend its use for interpreting null results.

Zerweck, I. A., Kao, C. S., Meyen, S., Amado, C., Eltz, M., Klimm, M., & Franz, V. H. (2021).

Number processing outside awareness? Systematically testing sensitivities of direct and indirect measures of consciousness. Attention, Perception,& Psychophysics,83(6), 2510-2529.

Figure 1 .

 1 Figure 1. Simulated data demonstrating how true effects of unconscious priming can be masked by

Figure 2 .

 2 Figure 2. The results of applying the PBT (A), QUID (B), and OANOVA (C) tests to effects that produced

  Panel B: Bayes factors for the comparison between a random effects model that takes into account potential differences in effect signs and the global null model. White markers depict cases where moderate evidence for the global null model was found, while grey markers indicate inconclusive results. The dashed black line indicates a BF of 1 (no preference for either model), and the solid orange lines indicate a BF cutoff of 3. Panel C: p-values obtained by the OANOVA test[START_REF] Miller | Implications of individual differences in on-average null effects[END_REF]. Blue and grey markers indicate significant and non-significant results, respectively. For illustration purposes, BF and significance values are presented on a logarithmic scale on the y-axis.

Figure 3 .

 3 Figure 3. A frequentist, non-parametric, test for sign consistency. Panel A: a schematic illustration, using the same conventions as in Figure 1 (C = congruent, I = incongruent). Participant-wise sign consistency is quantified as the proportion of random splits of experimental trials, for which both halves display the same qualitative effect (C>I or I>C). Group-level sign consistency is compared against a non-parametric null distribution to obtain a significance value. The left panel illustrates a subset of random splits from a hypothetical participant. The upper row illustrates the overall RT data for that participant, and each row below shows one split of the data, where for each half we compare the mean of congruent and

Figure 4 .

 4 Figure 4. Sign consistency test results for null directional effects from different cognitive psychology fields. Turquoise and purple markers indicate the outcomes for datasets from the Confidence Database (Rahnev et al., 2020) that were analysed to reveal differences in confidence and metacognitive sensitivity between responses, respectively. Orange markers indicate the outcomes for effects from various cognitive psychology studies. Finally, for comparison purposes, we also plot here in red the results of the studies on unconscious processing (N=26), reported in the previous section. Lower panel: each point depicts the transformed p-values obtained by the sign consistency test (x-axis) and a directional sign-flipping test (y-axis; datasets were filtered to exclude significant directional effects, hence the minimal directional p-value for all datasets is ). Upper panel: The p-values density distributions that summarize the results in the lower panel for datasets in each field.

Across all RT effects, our analysis used raw RT scores, and thus our results diverged from the original results when log transformations were used (see the notes column in Supplementary Table1for details).

Effect labels abbreviations (sorted alphabetically): BH = Benthien and Hesselmann (2021), BM = Biderman and Mudrik (2018), C = Chien et al. (2022), F = Faivre et al. (2014), H = Hurme et al. (2020), S = Skora et al. (2021), SV = Stein and Peelen (2021), Z = Zerweck et al. (2021). For all labels, numbers denote effect indices within each study (see Supplementary Table1for the full mapping between labels and effects).

As we note above, since in most of these studies responses and stimuli are closely correlated, these effects cannot be unambiguously attributed to stimulus preferences or response priming effects.Relatedly, more recent work reveals that such inter-individual differences in preference for specific responses or stimuli can be traced back to heterogeneity in sensory encoding[START_REF] Rahnev | Response bias reflects individual differences in sensory encoding[END_REF]

metacognitive sensitivity effects of response. Metacognitive sensitivity was quantified as the area under the response-conditional type-2 Receiver Operating Characteristic curve [START_REF] Meuwese | The subjective experience of object recognition: Comparing metacognition for object detection and object categorization[END_REF] here we also excluded datasets that did not include accuracy scores; the remaining 47 effects were analyzed). Third, we grouped effects from the Reproducibility Project (Open Science Collaboration, 2015), the Classic Visual Search Effects open dataset [START_REF] Adam | Classic visual search effects in an additional singleton task: An open dataset[END_REF], and a single study from the social media query [START_REF] Battich | The impact of joint attention on the sound-induced flash illusions[END_REF] under a more general "Cognitive Psychology" category. For these studies, we tested the sign consistency of the effect tested by the original authors (averaged difference or interaction effects).

Across the entire sample, including all analyzed effects (N = 136), most effects (63.24%) showed significant sign consistency. This trend was further explored within each category, revealing significant effects in 89.19% 27.66%, and 46.67%, of the Confidence, Metacognitive Sensitivity, and Cognitive Psychology effects (four visual search effects and all three effects from [START_REF] Battich | The impact of joint attention on the sound-induced flash illusions[END_REF], compared with only 7.69% of the unconscious processing effects, as reported above (see Figure 4). These results validate the potential of using sign consistency to reveal effects on cognition and perception. In striking contrast to the absence of hidden effects in the field of unconscious processing, we found compelling evidence for pronounced interindividual differences that mask group-level effects in other domains.

However, special care should be taken when interpreting non-directional test results, and when designing experiments targeting non-directional effects (see Box A for best-practice recommendations). A case in point can be found in [START_REF] Battich | The impact of joint attention on the sound-induced flash illusions[END_REF], who examined the hypothesis that joint attention affects multisensory integration. Critically, this hypothesis was tested by comparing two social conditions that were counterbalanced across participants, such that for half of the participants a joint attention condition was performed before a baseline

Appendix A. Violating the equal within-individuals variance assumption

We used the simulation scheme described in the main text (see section 'Simulating nondirectional unconscious effects'), to test the consequences of violating the equal withinindividuals variance assumption for both QUID and the OANOVA test. We compared the distribution of Bayes factors and p-values obtained by applying QUID and the OANOVA test to generated data meeting and violating the equal within-participants variance assumption. In the first, equal-variance case, the within-individual standard deviation was low ( ) for all participants. In the second, unequal-variance case, the within-individual standard deviation was low ( ) for all participants except one, for whom it was set to a high value ( ).

As in the main simulation, the effect sizes of each participant were sampled from a normal distribution centred at zero ( ; where denotes the effect size of the participant). Within this framework, examined two scenarios: a non-directional differences scenario where participants are differentially affected by the experimental manipulation ( , ), and a global null condition where all participants are unaffected by the experimental manipulation ( , ). In both scenarios, we simulated random data in 100 iterations, and used the same number of trials per condition (the total number of trials,

).

To examine the tests' specificity, we measured the proportion of iterations where evidence for an effect was erroneously found in the global null condition. In the equal-variance case, all iterations provided evidence for the lack of an effect according to QUID (all BFs < ). Similarly, nonsignificant results were found by the OANOVA test in 93% of the iterations (since the 95% CI of a random binomial variable with and n = 100 is [1 10], the finding 7% false-positive results is expected due to randomness). However, in the unequal-variance case, falsepositives were obtained in 22% of the QUID Bayes Factors (BF > 3), and 2% showed inconclusive evidence. Again, the OANOVA test showed a similar pattern, detecting falsely significant effects in 27% of the iterations. Thus, we show that the specificity of these tests is compromised by violations of the equal-variance assumption.

We then analyzed the tests' outcomes in the non-directional differences scenario to examine their sensitivity. When the equal-variance assumption was met, both tests found evidence for an effect (all BFs > 3, and all p-values < 0.05). In contrast, in the unequal-variance case, only 29% of QUIDs BFs showed evidence for an effect, whilst 64% showed evidence for no effect (the remaining 7% were inconclusive). Similarly, the OANOVA test found significant effects in only 35% of the iterations. Hence, both tests missed true effects when the assumption was not met, demonstrating that their sensitivity is compromised by violations of the equalvariance assumption.

Appendix B. Comparing the power of the PBT and sign consistency test

To examine the differential sensitivity of PBT and the sign consistency tests, we conducted a power analysis, simulating two scenarios under the simulations scheme described in the main text (see section 'Simulating non-directional unconscious effects'): First, a nondirectional differences scenario where an effect exists for each participant but it is inconsistent within participants ( ; where =2). Second, a directional effect scenario, with individual variation around a positive mean effect size ( ; where =2). For each scenario, we manipulated the number of simulated participants ( =10/30/50) and trials ( =20/100/500) across 1000 random iterations, with the within-participant SD ( ) set to 10 in both scenarios. Statistical power was defined as the proportion of significant results in the Sign Consistency case, and as the proportion of intervals that do not include zero in the PBT case. While both tests were similarly sensitive when applied to well-powered datasets (e.g., when =50 or =500), the sign consistency test proved to be more sensitive in the remaining conditions (see Figure 7 in [START_REF] Ince | Bayesian inference of population prevalence[END_REF], for a comparison between the power of PBT and a standard t-test for the directional effect scenario).

Supplementary Figure B1. Power analysis for the sign consistency (SC) and Prevalence Bayesian (PBT) tests for simulated datasets. Each cell depicts the percent of iterations where SC resulted in significant effects, and where the prevalence precluded zero (upper and lower panels, respectively). Rows and columns correspond to the number of simulated participants ( ), and the number of simulated trials per participant ( ), respectively, Left panel: the results of the PBT and SC in the non-directional differences scenario. Right: the results of the PBT and SC in the directional effect scenario.

BOX A: Non-directional testing: best practice recommendations

• When should we use the non-directional approach?

-Not all hypotheses are suitable for examination under the non-directional approach. Since the non-directional approach is targeted at detecting the presence of effects rather than their direction, it cannot be used to establish average differences between conditions at the group level (e.g., when comparing memory performance for items presented first and later in an experiment, rejecting a nondirectional hypothesis does not entail evidence for an overall primacy or recency effect on recollection). In the case of unconscious processing, the theoretical question is regarding the presence or absence of a difference between the two conditions at the single-participant level, and as such, it lends itself to nondirectional testing. Thus, selecting whether to use the non-directional or directional approach is directly linked to the theoretical question at stake.

• Which test should be used?

-When testing whether the experimental manipulation affects the dependent variable (e.g., either main or interaction effects on reaction times, accuracy, brain activity etc.), unless normality and equal variance of within-participant variability can be assumed with high certainty, we recommend using the sign consistency test.

-When these assumptions hold, QUID or OANOVA can be used for effects that are measured on a trial-by-trial basis (as opposed to effects measured by summarizing data from multiple trials, e.g., d' or correlation effects). Specifically, when prior data is available, we advise incorporating it into the analysis using QUID, and when examining an interaction effect, OANOVA provides an easy-to-use solution.

-For an estimation of the true prevalence of individual-level effects, rather than the mere existence of an effect at the group level, we recommend using PBT.

• Non-directional tests require within-participant counterbalancing of confounding variables -As we discuss in the text, special care should be given to counterbalancing of confounding variables when using the non-directional approach. Specifically, unlike standard directional tests, the effects of confounders are not averaged out at the group level when counterbalanced across participants. Thus, counterbalancing should be done not only across participants but also across trials within participants.

• How to interpret non-directional effects?

-In contrast to directional tests, where signal is measured relative to variability across individuals, in non-directional tests it is measured relative to variability across different trials, within an individual. Hence, a positive result of a directional test indicates that effects are consistent between participants, while a non-directional test reveals the presence of an effect on the dependent variable within participants, regardless of the alignment of within participants effects across participants.

-A significant non-directional effect without a corresponding directional effect suggests reliable variability in effect signs across individuals. Whether this variability reflects transient or stable individual differences can be further tested by correlating individual effect scores from two experimental sessions: stable differences should result in a correlation. Whenever stable individual differences are observed, further research may be needed to identify the relevant personal traits that interact with the experimental manipulation.