Gilles Blanchard
email: gilles.blanchard@math.u-psud.fr

Pierre Neuvial
email: pierre.neuvial@math.univ-toulouse.fr

Etienne Roquain
email: etienne.roquain@upmc.fr

On agnostic post hoc approaches to false positive control

gives a partial survey on post hoc approaches to false positive control.

Classical approaches to multiple testing grant control over the amount of false positives for a specific method prescribing the set of rejected hypotheses. On the other hand, in practice many users tend to deviate from a strictly prescribed multiple testing method and follow ad-hoc rejection rules, tune some parameters by hand, compare several methods and pick from their results the one that suits them best, etc. This will invalidate standard statistical guarantees because of the selection effect. To compensate for any form of such "data snooping", an approach which has garnered significant interest recently is to derive "user-agnostic", or post hoc, bounds on the false positives valid uniformly over all possible rejection sets; this allows arbitrary data snooping from the user. In this chapter, we start from a common approach to post hoc bounds taking into account the p-value level sets for any candidate rejection set, and explain how to calibrate the bound under different assumption concerning the distribution of pvalues. We then build towards a general approach to this problem using a family of candidate rejection subsets (call this a reference family) together with associated bounds on the number of false positives they contain, the latter holding uniformly over the family. It is then possible to interpolate from this reference family to find a bound valid for any candidate rejection subset. This general program encompasses in particular the p-value level sets considered initially in the chapter; we illustrate its interest in a different context where the reference subsets are fixed and spatially structured. These methods are then applied to a genomic example (differential gene expression), and a neuromaging example (functional Magnetic Resonance Imaging). Code vignettes to reproduce these examples using the R [START_REF] Core | R: A Language and Environment for Statistical Computing[END_REF] package sansSouci [START_REF] Neuvial | sanssouci: Post hoc multiple testing inference[END_REF] are provided as Supplementary Materials 1 . In this chapter, all references are gathered in Section 11.

A motivating example

Differential gene expression studies in cancerology aim at identifying genes whose activity differs significantly between two (or more) cancer populations, based on a sample of measurements from individuals from these populations. The activity of a gene is usually quantified by its level of expression in the cell. We consider here a microarray data set 2 consisting of expression measurements for more than 12, 000 genes for biological samples from n = 79 individuals with B-cell acute lymphoblastic leukemia (ALL). A subset of cardinal n 1 = 37 of these individuals harbor a specific mutation called BCR/ABL, while the remaining n 2 = 42 do not. One of the goals of this study is to identify, from this sample, those genes for which there is a difference in the mean expression level between the mutated and non-mutated populations. This question can be addressed, after relevant data preprocessing, by performing a statistical test of equality in means for each gene. A classical approach is then to derive a list of "differentially expressed" genes (DEG) as those passing a FDR correction by the Benjamini-Hochberg (BH) procedure at a user-defined level. This is illustrated by Figure 1 for the Leukemia data set, where 163 genes are called "differentially expressed" at FDR level q = 0.05. Dashed line: y = x/m; bold pink solid line: y = qx/m (for q = 0.05), whose intersection with the p-value curve determines the rejections of the BH procedure at level q. Pink dotted line: y = q. Here, 163 genes are declared as differentially expressed.

Setting and basic assumptions

Let us observe a random variable X with distribution P belonging to some model P. Consider m null hypotheses H 0,i ⊂ P, i ∈ N m = {1, . . . , m}, for P . We denote H 0 (P ) = {i ∈ N m : P satisfies H 0,i } the set of true null hypotheses and H 1 (P ) = N m \H 0 (P ) its complement. We assume that a p-value p i (X) is available for each null hypothesis H 0,i , for each i ∈ N m . We introduce the following assumptions on the p-values and the distribution P , that will be useful in the sequel:

∀i ∈ H 0 (P ), ∀t ∈ [0, 1], P (p i (X) ≤ t) ≤ t;
(Superunif)

{p i (X)} i∈H0(P ) is a family of independent variables, which is independent of {p i (X)} i∈H1(P ) . (Indep)
3. From confidence bounds ...

Consider some fixed deterministic S ⊂ N m . A (1 -α)-confidence bound V = V (X)
for |S ∩ H 0 (P )|, the number of false positives in S, is such that A first example is given by the k 0 -Bonferroni bound V (X) = i∈S 1 {p i (X) ≥ αk 0 /|S|}+ k 0 -1, for some fixed k 0 ∈ N m such that k 0 ≤ |S| (otherwise the bound is trivial). The coverage probability is ensured under (Superunif) by the Markov inequality:

P(|S ∩ H 0 (P )| ≥ V + 1) ≤ P |S ∩ H 0 (P )| ≥ i∈S∩H0(P ) 1 {p i (X) ≥ αk 0 /|S|} + k 0 = P i∈S∩H0(P ) 1 {p i (X) < αk 0 /|S|} ≥ k 0 ≤ |S ∩ H 0 (P )|αk 0 /|S| k 0 ≤ α.
However, in practice, S is often chosen by the user and possibly depends on the same data set, then denoted S to emphasize this dependence; it typically corresponds to items of potential strong interest. The most archetypal example is when S consists of the s 0 smallest p-values p (1:m) , . . . , p (s0:m) , for some fixed value of s 0 ∈ N m . In that case, it is easy to check that the above bound does not have the correct coverage: for instance, when the p-values are i.i.d. U (0, 1) and H 0 (P ) = N m , we have for k 0 ≤ s 0 (that is, when the bound is informative),

P(| S ∩ H 0 (P )| ≤ V ) = P k 0 -1 + i∈ Ŝ 1 {p i (X) ≥ αk 0 /s 0 } ≥ s 0 = P i∈ Ŝ 1 {p i (X) < αk 0 /s 0 } ≤ k 0 -1 = P p (k0:m) (X) ≥ αk 0 /s 0 = P(β(k 0 , m -k 0 + 1) ≥ αk 0 /s 0 ),
where β(k 0 , m -k 0 + 1) denotes the usual beta distribution with parameters k 0 and m -k 0 + 1. For instance, taking s 0 = 10, k 0 = 5, α = 0.05 and m = 500, the latter is approximately equal to 0.005, while the intended target is 1 -α = 0.95.

This phenomenon is often referred to as the selection effect: after some data driven selection, the probabilities change and thus the usual statistical inferences are not valid.

... to post hoc bounds

To circumvent the selection effect, one way is to aim for a function V (X, •) :

S ⊂ N m → V (X, S) ∈ N (denoted by V (S) for short) satisfying ∀P ∈ P, P X∼P ∀S ⊂ N m , |S ∩ H 0 (P )| ≤ V (S) ≥ 1 -α, (1)
that is, a (1 -α) confidence bound that is valid uniformly over all subsets S ⊂ N m . As a result, for any particular algorithm S, inequality (1) entails imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

P | Ŝ ∩ H 0 (P )| ≤ V ( Ŝ) ≥ 1 -α,
and thus does not suffer from the selection effect. Such a bound will be referred to as a (1 -α)-post hoc confidence bound throughout this chapter, "post hoc" meaning that the set S can be chosen after having seen the data, and possibly using the data several times. As a first example, the k 0 -Bonferroni post hoc bound is

V k 0 Bonf (S) = |S| ∧ i∈S 1 {p i (X) ≥ αk 0 /m} + k 0 -1 . (2)
Following the same reasoning as above, it has a coverage at least 1 -α under (Superunif):

P(∃S ⊂ N m : |S ∩ H 0 (P )| ≥ V k 0 Bonf (S) + 1) ≤ P ∃S ⊂ N m : i∈S∩H0(P ) 1 {p i (X) < αk 0 /m} ≥ k 0 = P i∈H0(P ) 1 {p i (X) < αk 0 /m} ≥ k 0 ≤ |H 0 (P )|αk 0 /m k 0 ≤ α.
Remark 0.1 Compared to the k 0 -Bonferroni confidence bound of Section 3, α has been replaced by α|S|/m, so that the post hoc bound is much more conservative than a (standard, non uniform, S fixed) confidence bound when |S|/m gets small, which is well expected. This scaling factor is the price paid here to make the inference post hoc. We will see in Sections 5 and 8 that it can be diminished when considering bounds of a different nature.

Coming back to the motivating example of Section 1, if we choose k 0 = 100, the k 0 -Bonferroni post-hoc bound (2) ensures that with probability at least 90%, the number of false positives among the 163 genes selected by the BH procedure at level q = 0.05 is upper bounded by 99.

Example 0.1 For k 0 = 1, when the p-values are i.i.d. U (0, 1) and H 0 (P ) = N m , the coverage probability of the k 0 -Bonferroni post hoc bound is equal to (1 -α/m) m = e m log(1-α/m) , which is very close to 1 -α when α is small.

The Bonferroni post hoc bound, while it is valid under no assumption on the dependence structure of the p-value family, may be conservative, in the sense that V (S) will be large for many subsets S. For instance, one has V k 0 Bonf (S) = |S| (trivial bound) for all the sets S such that S ⊂ {i ∈ N m : p i (X) > αk 0 /m}.

The Bonferroni bound can be further improved under some dependence restriction, with the Simes post hoc bound:

V Sim (S) = min 1≤k≤|S| i∈S 1 {p i (X) ≥ αk/m} + k -1 = min 1≤k≤|S| {V kBonf (S)}. (3)
imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

Its coverage can be computed as follows (using arguments similar as above):

P(∃S ⊂ N m : |S ∩ H 0 (P )| ≥ V Sim (S) + 1)
≤ P ∃S ⊂ N m , ∃k ∈ {1, . . . , m} :

Under (Superunif) and (Indep), this is lower than or equal to α|H 0 (P )|/m ≤ α by using the Simes inequality. More generally, the Simes post-hoc bound is valid in any setting where the Simes inequality holds. This is the case under a specific positive dependence assumption called Positive Regression Dependency on a Subset of hypotheses (PRDS), which is also the assumption under which the Benjamini-Hochberg (BH) procedure has been shown to control the false discovery rate (FDR). While it uses more stringent assumptions, V Sim (S) can be much less conservative than V k 0 Bonf . For instance, if S = {i ∈ N m : 5α/m ≤ p i (X) < 10α/m}, we have V 5Bonf (S) = |S| and V Sim (S) ≤ |S| ∧ 9, which can lead to a substantial improvement. Coming back to the motivating example of Section 1, the Simes post-hoc bound (3) ensures that with probability at least 90%, the number of false positives among the 163 genes selected by the BH procedure at level q = 0.05 is upper bounded by 78.

From Example 0.2 below, the Simes bound has a nice graphical interpretation: |S| -V Sim (S) can be interpreted as the smallest integer u for which the shifted line v → α(v -u)/m is strictly below the ordered p-value curve, see Figure 2. q q q q q q q q q q q q q q q q q q q q 0 5 10 15 20 0.00 0.05 0.10 0.15 0.20 q q q q q q q q q q q q q q q q 0 5 10 15 20 0.00 0.05 0.10 0.15 0.20

where p (1:S) , . . . , p (|S|:S) denote the ordered p-values of {p i (X), i ∈ S}.

Example 0.3 In Figure 2, we have V Sim (S) = 18 (resp. V Sim (S) = 12) in the left (resp. right) situation. Instead, V k 0 Bonf for k 0 = 7 is equal to 18 (resp. 16).

The Simes post hoc bound (3) has, however, some limitations: first, the coverage is only valid when the Simes inequality holds. This imposes restrictive conditions on the model used, which are rarely met or provable in practice. As noted above, the same caveat applies to the BH procedure.

Second, even in that case, the bound does not incorporate the dependence structure, which may yield conservativeness (see Example 0.4 below). Finally, this bound intrinsically compares the ordered p-values to the threshold k → αk/m (possibly shifted). We can legitimately ask whether taking a different threshold (called template below) does not provide a better bound.

Example 0.4 Consider the case H 0 (P ) = N m , for which m is even, and denote Φ the upper-tail distribution function of a standard N (0, 1) variable. Consider the one-sided testing situation where p i = Φ(X 1 ), 1 ≤ i ≤ m/2 and p i = Φ(X 2 ), m/2+1 ≤ i ≤ m, for a 2-dimensional Gaussian vector (X 1 , X 2 ) that is centered, with covariance matrix having 1 as diagonal elements and ρ ∈ [-1, 1] as offdiagonal elements. In this case, one can show that the coverage probability of the Simes post hoc bound is equal to

α/2 + α α/2 Φ Φ -1 (α) -ρΦ -1 (w) (1 -ρ 2 ) 1/2 dw + ∞ α Φ Φ -1 (α/2) -ρΦ -1 (w) (1 -ρ 2 ) 1/2 dw (6)
The above quantity is displayed in Figure 3 for α = 0.2, as a function of ρ.

Threshold-based post hoc bounds

This section presents the λ-calibration method, which allows to derive more accurate threshold-based post hoc bounds under mild assumptions. This is of major interest from a practical perspective, since these assumptions are met in the two-sample multiple testing setting, which is often encountered in applications.

Let us consider bounds of the form

V λ (S) = min 1≤k≤|S| i∈S 1 {p i (X) ≥ t k (λ)} + k -1 , λ ∈ [0, 1], (7)
where

t k (λ), λ ∈ [0, 1], 1 ≤ k ≤ m, is a family of functions, called a template.
A template can be seen as a spectrum of curves, parametrized by λ. We focus here on the two following examples:

imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

-1.0 -0.5 0.0 0.5 1.0 0.17 0.18 0.19 0.20 • Linear template:

t k (λ) = λk/m, t -1 k (y) = ym/k; • Beta template: t k (λ) =λ-quantile of β(k, m -k + 1), t -1 k (y) = P(β(k, m - k + 1) ≤ y).
An illustration for the above templates is provided in Figure 4.

For a fixed template, the idea is now to choose one of these curves, that is, one value of the parameter λ = λ(α), so that the overall coverage is larger than 1 -α. Following exactly the same reasoning as the one leading to (4), we obtain

P(∃S ⊂ N m : |S ∩ H 0 (P )| ≥ V λ (S) + 1) ≤ P ∃k ∈ {1, . . . , |H 0 (P )|} : p (k:H0(P )) < t k (λ) (8)
= P min k∈{1,...,|H0(P

)|} t -1 k (p (k:H0(P )) ) < λ , (9)
by letting t -1 k (y) = max{x ∈ [0, 1] : t k (x) ≤ y} the generalized inverse of t k (in general, this is valid provided that for all k ∈ {1, . . . , m}, t k (0) = 0 and t k (•) is non-decreasing and left-continuous on [0, 1], as in the case of the two above examples). What remains to be done is thus to calibrate λ = λ(α, X) such that the quantity ( 9) is below α.

Several approaches can be used for this. It is possible that for the model under consideration, the joint distribution of (p i (X)) i∈H0(P ) is equal to the restriction of some known, fixed distribution on [0, 1] Nm to the coordinates of H 0 (P ) (this is a version of the so-called subset-pivotality condition). It is met under condition (Indep), but it is also possible that the dependence structure of the p-values is known (for example, in genome-wide association studies, the structure and strength of linkage disequilibrium can be tabulated from previous studies and give rise to a precise dependence model). In such a situation, the calibration of λ = λ(α, X) can be obtained either by exact computation, numerical approximation or Monte-Carlo approximation under the full null.

Another situation of interest, on which we focus for the remainder of this section, is when the null corresponds to an invariant distribution with respect to a certain group of data transformations, which is the setting for (generalized) permutation tests, allowing for the use of an exact randomization technique. More precisely, assume the existence of a finite transformation group G acting onto the observation set X . By denoting p H0 (x) the null p-value vector (p i (x)) i∈H0(P ) for x ∈ X , we assume that the joint distribution of the transformed null p-values is invariant under the action of any g ∈ G, that is,

∀P ∈ P, ∀g ∈ G, (p H0 (g .X)) g ∈G ∼ (p H0 (g .g.X)) g ∈G , (Rand)
where g.X denotes X that has been transformed by g. Let us consider a (random) B-tuple (g 1 , g 2 , . . . , g B ) of G (for some B ≥ 2), where g 1 is the identity element of G and g 2 , . . . , g B have been drawn (independently of the other variables) as i.i.d. variables, each being uniformly distributed on G. Now, let for all x ∈ X , Ψ

(x) = min 1≤k≤m t -1 k p (k:m) (x) and consider λ(α, X) = Ψ ( αB +1) where Ψ (1) ≤ Ψ (2) ≤ • • • ≤ Ψ (B) denote the ordered sample (Ψ(g j .X), 1 ≤ j ≤ B).
The following result holds. Theorem 0.1 Under (Rand), for any deterministic template, the bound V λ(α,X) is a post hoc bound of coverage 1 -α. This level is to be understood as a joint probability with respect to the data and the draw of the group elements (g i ) 2≤i≤B .

As a case in point, let us consider a two-sample framework where X = (X (1) , . . . , X (n1) , X (n1+1) , . . . , X (n1+n2) ) ∈ (R m ) n imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020 is composed of n = n 1 + n 2 independent m-dimensional real random vectors with X (j) , 1 ≤ j ≤ n 1 , i.i.d. N (θ (1) , Σ) (case) and X (j) , n 1 + 1 ≤ j ≤ n, i.i.d. N (θ (2) , Σ) (control). Then we aim at testing the null hypotheses H 0,i : "θ

(1) i = θ (2)
i ", simultaneously for 1 ≤ i ≤ m, without knowing the covariance matrix Σ. Consider any family of p-values (p i (X)) 1≤i≤m such that p i (X) only depends on the i-th coordinate (X (j) i ) 1≤j≤n of the observations (e.g., based on difference of the coordinate means of the two groups). Note that p H0 (X) is thus a measurable function of (X (j) i ) i∈H0,1≤j≤n . Now, the group G of permutations of {1, . . . , n} is naturally acting on X = (R m ) n via the permutation of the individuals: for all σ ∈ G, σ.X = (X (σ(1)) , . . . , X (σ(n1)) , X (σ(n1+1)) , . . . , X (σ(n)) ).

Since the variables (X

(1) i ) i∈H0 , . . . , (X (n) i ) i∈H0 are i.i.d., it is clear that (Rand) holds in this case.
The practical interest of Theorem 0.1 is illustrated in Section 9 for the differential gene expression study introduced in Section 1, and in Section 10 for functional Magnetic Resonance Imaging (fMRI) data. These numerical results demonstrate that substantial gains in power may be obtained by λ-calibration: in both cases, the lower bounds on the number of true positives are two to three times higher than with the classical Simes bounds.

An illustration of the above λ-calibration method is provided in Figure 5 in the case where Σ = I m ,

p i (X) = 2   1 -Φ s -1 n1,n2 n -1 2 n1+n2 j=n1+1 X (j) i -n -1 1 n1 j=1 X (j) i   , for s n1,n2 = (n -1 1 + n -1
2 ) 1/2 and using a Beta template. In the left panel (full null), we have θ (1) = θ (2) = 0, so that H 0 (P ) = N m . In the right panel (half of true nulls), we have θ

(1) i = θ (2) i = 0 for 1 ≤ i ≤ m/2 and θ (1) i = 0, θ (2) i = δ/s n1,n2
for m/2 + 1 ≤ i ≤ m, for some δ > 0, so that H 0 (P ) = {1, . . . , m/2}. Following expression [START_REF] Durand | Post hoc false positive control for spatially structured hypotheses[END_REF], k → t k (λ(α, X)) is the highest Beta curve such that at most Bα orange curves have a point situated below it. This also shows that the above λ-calibration is slightly more severe when part of the data follows the alternative distribution. This is a commonly observed phenomenon: although the permutation approach is valid even when part of the null hypotheses are false, their inclusion in the permutation procedure tends to yield test statistics that exhibit more variation under permutation, thus inducing more conservativeness in the calibration.

Reference families

We cast the previous bounds in a more general setting, where (1 -α)-post hoc bounds are explicitly based on a reference family with some joint error rate (JER In general, a reference family is defined by a collection R = (R 1 (X), ζ 1 (X)), . . . , (R K (X), ζ K (X)) , where the R k 's are data-dependent subsets of N m and the ζ k 's are data dependent integer numbers (we will often omit the dependence in X to ease notation). The reference family R is said to control the JER at level α if

∀P ∈ P, P X∼P (∀k ∈ N K : |R k (X) ∩ H 0 | ≤ ζ k (X)) ≥ 1 -α. (10)
Markedly, ( 10) is similar to [START_REF] Abraham | Machine learning for neuroimaging with scikit-learn[END_REF], but restricted to some subsets R k , k ∈ N K . The rationale behind this approach is that, while the choice of S is let completely free in (1) (to accommodate any choice of the practitioner), the choice of the R k 's and ζ k 's in [START_REF] Genovese | Exceedance control of the false discovery proportion[END_REF] is done by the statistician and is part of the procedure.

Once we obtain a reference family R satisfying [START_REF] Genovese | Exceedance control of the false discovery proportion[END_REF], we obtain a post hoc bound by interpolation:

V * R (S) = max{|S ∩ A| : A ⊂ N m , ∀k ∈ N K , |R k ∩ A| ≤ ζ k }, S ⊂ N m . ( 11
)
We call V * R the optimal post hoc bound (built upon the reference family R). Computing the bound V * R (S) can be time-consuming, it actually has NP-hard complexity in a general configuration. We can introduce the following comimsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

putable relaxations: for S ⊂ N m , V R (S) = min k∈N K (|S \ R k | + ζ k ) ∧ |S|; ( 12
) V R (S) = k∈N K |S ∩ R k | ∧ ζ k + S \ k∈N K R k ∧ |S|. (13)
One can easily check that

V * R (S) ≤ V R (S) and V * R (S) ≤ V R (S)
for all S ⊂ N m . Moreover, provided that (10) holds, V * R , V R and V R are all valid (1 -α)-post hoc bounds. The details are left to the reader.

In addition, the following result shows that the relaxed versions coincide with the optimal bound if the reference sets have some special structure:

Lemma 0.1 • In the nested case, that is, R k ⊂ R k+1 , for 1 ≤ k ≤ K -1, we have V R = V * R ; • In the disjoint case, that is, R k ∩ R k = ∅ for 1 ≤ k = k ≤ K, we have V R = V * R .
We can briefly revisit the post-hoc bounds of the previous sections in this general framework. The k 0 -Bonferroni post hoc bound (2) derives from the oneelement reference family (R = {i ∈ N m : p i (X) < αk 0 /m} , ζ = k 0 -1). The Simes post hoc bound (3) derives from the reference family comprising the latter reference sets for all k 0 ∈ N m . More generally, the threshold-based post hoc bounds V λ of the form [START_REF] Dudoit | Multiple testing procedures with applications to genomics[END_REF] are equal to the optimal bound

V * R with R k = {i ∈ N m : p i (X) < t k (λ)} and ζ k = k -1, k ∈ N m (
indeed, these reference sets are nested, so that V * R = V R ). How to choose a suitable reference family in general? A general rule of thumb is to choose the reference sets R k of the same qualitative form as the sets S for which the bound is expected to be accurate. For instance, the Simes post hoc bound will be more accurate for sets S with the smallest p-values. In Section 8, we will choose reference sets R k with a spatial structure, which will produce a post hoc bound more tailored for spatially structured subsets S.

Case of a fixed single reference set

It is useful to focus first on the case of a single fixed (non-random) reference set R 1 , with (random) ζ 1 satisfying [START_REF] Genovese | Exceedance control of the false discovery proportion[END_REF], that is, • under (Superunif), for some fixed t ∈ (0, α),

P(|H 0 (P ) ∩ R 1 | ≤ ζ 1 (X)) ≥ 1 -α.
ζ 1 (X) = |R 1 | ∧ i∈R1 1 {p i (X) > t}/(1 -t/α) , (14)
where x denotes the largest integer smaller than or equal to x (this is a simple application of the Markov inequality). • under (Superunif) and (Indep),

ζ 1 (X) = |R 1 |∧ min t∈[0,1) C 2(1 -t) + C 2 4(1 -t) 2 + i∈R1 1{p i (X) > t} 1 -t 1/2 2 , ( 15
)
where C = (this can deduced by using the DKW inequality, that is, for any integer n ≥ 1, for U 1 , . . . , U n i.i.d. U (0, 1), we have

sup t∈[0,1] n -1 n i=1 1{U i > t} -(1 -t) ≥ -log(1/λ)/(2n) with prob- ability at least 1 -λ).
In addition to the two above bounds ( 14) and ( 15), we can elaborate another bound in the generalized permutation testing framework (Rand), as described in Section 5. Applying the result of that section, the following bound is also valid:

ζ 1 (X) = min 1≤k≤|R1| i∈R1 1 {p i (X) ≥ t k (λ(α, X))} + k -1 , (16)
where t k (λ) denotes the λ-quantile of a β(k, |R 1 | -k + 1) distribution and λ(α, X) = Ψ ( αB +1) , where Ψ (1) ≤ Ψ (2) ≤ • • • ≤ Ψ (B) denote the ordered sample (Ψ(g j .X), 1 ≤ j ≤ B) for which Ψ(x) = min 1≤k≤|R1| t -1 k p (k:|R1|) (x) (see the λ-calibration method of Section 5).

Once a proper choice of ζ 1 (X) has been done, the optimal post hoc bound can be computed as follows: for any

S ⊂ N m , V * R (S) = V R (S) = V R (S) = |S ∩ R c 1 |+ζ 1 (X)∧|S ∩R 1 |.
When S is large and does not contain very small p-values, this bound can be sharper than the Simes bound. For instance, let us consider the single reference family R 1 = N m and ζ 1 (X) as in [START_REF] Katsevich | Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings[END_REF]

(choosing t = 1/2). For S such that S ⊂ {i ∈ N m : p i (X) > α|S|/m}, we have V Sim (S) = |S| and V * R (S) = |S| ∧ ζ 1 (X) ≤ |S| ∧ 2 log 1 α + 2 i∈Nm 1 {p i (X) > 1/2}
. The latter can be smaller than |S| when many p-values are below 1/2.

Finally, while the case of a single reference set can be considered as an elementary example, the bounds developed in this section will be useful in the next section, for which several fixed reference sets R k are considered, and thus several (random) ζ k should be designed.

imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

Case of spatially structured reference sets

We consider here the case where the null hypotheses H 0,i , 1 ≤ i ≤ m, have a spatial structure, and we are interested in obtaining accurate bounds on |S ∩ H 0 (P )| for subsets S of the form S = {i ∈ N m : i 0 ≤ i ≤ j 0 }, for some 1 ≤ i 0 < j 0 ≤ m.

In that case, it is natural to choose R k formed of contiguous indices. To be concrete, consider reference sets consisting of disjoint intervals of the same size : assume m = Ks for some integers K > 0 and s > 0 and let

R k = {(k -1)s + 1, . . . , ks}, k ∈ N K . ( 17
)
When each of these regions is considered in isolation, Section 7 suggested several approaches (in the appropriate settings (Superunif), (Indep) or (Rand)) of a specific form ζ k (X) = f (R k , α, X), to underline the dependence of ζ k (X) in R k and α. By using a simple union bound, it is then straightforward to show that the JER control ( 10) is satisfied for

ζ k (X) = f (R k , α/K, X), k ∈ N K . ( 18
)
When the reference regions R k are disjoint as in the example ( 17) above, we can use the proxy V R (S) (see [START_REF] Hemerik | False discovery proportion estimation by permutations: confidence for significance analysis of microarrays[END_REF]) which is known to coincide with the optimal bound V * R (S). This gives rise to a post hoc bound that accounts for the spatial structure of the data.

Example 0.6 In the case where ζ 1 (X) = f (R 1 , α, X) is given by (14) (t = α 2 , K < 1/α), we obtain

ζ k (X) = |R k | ∧ i∈R k 1 p i (X) > α 2 /(1 -αK) .
Note that this bound quickly increases as the size of the family K increases. By contrast, when ζ 1 (X) = f (R 1 , α, X) is given by (15), one can derive

ζ k (X) = |R k | ∧ min t∈[0,1) 2(1 -t) + C 2 4(1 -t) 2 + i∈R k 1{p i (X) > t} 1 -t 1/2 2 , for C = 1 2 log K α .
The size of the family K appears here in a logarithmic term, which makes this bound less sensitive to the parameter K.

When considering the reference regions defined by segments [START_REF] Meijer | A region-based multiple testing method for hypotheses ordered in space or time[END_REF], we have to prescribe a scale (s here, the size of the segments). It is possible to extend this to a multi-scale approach, choosing overlapping reference intervals R k at different resolutions arranged in a tree structure, where parent sets are formed by taking union of (disjoint) children sets taken at a finer resolution. Furthermore, the proxy (13) has to be replaced by a more elaborate one, minimizing over all imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020 / possible multi-scale partitions made of such reference regions. This can still be computed efficiently by exploiting the the tree structure. Doing so, the post hoc bound will be more scale adaptive to sets S with possibly various sizes. The price to pay lies in the cardinality K of the family, which gets larger. However, this does not necessarily make the corresponding bound much larger, as Example 0.6 shows when using the bound [START_REF] Katsevich | Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings[END_REF], since the level α only enters it logarithmically.

Application to differential gene expression studies

In this section, we illustrate how the post hoc inference framework introduced in the preceding sections can be applied to the case of differential gene expression introduced in Section 1 to build confidence envelopes for the proportion of false positives (Section 9.1), and to obtain bounds on data-driven sets of hypotheses (Section 9.2), and on sets of hypotheses defined by an a priori structure (Section 9.3). These numerical results were obtained using the R package sansSouci, version 0.9.0. A Rmarkdown vignette3 to reproduce results and plots from this section is provided as Supplementary Material.

Confidence envelopes

In absence of specific prior information on relevant subsets of hypotheses to consider, it is natural to focus on subsets consisting of the most significant hypotheses. Specifically, we define the k-th p-value level set S k as the set of the k most significant hypotheses, corresponding to the p-values (p (1:m) , p (2:m) , . . . , p (k:m) ), and consider post hoc bounds associated to S k for k ∈ N m . Figure 6 The confidence envelopes are built from the Simes bound (3) (long-dashed purple curve), and from two bounds obtained from Theorem 0.1 by λ-calibration using B = 1, 000 permutation of the sample labels, based on the two templates introduced in Section 5: the dashed orange curve corresponds to the linear template with K = m, and the solid green curve to the Beta template with K = 50. Note that Assumption (Rand) holds because we are in the two-sample framework described after Theorem 0.1.

The vertical line in Figure 6 corresponds to the 163 genes selected by the BH procedure at level 5%. The Simes bound ensures that the FDP of this subset is not larger than 0.48. As noted above concerning the BH procedure, we have a priori no guarantee that this bound is valid, because such multiple two-sample testing situations have not been shown to satisfy the PRDS assumption under which the Simes inequality is valid 4 . In contrast, the λ-calibrated bounds built by permutation are by construction valid here. Moreover, both are much sharper than the Simes bound while the λ-calibrated bound using the linear template is twice smaller, ensuring FDP< 0.23, and even smaller for the Beta template with K = 50. The bound obtained by λ-calibration of the linear template is uniformly sharper that the original Simes bound (3), which corresponds to λ = α. This illustrates the adaptivity to dependence achieved by λ-calibration. The bound obtained from the Beta template is less sharp for p-value level sets S k of cardinal less than k = 120, and then sharper. This is consistent with the shape of the threshold functions displayed in Figure 4.

Data-driven sets

A common practice in the biomedical literature is to only retain, among the genes called significant after multiple testing correction, those whose "fold change" exceeds a prescribed level. The fold change is the ratio between the mean expression levels of the two groups. With the notation of Section 5, the foldchange of gene i is given by ∆ i = X

(2)

i /X

(1)

i , where

X (1) i = n -1 1 n1 j=1 X (j) i and X (2) i = n -1 2 n2 j=1 X (j)
i . This is illustrated by Figure 7, where each gene is represented as a point in the (log(fold change), -log(p)) plan. This representation is called a "volcano plot" in the biomedical literature. Among the 163 genes selected by the BH procedure at level 0.05, 151 have an absolute log fold change larger than 0.3. As FDR is not preserved by selection, FDR controlling procedures provide no statistical guarantee on such data-driven lists of hypotheses.

In contrast, the post hoc bounds proposed in this chapter are valid for such data-driven sets. The two shaded boxes in Figure 7 correspond to the datadriven subsets S BH ∩ S -and S BH ∩ S + , where S BH is the set of 163 genes selected by the BH procedure at level 0.05, S -= {i ∈ N m , log(∆ i ) < -0.3} and S + = {i ∈ N m , log(∆ i ) > +0.3}. The post hoc bounds on the number of true positives in S BH ∩ S + , S BH ∩ S -and S BH ∩ (S + ∪ S -) obtained by the Simes bound and by the λ-calibrated linear and Beta templates are given in Table 1. Both λ-calibrated bounds are more informative than the Simes bound, in the sense that they provide a higher bound on the number of true confidence. Moreover, they have proven (1 -α)-coverage, whereas the coverage of the Simes bound is a priori unknown for multiple two-sample tests. None of the two λcalibrated bounds dominates the other one, which is in line with the fact that the linear template is well-adapted to situations with smaller p-value level sets than the Beta template.

Finally, we also note that the bound obtained for S + ∪ S -is systematically larger than the sum of the two individual bounds, which, again, is in accordance with the theory. Post-hoc inference for volcano plots. Each dot corresponds to a gene in the (fold change, p-value space) on a logarithmic scale. Bold red dots corresponds to 151 genes that (i) are selected by the BH procedure at level q = 0.05, and (ii) have a absolute log fold change larger than 0. 1 Post hoc bounds on the number of true positives in S BH ∩ S + , S BH ∩ S -and S BH ∩ (S + ∪ S -) obtained by the post hoc bounds displayed in Figure 6.

Structured reference sets

In this section we give an example of application of the bounds mentioned in Section 8. Our biological motivation is the fact that gene expression activity can be clustered along the genome. The m individual hypotheses are naturally partitioned into 23 subsets, each corresponding to a given chromosome. Within each chromosome, we consider sets of s = 10 successive genes as in [START_REF] Meijer | A region-based multiple testing method for hypotheses ordered in space or time[END_REF]. Hence, we focus on a reference family with the following elements

R c,k = {(k -1)s + 1, . . . , min(ks, m c )}, k ∈ N Kc , c ∈ {1, . . . , 23},
where, in chromosome c, m c denotes the number of genes, K c = m c /s the number of corresponding regions. In addition, for each (c, k) we use ζ c,k (X) = f (R c,k , α c /K c , X) coming from the union bound [START_REF] Meinshausen | False discovery control for multiple tests of association under general dependence[END_REF] in combination with the device [START_REF] Katsevich | Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings[END_REF] and α c = αm c /m. This choice accounts for a union bound over all the chromosomes. As shown in Example 0.5, ζ c,k (X) is a valid upper confidence bound for |H 0 (P ) ∩ R c,k | under (Superunif) and (Indep). In this genomic example, (Indep) may not hold, so we have in fact no formal guarantee that this bound is valid. Therefore, the results obtained below are merely illustrative of the approach and may not have biological relevance.

We report the results for chromosome c = 19, which contains m c = 626 genes. In this particular case, we obtain trivial bounds ζ c,k (X) = |R c,k | for all k ∈ N Kc . Therefore, the proxy Ṽ * R defined in [START_REF] Hemerik | False discovery proportion estimation by permutations: confidence for significance analysis of microarrays[END_REF] for disjoint sets does not identify any signal for this chromosome. However, non-trivial bounds can be obtained via the multi-scale approach briefly mentioned in Section 8. The idea is to enrich the reference family by recursive binary aggregation of the neighboring R c,k . The total number of elements in this family is less than 2K c . In our example, it turns out that (15) yields 6 true discoveries in the interval R 17:24 and 1 true discovery in the interval R 53:54 , where we have denoted

R u:v = u≤k≤v R c,k .
This is illustrated by Figure 8 where the individual p-values are displayed (on the -log 10 scale) as a function of their order on chromosome 19. The sets R 17:24 and R 53:54 are highlighted in orange, with the corresponding number of true discoveries marked in each region. We obtain a non-trivial bound not because of the large effect of any individual gene, but because of the presence imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 0 100 200 300 400 500 600 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q Fig 8 . Evidence of locally-structured signal on chromosome 19 detected by the bound [START_REF] Katsevich | Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings[END_REF]. The numbers correspond to the lower bound on the false positives in each of the highlighted regions.

of sufficiently many moderate effects. In particular, in the rightmost orange region in Figure 8, the distribution of -log 10 (p) is shifted away from 0 when compared to the rest of chromosome 19. In comparison, we obtain trivial bounds V R (R 53:54 ) = |R 53:54 | = 2s and V R (R 17:24 ) = |R 17:24 | = 8s from (12) both for the linear or the Beta template. These numerical results illustrate the interest of the bounds introduced in Section 8 in situations where one expects the signal to be spatially structured.

Application to fMRI studies

We focus on the problem of detecting brain regions whose activity is significantly different between two motor tasks performed by subjects: left versus right click. The fMRI data have been extracted from the Localizer data set 5 . A Rmarkdown vignette to reproduce results and plots from this section is provided as Supplementary Material.

Confidence envelopes

As in Section 9, we begin by constructing confidence envelopes for top-k feature lists. Figure 9 The confidence envelopes are built from the Simes bound (3) (long-dashed purple curve), and from two bounds obtained from Theorem 0.1 by λ-calibration using B = 1, 000 permutation of the sample labels, based on the two templates introduced in Section 5: the dashed orange curve corresponds to the linear template with K = m, and the solid green curve to the Beta template with K = 500. Assumption (Rand) holds because we are in the two-sample framework described after Theorem 0.1. The Simes bound is also called All Resolution Inference (ARI) in that context, see Supplementary Material 2 for more details and references. The results are qualitatively similar as for the genomic example given in Section 9. Both permutation-based post hoc bounds are much sharper than the Simes bound, illustrating the adaptivity to dependence achieved by λcalibration.

Post hoc bounds on brain atlas areas

The goal in this section is to calculate post hoc bound on user-defined brain regions. One definition of such regions is given by the Harvard-Oxford brain imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020 atlas6 . We have calculated the post hoc bounds associated to each of these 48 areas: by definition, these are confidence bounds valid simultaneously for all areas. In this particular example, evidence of signal is obtained for three of these atlases, as summarized in Table 2 both permutation-based bounds are much less conservative than the classical Simes/ARI bound, showing that permutation-based approaches are able to adapt to unknown dependency.

Bibliographical notes

The material exposed in this chapter is mainly a digested account of the article [START_REF] Blanchard | Post hoc confidence bounds on false positives using reference families[END_REF]. The seminal work [START_REF] Genovese | Exceedance control of the false discovery proportion[END_REF] introduced the idea of false positive bounds for arbitrary rejection sets. It started from the idea of building a confidence set on the set of null hypotheses H 0 (P ), and introduced the concepts of augmentation procedure and inversion procedure. The latter consists in building a confidence set based on the inversion of tests for H 0 (P ) = A for all A ⊂ N m . The former starts from a set R with controlled k-familywise error rate, and the proposed associated post hoc bound is (10) (for the one-element reference family (R, ζ = k -1)). The name augmentation refers to a similar idea found in [START_REF] Dudoit | Multiple testing procedures with applications to genomics[END_REF]. The relaxation [START_REF] Genovese | Exceedance control of the false discovery proportion[END_REF] can in this sense be called "generalized augmentation procedure". A post hoc bound for an arbitrary rejection set based on a closed test principle was proposed in [START_REF] Goeman | Multiple testing for exploratory research[END_REF]. It can also be seen as a reformulation of the inversion procedure of [START_REF] Genovese | Exceedance control of the false discovery proportion[END_REF]. Post-hoc bounds over a large class of reference families extracted from classical FDR control procedures combined with martingale techniques were recently proposed in [START_REF] Katsevich | Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings[END_REF]. The principle of the graphical representation used in Figure 2 to visualize the Simes inequality-based bound originates from J. Goeman.

The use of generalized permutation procedures in a multiple testing framework has been explored in several landmark works [START_REF] Westfall | Resampling-Based Multiple Testing[END_REF][START_REF] Romano | Exact and approximate stepdown methods for multiple hypothesis testing[END_REF][START_REF] Meinshausen | False discovery control for multiple tests of association under general dependence[END_REF][START_REF] Dudoit | Multiple testing procedures with applications to genomics[END_REF][START_REF] Hemerik | Exact testing with random permutations[END_REF][START_REF] Hemerik | Permutation-based simultaneous confidence bounds for the false discovery proportion[END_REF]. The subset-pivotality condition has been defined in [START_REF] Westfall | Resampling-Based Multiple Testing[END_REF]. Assumption (Rand) has been introduced in [START_REF] Hemerik | False discovery proportion estimation by permutations: confidence for significance analysis of microarrays[END_REF] and is a weaker version of the randomization hypothesis of [START_REF] Romano | Exact and approximate stepdown methods for multiple hypothesis testing[END_REF]. The phenomenon of conservativeness in the permutation-based calibration mentioned at the end of Section 5, when not all the null hypotheses are true, can be in part alleviated by using a step-down principle (see [START_REF] Romano | Exact and approximate stepdown methods for multiple hypothesis testing[END_REF] for a seminal work on this topic and [START_REF] Blanchard | Post hoc confidence bounds on false positives using reference families[END_REF] for more details on this approach in the specific setting considered here). The choice of the size K of the reference family, which can be crucial in practice, is also discussed in [START_REF] Blanchard | Post hoc confidence bounds on false positives using reference families[END_REF].

Multiple testing for spatially structured hypotheses is in itself a very active and broad area of research. It has been specifically considered in conjunction with post-hoc bounds in [START_REF] Meijer | A region-based multiple testing method for hypotheses ordered in space or time[END_REF]. The use of the reference family approach for post-hoc bounds in combination with spatially structured hypotheses has been studied in [START_REF] Durand | Post hoc false positive control for spatially structured hypotheses[END_REF], where the notion of tree-(or forest-)structured reference regions is introduced, along with an efficient algorithm to compute the optimal bound V * R in this setting.

The Simes inequality [START_REF] Simes | An improved Bonferroni procedure for multiple tests of significance[END_REF] is a particularly nice and elegant theoretical device with manifold applications in multiple testing which is still a very active research area, see, e.g., [START_REF] Block | The multivariate-t distribution and the Simes inequality[END_REF][START_REF] Bodnar | On the Simes inequality in elliptical models[END_REF][START_REF] Finner | On the Simes test under dependence[END_REF]. The DKW inequality with optimal constant was proved in [START_REF] Massart | The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality[END_REF]. The Benjamini-Hochberg (BH) procedure has been introduced in [START_REF] Benjamini | Controlling the false discovery rate: a practical and powerful approach to multiple testing[END_REF], where it is also proved to control the false discovery rate (FDR). A huge literature on FDR control has followed this seminal paper.

The data used for the genomics application in Section 9 are taken from [START_REF] Chiaretti | Gene expression profiles of b-lineage adult acute lymphocytic leukemia reveal genetic patterns that identify lineage derivation and distinct mechanisms of transformation[END_REF]. The fact that the signal is clustered along the genome is motivated by previous studies showing possible links between gene expression and DNA copy number changes or other regulation mechanisms [START_REF] Reyal | Visualizing chromosomes as transcriptome correlation maps: evidence of chromosomal domains containing coexpressed genes -a study of 130 invasive ductal breast carcinomas[END_REF][START_REF] Stransky | Regional copy number-independent deregulation of transcription in cancer[END_REF]. The data used for the neuroimaging application in Section 10 were obtained from the Brainomics/Localizer database [START_REF] Orfanos | The brainomics/localizer database[END_REF] and the Harvard-Oxford brain atlas7 , using the Python package nilearn [START_REF] Abraham | Machine learning for neuroimaging with scikit-learn[END_REF]. The ARI method mentioned in that section [START_REF] Rosenblatt | Allresolutions inference for brain imaging[END_REF] corresponds to the Simes post hoc bound of [START_REF] Goeman | Multiple testing for exploratory research[END_REF].

Vignette: "Permutation-based post hoc inference for fMRI studies" This Rmarkdown vignette9 demonstrates how the R package sansSouci may be used to obtain post hoc confidence bounds on false positives in the case of functional Magnetic Resonance Imaging (fMRI) studies. In particular, in contains R code to reproduce Figure 9 and Table 2.

11

  Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Supplementary Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Fig 1 .

 1 Fig 1. Left: sorted p-values for the Leukemia data set (thin black solid line). Right: zoom on the smallest 400 p-values. Dashed line: y = x/m; bold pink solid line: y = qx/m (for q = 0.05), whose intersection with the p-value curve determines the rejections of the BH procedure at level q. Pink dotted line: y = q. Here, 163 genes are declared as differentially expressed.

i∈S∩H0(P ) 1

 1 {p i (X) < αk/m} ≥ k = P ∃k ∈ {1, . . . , |H 0 (P )|} : p (k:H0(P )) < αk/m .

Fig 2 .

 2 Fig 2. Illustration of the Simes post hoc bound (3) according to the expression (5), for two subsets of Nm (left display/right display), both of cardinal 20 and for m = 50. The level is α = 0.5 (taken large only for illustration purposes). Black dots: sorted p-values in the respective subsets. Lines: thresholds k ∈ {u + 1, . . . , |S|} → α(k -u)/m (in bold red for u = |S| -V (S), in light gray otherwise). The post hoc bound V Sim (S) corresponds the length of the bold line on the X-axis.

Fig 3 .

 3 Fig 3. Coverage of the Simes post hoc bound (6) in the setting of Example 0.4 as a function of ρ and for α = 0.2.

  imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

Fig 4 .

 4 Fig 4. Curves k → t k (λ) for a wide range of λ values. Left: linear template. Right: Beta template.

Fig 5 .

 5 Fig 5. Illustration of the λ = λ(α, X) calibration method on one realization of the data X. Black curves: Beta template k → t k (λ) for some range of λ values. Orange curves: ordered p-values (after permutation) k → p (k:m) (g j .X) for 1 ≤ j ≤ B = 1000. Bold red curve: k → t k (λ(α, X)). Left panel : full null, right panel : half of true nulls (see text). (Parameters m = 50, α = 0.2, n 1 = 50, n 2 = 50, δ = 3.)

(

  In contrast with the k 0 -Bonferroni bound[START_REF] Benjamini | Controlling the false discovery rate: a practical and powerful approach to multiple testing[END_REF] where ζ was fixed and R variable, here R 1 is fixed andζ 1 is variable.) In other words, ζ 1 (X) is a (1-α)-confidence bound of |H 0 (P ) ∩ R 1 |.Several examples of such ζ 1 (X) can be built, under various assumptions.imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020Example 0.5 For R 1 ⊂ N m fixed, the following bounds are (1 -α)-confidence bounds for |H 0 (P ) ∩ R 1 |:

1 2 log 1 α

 1

  provides post hoc confidence envelopes for the ALL data set, for α = 0.1. While (1 -α)-lower confidence bounds on the proportion of false positives k, V (S k )/ |S k | : k ∈ N m are displayed in the left panel, (1 -α)-upper confidence bounds on the number of true positives of the form k, |S k | -V (S k ) : k ∈ N m are shown in the right panel.

Fig 6 .

 6 Fig 6. Confidence bounds on the proportion of false positives (left) and on the number of true positives (right) for the Leukemia data set. Reference families: Simes reference family (long-dashed purple curve), linear template after λ-calibration (dashed orange curve), and Beta template after λ-calibration (solid green curve).

  ≥ 96 ; FDP ≤ 0.23 27 genes TP ≥ 9 ; FDP ≤ 0.67 151 genes selected At least 123 true positives (FDP ≤ 0.19 )

Fig 7 .

 7 Fig 7. Post-hoc inference for volcano plots. Each dot corresponds to a gene in the (fold change, p-value space) on a logarithmic scale. Bold red dots corresponds to 151 genes that (i) are selected by the BH procedure at level q = 0.05, and (ii) have a absolute log fold change larger than 0.3.

  provides post hoc confidence envelopes for the Localizer data set, for α = 0.1. While (1 -α)-lower confidence bounds on the proportion of false positives k, V (S k )/ |S k | : k ∈ N m are displayed in the left panel, (1 -α)-upper confidence bounds on the number of true positives of the form k, |S k | -V (S k ) : k ∈ N m are shown in the right panel.

Fig 9 .

 9 Fig 9. Confidence bounds on the proportion of false positives (left) and on the number of true positives (right) for the fMRI data set. Reference families: Simes reference family (longdashed purple curve), linear template after λ-calibration (dashed orange curve), and Beta template after λ-calibration (solid green curve).

Table 2

 2 . For the same target risk level (α = 0.1), Lower bounds on the number of true positives in 3 brain regions.

		area	size	Simes/ARI Linear Beta (K=500)
	7	Precentral Gyrus	6640	79	254	281
	17	Postcentral Gyrus	5030	69	202	196
	19 Supramarg. Gyrus (ant. div.) 2017	0	3	0

The corresponding source files are available from the package web site: https://pneuvial.github.io/sanssouci/.

Taken from Chiaretti et. al., Clinical cancer research, 11(20):7209-7219, 2005.

See https://github.com/rstudio/rmarkdown imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

In this particular case, λ-calibration with the linear template yields λ(α) > α, which a posteriori implies that the Simes inequality was indeed valid.

Orfanos, D. P. et al. Neuroimage, 181:786-796 (2017). imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Atlases imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Atlases

Available athttps://pneuvial.github.io/sanssouci/articles/post-hoc_ differential-expression.html.

Available at https://pneuvial.github.io/sanssouci/articles/post-hoc_fMRI.html. imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

imsart-generic ver. 2014/10/16 file: BNR-chap_HAL-revised.tex date: October 13, 2020

Acknowledgements

We would like to thank referees for their helpful comments. PN would like to thank Alexandre Blain who contributed to the application to fMRI studies, and Bertrand Thirion for constructive feedback on this application. This work has been supported by ANR-16-CE40-0019 (SansSouci), ANR-17-CE40-0001 (BASICS) and by the GDR ISIS through the "projets exploratoires" program (project TASTY). GB acknowledges the support from the german DFG under the Collaborative Research Center SFB-1294 "Data Assimilation". GB and PN acknowledge the support from the Franco-German University through the binational Doktorandenkolleg CDFA 01-18.

Supplementary Materials

Vignette "Permutation-based post hoc inference for differential gene expression studies":

This Rmarkdown vignette 8 demonstrates how the R package sansSouci may be used to obtain post hoc confidence bounds on false positives in the case of differential gene expression analysis. In particular, it contains the R code to reproduce Figures 1, 6 and 7, and Table 1.